# Study Room — complete public teaching content All subjects, M2/MT2 and TEE. Equations include original LaTeX. Diagrams include alt text, available SVG labels and original asset links. Private video media remain account-restricted; their existing catalog metadata are exported. # Deep Learning — all public teaching content Exact source JSON: https://deep-learning-study-room.vercel.app/exports/dl.json --- ## MT2 · Mean, variance and standard deviation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/1 The mean is a measure of location; variance and standard deviation quantify dispersion. Distinguish population and sample conventions before calculating these statistics. $$ \mu=\frac{1}{n}\sum_i x_i $$ $$ v=\frac{1}{n}\sum_i(x_i-\mu)^2 $$ $$ s=\sqrt{v} $$ ## Mean, variance and standard deviation from ordinary arithmetic **Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i,\qquad \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2,\qquad \sigma=\sqrt{\sigma^2} $$ These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable. **Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].** ### Given Observations = [3,3,7,7]; N = 4. Use population variance (divide by N). ### To find Find the population mean μ, variance σ² and standard deviation σ. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3. ### Population mean and spread: [1, 2, 3] **Question. For the population [1, 2, 3], find mean, population variance and standard deviation.** ### Given Values = [1, 2, 3]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 1+2+3=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (1-2)=-1 $$ $$ (-1)^2=1 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (3-2)=1 $$ $$ (1)^2=1 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{1+0+1}{3}\approx0.666666667 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.666666667}\approx0.816496581 $$ **Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581. ### Population mean and spread: [2, 2, 2] **Question. For the population [2, 2, 2], find mean, population variance and standard deviation.** ### Given Values = [2, 2, 2]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 2+2+2=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{0+0+0}{3}\approx0.000000000 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.000000000}\approx0.000000000 $$ **Final answer:** μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation. --- ## MT2 · Standardization Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/2 Standardization has a meaning before it has a formula: measure how far an input is from its training mean, then express that distance in standard-deviation units. The worked question below calculates the mean and standard deviation first, so neither number appears without explanation. $$ x_{scaled}=\frac{x-\mu_{train}}{s_{train}} $$ ## Standardization: what the answer 1 actually means **Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units. $$ z=\frac{x-\mu_{\mathrm{train}}}{\sigma_{\mathrm{train}}},\qquad \sigma_{\mathrm{train}}>0 $$ Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1]. **Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.** ### Given Training observations = [3,3,7,7]. New observation = 9. Estimate the population standard deviation from the training values. ### To find Find the training mean and standard deviation; standardize all four training values and the new observation. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. **Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units. x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Step 8: standardize each original training observation.** $$ \frac{3-5}{2}=-1 $$ $$ \frac{7-5}{2}=1 $$ The repeated observations give [-1, −1, 1, 1]. **Step 9: subtract the training mean from the new observation.** $$ 9-5=4 $$ **Step 10: divide by the training standard deviation.** $$ z_{\mathrm{new}}=\frac42=2 $$ **Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy. ### Standardization with x = 3 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.** ### Given x = 3; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=3-5=-2 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{-2}{2}=-1 $$ **Step 3: interpret sign and size.** The value is 1 standard-deviation unit(s) below the training mean. **Final answer:** z = -1. Use the same training statistics when transforming a new test input. ### Standardization with x = 5 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.** ### Given x = 5; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=5-5=0 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{0}{2}=0 $$ **Step 3: interpret sign and size.** The value is 0 standard-deviation unit(s) away from the training mean. **Final answer:** z = 0. Use the same training statistics when transforming a new test input. ### Standardization with x = 9 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.** ### Given x = 9; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=9-5=4 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{4}{2}=2 $$ **Step 3: interpret sign and size.** The value is 2 standard-deviation unit(s) above the training mean. **Final answer:** z = 2. Use the same training statistics when transforming a new test input. --- ## MT2 · Power, constant and sum derivatives Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/3 Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization. $$ \frac{d}{dx}(cx^n)=cnx^{n-1} $$ $$ \frac{d}{dx}c=0 $$ $$ (f+g)\prime=f\prime+g\prime $$ ## Derivatives without a calculus course **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector. $$ \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix} $$ The rules needed for the supplied neural-network numericals are: | Function or operation | Derivative | | --- | --- | | Constant c | 0 | | xⁿ | nxⁿ⁻¹ | | c f(x) | c f′(x) | | f(x) + g(x) | f′(x) + g′(x) | | f(x)g(x) | f′(x)g(x) + f(x)g′(x) | | eˣ | eˣ | | ln x, x > 0 | 1/x | | f(g(x)) | f′(g(x))g′(x) | ### A parameter derivative is not an input derivative For z = wx + b, the variable named in the denominator determines which derivative is required. $$ \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1 $$ At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities. ### Gradient direction For a small parameter displacement Δθ, first-order Taylor expansion gives: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large. ### Differentiate before substituting: 3w^2+5 **Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.** ### Given J(w) = 3w^2+5; w = 2. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(2)=3(2)^2+5=17 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=3(2)w^{2-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(2)=3(2)(2)^{1}=12 $$ **Final answer:** J = 17; derivative = 12. ### Differentiate before substituting: 0.7w^3+0 **Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.** ### Given J(w) = 0.7w^3+0; w = 1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(1)=0.7(1)^3+0=0.7 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=0.7(3)w^{3-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(1)=0.7(3)(1)^{2}=2.1 $$ **Final answer:** J = 0.7; derivative = 2.1. ### Differentiate before substituting: 2w^1+4 **Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.** ### Given J(w) = 2w^1+4; w = -1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(-1)=2(-1)^1+4=2 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=2(1)w^{1-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(-1)=2(1)(-1)^{0}=2 $$ **Final answer:** J = 2; derivative = 2. --- ## MT2 · Chain rule Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/4 The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ ## The chain rule is multiplying linked effects **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions. ### First understand why the rates multiply Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency. The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w. **Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add. ### Chain-rule example 1: two linear links **Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.** ### Given w = 1; u = 3w+1; v = 2u. ### To find The two function values and the rate dv/dw. ### Formula — which one and why v depends on w through u. Both links must be included, so multiply the two local derivatives. du/dw is the slope of 3w+1. dv/du is the slope of 2u. $$ \frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw} $$ **Yaad rakhne ka cue:** Andar ka slope × bahar ka slope. ### Solution — step by step **Step 1: calculate the intermediate value.** $$ u=3(1)+1=4 $$ **Step 2: calculate the final value.** $$ v=2(4)=8 $$ **Step 3: differentiate the first link.** $$ \frac{du}{dw}=3 $$ **Step 4: differentiate the second link.** $$ \frac{dv}{du}=2 $$ **Step 5: multiply along the path.** $$ \frac{dv}{dw}=2(3)=6 $$ **Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different. ### 2. One output: calculate both parameter gradients **Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1. ### Given Input x = 2; target y = 4. Old weight w = 1; old bias b = 0. Linear prediction z = wx+b; loss L = ½(z−y)². Learning rate η = 0.1; update both parameters once. ### To find Find the loss gradient for w and for b, then w\_new and b\_new. Finally evaluate the changed prediction and loss. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ![The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.](https://deep-learning-study-room.vercel.app/assets/figures/chain_linear_revision.svg) Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1). The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient. Tap the diagram to enlarge. **Step 1 — evaluate the original model.** Use the given parameters before applying any update. $$ z=wx+b=1(2)+0=2 $$ $$ e=z-y=2-4=-2 $$ $$ L=\frac12e^2=\frac12(-2)^2=2 $$ Forward pass: three distinct quantities Prediction z**2**1 × 2 + 0 → Error e**−2**Prediction 2 − target 4 → Half squared loss**2**½ × (−2)² The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide. **Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it. $$ \frac{\partial L}{\partial e}=\frac12(2e)=e $$ $$ \frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1 $$ $$ \delta_z=\frac{\partial L}{\partial z}=e(1)=-2 $$ The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection. **Step 3 — differentiate the neuron's affine expression.** When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1. $$ \frac{\partial z}{\partial w}=x=2 $$ $$ \frac{\partial z}{\partial b}=1 $$ **Step 4 — multiply along each parameter's path.** $$ g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4 $$ $$ g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2 $$ Backward pass: a separate derivative for each parameter Shared loss derivative δz**−2**Both parameter paths start here. Weight path: multiply by x = 2**−4**∂L/∂w = (−2) × 2 Bias path: multiply by 1**−2**∂L/∂b = (−2) × 1 Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate. **Step 5 — update both parameters simultaneously.** Gradient descent subtracts the learning rate times the corresponding gradient. $$ w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4 $$ $$ b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2 $$ One simultaneous parameter update Weight**1 → 1.4**Subtract 0.1 × (−4) Bias**0 → 0.2**Subtract 0.1 × (−2) Both updates use gradients from the same original forward pass. Re-evaluate the model afterward. Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure. **Step 6 — perform a new forward pass to evaluate the update.** $$ z_{\mathrm{new}}=1.4(2)+0.2=3 $$ $$ L_{\mathrm{new}}=\frac12(3-4)^2=0.5 $$ **Final answer:** weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3. ### 3. Two outputs: why backward contributions are added A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward. **Question.** A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed. ### Given h = wx+b; outputs o₁ = 2h and o₂ = −h. x = 2; w = 0.5; b = 0.1; targets = [1,0]. Loss = half the sum of the two squared output errors. Output multipliers 2 and −1 stay fixed. ### To find Find total loss and its derivatives with respect to h, w and b. ### Formula — which one and why **Chain rule with more than one path.** The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b. oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h} $$ $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h} $$ **Yaad rakhne ka cue:** Har raaste ka effect pehle multiply, junction par sab add. ### Solution — step by step ![The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.](https://deep-learning-study-room.vercel.app/assets/figures/chain_branch_revision.svg) Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5 The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias. Tap the diagram to enlarge. **Step 1 — forward values.** $$ h=0.5(2)+0.1=1.1 $$ $$ o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1 $$ $$ L_1=\frac12(2.2-1)^2=0.72 $$ $$ L_2=\frac12(-1.1-0)^2=0.605 $$ $$ L=L_1+L_2=0.72+0.605=1.325 $$ **Step 2 — output derivatives.** Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output. $$ \frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2 $$ $$ \frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1 $$ **Step 3 — calculate the contribution of each complete path.** The derivative of 2h with respect to h is 2; the derivative of −h is −1. $$ \frac{\partial L_1}{\partial h}=(1.2)(2)=2.4 $$ $$ \frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1 $$ **Step 4 — add the two contributions at the shared hidden value.** $$ \frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h} $$ $$ \frac{\partial L}{\partial h}=2.4+1.1=3.5 $$ Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path. **Step 5 — continue to the hidden weight and bias.** Since h = wx + b, its derivatives with respect to w and b are x and 1. $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7 $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5 $$ **Final answer:** L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation. ### Chain-rule example 4: a square after a linear link **Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.** ### Given w = 1; z = 3w+1; L = z². ### To find z, L and the derivative of L with respect to w. ### Formula — which one and why The square acts on z, not directly on w. Differentiate the square first and include the change of z with w. dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ $$ \frac{d(z^2)}{dz}=2z $$ **Yaad rakhne ka cue:** Square ka 2z; andar ka 3; dono multiply. ### Solution — step by step **Step 1: evaluate the inner expression.** $$ z=3(1)+1=4 $$ **Step 2: evaluate the loss expression.** $$ L=4^2=16 $$ **Step 3: differentiate the square and use z = 4.** $$ \frac{dL}{dz}=2z=2(4)=8 $$ **Step 4: differentiate the inner expression.** $$ \frac{dz}{dw}=3 $$ **Step 5: multiply, because the changes occur in sequence.** $$ \frac{dL}{dw}=8(3)=24 $$ **Final answer:** z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. --- ## MT2 · Dot product and a neuron Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/5 A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions. $$ z=\sum_i w_i x_i+b $$ $$ a=f(z) $$ ## Model answer: explain an artificial neuron ## Question **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example. ## Answer — step by step An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers. ![Follow each input through its weight, add one bias at the receiving neuron, then apply activation.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Follow each input through its weight, add one bias at the receiving neuron, then apply activation. Tap the diagram to enlarge. ### Formula and symbols $$ z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z) $$ | Part | Meaning | What it does | | --- | --- | --- | | x₁, x₂, … | Input features | Supply the information for this calculation | | w₁, w₂, … | Weights | Control the contribution of each matching input | | b | Bias | Shifts the weighted sum independently of the inputs | | z | Net input | The number before activation | | f | Activation rule | Transforms z, often nonlinearly | | a | Activated output | The number sent onward | ### Worked ReLU neuron example **Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU. ### To find Preactivation z and activated output a. ### Formula — which one and why Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total. z means before activation; a means after activation. $$ z=w_1x_1+w_2x_2+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Product, product, sum, bias, phir activation. ### Solution — step by step **Step 1: multiply the first pair.** $$ 0.5(2)=1 $$ **Step 2: multiply the second pair.** $$ (-0.2)(1)=-0.2 $$ **Step 3: add both contributions.** $$ 1+(-0.2)=0.8 $$ **Step 4: add the bias once.** $$ z=0.8+0.1=0.9 $$ **Step 5: apply ReLU.** $$ a=\max(0,0.9)=0.9 $$ **Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged. The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once. ## What one artificial neuron does **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output. ### What goes into a neuron? An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive. The **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**. **Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output. ### Mathematical model and notation $$ z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b $$ $$ a=f(z) $$ | Symbol | Meaning and role | | --- | --- | | xᵢ | Input component i; the neuron receives d such components | | wᵢ | Trainable weight multiplying that input component | | b | Trainable bias added once to this neuron's total | | z | Preactivation: the result before applying the activation function | | f | Chosen activation function, such as sigmoid or ReLU | | a | Activation: the neuron's output after applying f | The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs. ### Why both a bias and an activation are needed The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin. The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values. ![Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron. Tap the diagram to enlarge. ### Worked numerical **Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output. ### Given Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5]. Bias = 0.1; activation = sigmoid. ### To find Find the preactivation z, then the sigmoid output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. ### From one neuron to a layer With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron: $$ z=Wx+b,\qquad a=f(z) $$ Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together. ### Neuron example 2: ReLU output **Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2]. Weights = [3]; bias = 1. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(3)=6 $$ **Step 2: add the input contributions.** $$ s=(6)=6 $$ **Step 3: add the bias once.** $$ z=6+(1)=7 $$ **Step 4: apply the stated activation.** $$ a=\max(0,7)=7 $$ **Final answer:** z = 7; a = 7. ### Neuron example 3: ReLU output **Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.** ### Given Inputs = [2, 1]. Weights = [-1, 0.5]; bias = 0.2. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(-1)=-2 $$ **Step 2: multiply input 2 by its own weight.** $$ (1)(0.5)=0.5 $$ **Step 3: add the input contributions.** $$ s=(-2)+(0.5)=-1.5 $$ **Step 4: add the bias once.** $$ z=-1.5+(0.2)=-1.3 $$ **Step 5: apply the stated activation.** $$ a=\max(0,-1.3)=0 $$ **Final answer:** z = -1.3; a = 0. ### Neuron example 4: sigmoid output **Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.** ### Given Inputs = [1, 2]. Weights = [0.2, -0.1]; bias = 0. Activation = sigmoid. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (1)(0.2)=0.2 $$ **Step 2: multiply input 2 by its own weight.** $$ (2)(-0.1)=-0.2 $$ **Step 3: add the input contributions.** $$ s=(0.2)+(-0.2)=0 $$ **Step 4: add the bias once.** $$ z=0+(0)=0 $$ **Step 5: apply the stated activation.** $$ a=\frac1{1+e^{0}}=\frac1{1+1}=0.5 $$ **Final answer:** z = 0; a = 0.5. ## Drill 4 Dot product and bias ## Question **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU. ## Answer — step by step ### Given Inputs x = [2, −1, 0.5]. Weights w = [0.3, 0.4, −0.2]; bias b = 0.1. Activation: ReLU. ### To find Find the weighted sum z and ReLU output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero. $$ z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z) $$ Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output. ### Step 1: multiply the matching pairs $$ w_1x_1=0.3\times2=0.6 $$ $$ w_2x_2=0.4\times(-1)=-0.4 $$ $$ w_3x_3=(-0.2)\times0.5=-0.1 $$ ### Step 2: add all products and add bias once $$ z=0.6+(-0.4)+(-0.1)+0.1 $$ $$ z=0.2-0.1+0.1=0.2 $$ ### Step 3: apply the activation $$ a=\max(0,0.2)=0.2 $$ **Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive. --- ## MT2 · Matrix multiplication Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/6 Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias. $$ (AB)_{ij}=\sum_k A_{ik}B_{kj} $$ ## Matrix multiplication: every row has a job ### Row, column, vector and matrix — read these first A **row** runs horizontally, from left to right. A **column** runs vertically, from top to bottom. A **matrix** is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows. A **vector** is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors. ### Transpose: rows become columns The superscript T means **transpose**, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m. $$ [A^T]_{ij}=A_{ji} $$ The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1. **Yaad rakhne ka cue:** transpose = row ko column bana; number aur sign ko haath mat laga. ### Transpose example 1 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}2&1&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 1 row(s) and 3 column(s), so its size is 1 × 3. **Step 2: move original row 1 into new column 1.** Copy [2, 1, 3] down column 1, in the same order. **Step 3: write the transposed matrix.** $$ A^T=\begin{bmatrix}2\\1\\3\end{bmatrix} $$ **Step 4: state its new size.** The result has 3 row(s) and 1 column(s): 3 × 1. **Final answer:** Aᵀ has size 3 × 1. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 1 rows × 3 columns A: 1 rows × 3 columns | | | | | --- | --- | --- | | 2 | 1 | 3 | Read each colored row left to right. Aᵀ: 3 rows × 1 columns Aᵀ: 3 rows × 1 columns | | | --- | | 2 | | 1 | | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 2 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 3 column(s), so its size is 2 × 3. **Step 2: move original row 1 into new column 1.** Copy [1, 2, 3] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4, 5, 6] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix} $$ **Step 5: state its new size.** The result has 3 row(s) and 2 column(s): 3 × 2. **Final answer:** Aᵀ has size 3 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 3 columns A: 2 rows × 3 columns | | | | | --- | --- | --- | | 1 | 2 | 3 | | 4 | 5 | 6 | Read each colored row left to right. Aᵀ: 3 rows × 2 columns Aᵀ: 3 rows × 2 columns | | | | --- | --- | | 1 | 4 | | 2 | 5 | | 3 | 6 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 3 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&-2\\0&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 2 column(s), so its size is 2 × 2. **Step 2: move original row 1 into new column 1.** Copy [1, -2] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [0, 3] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix} $$ **Step 5: state its new size.** The result has 2 row(s) and 2 column(s): 2 × 2. **Final answer:** Aᵀ has size 2 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 2 columns A: 2 rows × 2 columns | | | | --- | --- | | 1 | −2 | | 0 | 3 | Read each colored row left to right. Aᵀ: 2 rows × 2 columns Aᵀ: 2 rows × 2 columns | | | | --- | --- | | 1 | 0 | | −2 | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 4 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}-1\\4\\2\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 3 row(s) and 1 column(s), so its size is 3 × 1. **Step 2: move original row 1 into new column 1.** Copy [-1] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4] down column 2, in the same order. **Step 4: move original row 3 into new column 3.** Copy [2] down column 3, in the same order. **Step 5: write the transposed matrix.** $$ A^T=\begin{bmatrix}-1&4&2\end{bmatrix} $$ **Step 6: state its new size.** The result has 1 row(s) and 3 column(s): 1 × 3. **Final answer:** Aᵀ has size 1 × 3. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 3 rows × 1 columns A: 3 rows × 1 columns | | | --- | | −1 | | 4 | | 2 | Read each colored row left to right. Aᵀ: 1 rows × 3 columns Aᵀ: 1 rows × 3 columns | | | | | --- | --- | --- | | −1 | 4 | 2 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Dot product: multiply matching entries, then add A **dot product** turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed. $$ w^Tx=w_1x_1+w_2x_2+w_3x_3 $$ This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products. ### Dot-product example 1 **Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.** ### Given Weights w = [2, 3]. Matching inputs x = [4, 1]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (2)(4)=8 $$ **Step 2: multiply matching pair 2.** $$ (3)(1)=3 $$ **Step 3: add the products.** $$ w^Tx=(8)+(3)=11 $$ **Final answer:** wᵀx = 11. ### Dot-product example 2 **Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.** ### Given Weights w = [1, -2]. Matching inputs x = [2, 3]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (1)(2)=2 $$ **Step 2: multiply matching pair 2.** $$ (-2)(3)=-6 $$ **Step 3: add the products.** $$ w^Tx=(2)+(-6)=-4 $$ **Final answer:** wᵀx = -4. ### Dot-product example 3 **Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.** ### Given Weights w = [0.5, 1, -1]. Matching inputs x = [4, 2, 3]. Both lists contain 3 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (0.5)(4)=2 $$ **Step 2: multiply matching pair 2.** $$ (1)(2)=2 $$ **Step 3: multiply matching pair 3.** $$ (-1)(3)=-3 $$ **Step 4: add the products.** $$ w^Tx=(2)+(2)+(-3)=1 $$ **Final answer:** wᵀx = 1. ### Matrix multiplication: each row makes one output To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B. $$ C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj} $$ For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once. **Yaad rakhne ka cue:** multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column. **Definition.** A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation. $$ z=Wx+b $$ For d input features and m receiving neurons, the dimensions are: $$ W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1} $$ The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product. ![Each row of the weight matrix calculates one receiving neuron's preactivation.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge. **Question.** Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b. ### Given Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ. ### To find Find both components of z = Wx+b. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. ### Solution — step by step **Step 1: verify compatible dimensions.** W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column. **Step 2: calculate the first receiving row products.** $$ 1(2)=2 $$ $$ (-2)(1)=-2 $$ **Step 3: sum row 1 and add its bias.** $$ z_1=2+(-2)+0.1=0.1 $$ **Step 4: calculate the second receiving row products.** $$ 0.5(2)=1 $$ $$ 3(1)=3 $$ **Step 5: sum row 2 and add its bias.** $$ z_2=1+3-0.2=3.8 $$ $$ z_1=1(2)+(-2)(1)+0.1=0.1 $$ Row 1 produces output component z1 1. Select weight row 1 1. Select weight row 1 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 2 | −2 | First × first; second × second. Sum the products**0** + Bias for this row**0.1** = z1**0.1** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z_2=0.5(2)+3(1)-0.2=3.8 $$ Row 2 produces output component z2 1. Select weight row 2 1. Select weight row 2 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 1 | 3 | First × first; second × second. Sum the products**4** + Bias for this row**−0.2** = z2**3.8** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z=\begin{bmatrix}0.1\\3.8\end{bmatrix} $$ An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W. ### Two output columns **Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.** ### Given $$ A=\begin{bmatrix}1&2\\3&4\end{bmatrix} $$ $$ B=\begin{bmatrix}2&0\\1&3\end{bmatrix} $$ ### To find The 2 × 2 matrix C = AB. ### Formula — which one and why Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication. Cᵢⱼ is the entry in output row i, column j. $$ C_{ij}=\sum_k A_{ik}B_{kj} $$ **Yaad rakhne ka cue:** Output ki position dekho: wahi row, wahi column. ### Solution — step by step **Step 1: check the dimensions.** (2 × 2)(2 × 2) is valid; output size = 2 × 2. **Step 2: row 1 with column 1.** $$ C_{11}=1(2)+2(1)=2+2=4 $$ **Step 3: row 1 with column 2.** $$ C_{12}=1(0)+2(3)=0+6=6 $$ **Step 4: row 2 with column 1.** $$ C_{21}=3(2)+4(1)=6+4=10 $$ **Step 5: row 2 with column 2.** $$ C_{22}=3(0)+4(3)=0+12=12 $$ **Step 6: assemble the output by position.** $$ C=\begin{bmatrix}4&6\\10&12\end{bmatrix} $$ **Final answer:** C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2. ### Two products that must not be confused **Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.** ### Given a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5. ### To find The 2 × 1 entry-by-entry product and the 2 × 2 outer product. ### Formula — which one and why ⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons. ⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row. $$ (a\odot b)_i=a_i b_i $$ $$ (ab^T)_{ij}=a_i b_j $$ **Yaad rakhne ka cue:** ⊙ mein same jagah; outer product mein har pair. ### Solution — step by step **Step 1: multiply matching entries.** $$ a\odot b=\begin{bmatrix}8\\15\end{bmatrix} $$ **Step 2: transpose b.** $$ b^T=\begin{bmatrix}4&5\end{bmatrix} $$ **Step 3: form row 1 of the outer product.** $$ 2(4)=8,\qquad2(5)=10 $$ **Step 4: form row 2 of the outer product.** $$ 3(4)=12,\qquad3(5)=15 $$ **Step 5: write both rows.** $$ ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix} $$ **Final answer:** Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ . ## Drill 5 Matrix rows receive inputs ## Question **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ed1fcbfdf5.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ. Activation: ReLU on each result. ### To find Find Wx+b and the activated output. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formula Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row. $$ z=Wx+b,\qquad a_j=\max(0,z_j) $$ $$ W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix} $$ ### Step 1: calculate receiving neuron 1 from row 1 $$ z_1=(1\times2)+(2\times1)+0.5 $$ $$ z_1=2+2+0.5=4.5 $$ ### Step 2: calculate receiving neuron 2 from row 2 $$ z_2=((-1)\times2)+(3\times1)+(-0.5) $$ $$ z_2=-2+3-0.5=0.5 $$ ### Step 3: apply ReLU to each entry $$ a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5 $$ $$ z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix} $$ The shape is 2 rows by 1 column: one output for each receiving row. --- ## MT2 · Layer forward pass Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/7 A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f(z^{(l)}) $$ ## Forward propagation through multiple neurons **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations. ### Input, hidden and output layers The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task. In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections. ![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg) Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights. A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge. ### General layer equations and dimensions Let a⁽⁰⁾ = x. For layer l: $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f^{(l)}(z^{(l)}) $$ If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j. $$ W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1} $$ For one hidden layer, these computations compose to give: $$ h=f^{(1)}(W^{(1)}x+b^{(1)}) $$ $$ \hat y=f^{(2)}(W^{(2)}h+b^{(2)}) $$ The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer. ### Forward propagation versus training Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture. ### Complete two-layer numerical **Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.** ### Given A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. ### To find Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg) Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge. **Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output. **Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias. $$ z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3 $$ **Step 3: calculate the second hidden preactivation.** $$ z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5 $$ **Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them. $$ a^{(1)}_1=\max(0,0.3)=0.3 $$ $$ a^{(1)}_2=\max(0,0.5)=0.5 $$ $$ a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix} $$ **Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs. $$ z^{(2)}=0.5(0.3)+0.6(0.5)+0.1 $$ $$ z^{(2)}=0.15+0.30+0.10=0.55 $$ **Step 6: apply the output activation.** $$ \hat y=\max(0,0.55)=0.55 $$ **Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass. ## Model answer: forward propagation, backpropagation and updating ## Question **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations. ## Answer — step by step Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs. ![Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge. | Operation | Direction or action | Result | | --- | --- | --- | | Forward propagation | Inputs move through successive layers | Predictions and intermediate activations | | Loss calculation | Compare predictions with targets | A scalar objective for the example or batch | | Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters | | Optimizer update | Use those gradients to change parameters | A new set of weights and biases | ### 1. Forward propagation At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ ### 2. Loss Use the loss specified by the task. For one regression output, half squared loss is one possible choice: $$ L=\frac12(a-y)^2 $$ Here a is the prediction and y is the target. Other stated losses produce different derivatives. ### 3. Backpropagation Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it. ### 4. Update and repeat $$ w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w} $$ Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 11 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 12 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## MT2 · Sigmoid and its derivative Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/8 Sigmoid takes a raw score and changes it to a number between 0 and 1. Its derivative is a different number: it tells backpropagation how sensitive that output is to its raw score. The complete example below calculates both and shows where each is used. $$ \sigma(z)=\frac{1}{1+e^{-z}} $$ $$ \sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ### sigmoid example 1 **Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.** ### Given z = -2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(-2)=2 $$ **Step 2: calculate the exponential.** $$ e^{2}\approx7.389056099 $$ **Step 3: add one in the denominator.** $$ 1+7.389056099\approx8.389056099 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: use the activated output in the slope.** $$ f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585 $$ **Final answer:** a ≈ 0.119202922. Derivative ≈ 0.104993585. ### sigmoid example 2 **Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.** ### Given z = 0; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: add one in the denominator.** $$ 1+1.000000000\approx2.000000000 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: use the activated output in the slope.** $$ f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000 $$ **Final answer:** a ≈ 0.500000000. Derivative ≈ 0.250000000. ### sigmoid example 3 **Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.** ### Given z = 2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(2)=-2 $$ **Step 2: calculate the exponential.** $$ e^{-2}\approx0.135335283 $$ **Step 3: add one in the denominator.** $$ 1+0.135335283\approx1.135335283 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: use the activated output in the slope.** $$ f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585 $$ **Final answer:** a ≈ 0.880797078. Derivative ≈ 0.104993585. --- ## MT2 · Tanh and its derivative Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/9 Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input. $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ \tanh\prime(z)=1-\tanh^2(z) $$ **Question. Calculate tanh(0.5) and its derivative with respect to the input.** ### Given z = 0.5. ### To find Find tanh(z) and its derivative. ### Formula — which one and why **Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a². z = value entering tanh; a = its output. A prime means derivative, or local rate of change. $$ a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}} $$ $$ \frac{da}{dz}=1-a^2 $$ **Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square. ### Solution — step by step **Step 1: identify the input.** The input z is 0.5. In this version of the tanh formula, the exponential uses 2z. $$ 2z=2(0.5)=1,\qquad e^{2z}=e^1\approx2.718281828 $$ **Step 2: calculate the numerator.** Subtract 1 from that exponential. $$ e^{2z}-1=2.718281828-1=1.718281828 $$ **Step 3: calculate the denominator.** Add 1 to the same exponential. $$ e^{2z}+1=2.718281828+1=3.718281828 $$ **Step 4: divide the top by the bottom.** $$ a=\tanh(0.5)=\frac{1.718281828}{3.718281828}\approx0.462117157 $$ **Step 5: calculate the local derivative.** Square the output a and subtract it from 1. Keep the calculator's unrounded result. $$ a^2\approx0.213552267,\qquad1-a^2\approx0.786447733 $$ **Final answer:** tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward. ### tanh example 1 **Question. For preactivation z = -2, calculate the tanh output and state its derivative.** ### Given z = -2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(-2)=-4 $$ **Step 2: calculate the exponential.** $$ e^{-4}\approx0.018315639 $$ **Step 3: calculate numerator and denominator separately.** $$ 0.018315639-1\approx-0.981684361 $$ $$ 0.018315639+1\approx1.018315639 $$ **Step 4: divide.** $$ a=\frac{-0.981684361}{1.018315639}\approx-0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(-2)=1-(-0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ -0.964027580. Derivative ≈ 0.070650825. ### tanh example 2 **Question. For preactivation z = 0, calculate the tanh output and state its derivative.** ### Given z = 0; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: calculate numerator and denominator separately.** $$ 1.000000000-1\approx0.000000000 $$ $$ 1.000000000+1\approx2.000000000 $$ **Step 4: divide.** $$ a=\frac{0.000000000}{2.000000000}\approx0.000000000 $$ **Step 5: square the activated output for its slope.** $$ f\prime(0)=1-(0.000000000)^2\approx1.000000000 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 1.000000000. ### tanh example 3 **Question. For preactivation z = 2, calculate the tanh output and state its derivative.** ### Given z = 2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(2)=4 $$ **Step 2: calculate the exponential.** $$ e^{4}\approx54.598150033 $$ **Step 3: calculate numerator and denominator separately.** $$ 54.598150033-1\approx53.598150033 $$ $$ 54.598150033+1\approx55.598150033 $$ **Step 4: divide.** $$ a=\frac{53.598150033}{55.598150033}\approx0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(2)=1-(0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ 0.964027580. Derivative ≈ 0.070650825. --- ## MT2 · ReLU and its derivative Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/10/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/10 ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side. $$ f(z)=\max(0,z) $$ $$ f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### ReLU example 1 **Question. For preactivation z = -2, calculate the ReLU output and state its derivative.** ### Given z = -2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 0.000000000. ### ReLU example 2 **Question. For preactivation z = 0, calculate the ReLU output and state its derivative.** ### Given z = 0; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### ReLU example 3 **Question. For preactivation z = 2, calculate the ReLU output and state its derivative.** ### Given z = 2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. --- ## MT2 · Leaky ReLU Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/11/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/11 Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function. $$ f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### Leaky ReLU example 1 **Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = -2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=-0.2 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0.1 $$ **Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000. ### Leaky ReLU example 2 **Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 0; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### Leaky ReLU example 3 **Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. --- ## MT2 · Stable softmax Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/12/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/12 Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ $$ c=\max_j z_j $$ ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. ### Softmax for [0, 0, 0] **Question. Find stable softmax probabilities for logits [0, 0, 0].** ### Given Logits = [0, 0, 0]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}0&0&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{0}\approx1.000000000 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{0}\approx1.000000000 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=1.000000000+1.000000000+1.000000000\approx3.000000000 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding. ### Softmax for [1, 2, 3] **Question. Find stable softmax probabilities for logits [1, 2, 3].** ### Given Logits = [1, 2, 3]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}-2&-1&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{-2}\approx0.135335283 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{-1}\approx0.367879441 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=0.135335283+0.367879441+1.000000000\approx1.503214724 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{0.135335283}{1.503214724}\approx0.090030573 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{0.367879441}{1.503214724}\approx0.244728471 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{1.503214724}\approx0.665240956 $$ **Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding. --- ## MT2 · Squared and absolute losses Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/13/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/13 SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate. $$ \mathrm{SSE}=\sum_i(o_i-y_i)^2 $$ $$ L_{\mathrm{half}}=\frac{\mathrm{SSE}}2 $$ $$ L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n $$ $$ L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i| $$ ## Loss: how one wrong prediction becomes a number **Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative. $$ e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2 $$ For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective. ### Derivatives and their scaling When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_j}=2e_j,\qquad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_j}=e_j,\qquad \frac{\partial\mathrm{MSE}}{\partial\hat y_j}=\frac{2}{n}e_j $$ The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight. **Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.** ### Given Predictions = [0.2,0.7,0.4]. Targets = [0,1,1]; n = 3 scalar predictions. ### To find Find SSE, half SSE and MSE, then all three prediction-gradient lists. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step **Step 1: calculate the three signed prediction errors.** $$ e=[0.2-0,\ 0.7-1,\ 0.4-1]=[0.2,-0.3,-0.6] $$ **Step 2: square every error.** $$ e\odot e=[0.04,0.09,0.36] $$ **Step 3: add the three squared errors.** $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ **Step 4: apply the requested loss divisors.** $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245,\qquad \mathrm{MSE}=\frac{0.49}{3}\approx0.163333333 $$ **Step 5: multiply every error by 2 for the SSE gradient.** $$ \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2] $$ **Step 6: use each error itself for the half-SSE gradient.** $$ \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6] $$ **Step 7: multiply every error by 2/3 for the MSE gradient.** $$ \nabla_{\hat y}\mathrm{MSE}=\frac23[0.2,-0.3,-0.6]=[0.133333333,-0.2,-0.4] $$ The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate. ## Drill 2 Half sum versus mean ## Question **Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE. ## Answer — step by step ### Given Predictions = [0.2, 0.7, 0.4]. Targets = [0, 1, 1]; number of outputs n = 3. ### To find Find SSE, half SSE, MSE and the prediction gradients requested in the question. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step ### Given values and loss definitions There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i. $$ \mathrm{SSE}=\sum_{i=1}^{n}e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}2 $$ $$ \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}n $$ SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively. ### Step 1: calculate each residual and its square | Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual | | --- | --- | --- | --- | --- | | 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 | | 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 | | 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 | ### Step 2: sum the squared residuals $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel. ### Step 3: apply each requested normalization For half SSE, the denominator is 2: $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245 $$ For MSE, the denominator is the number of pairs, n = 3: $$ \mathrm{MSE}=\frac{0.49}{3}=0.163333333\ldots $$ **Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question. ### Absolute error uses a different operation **Question. Targets are [2, 4] and predictions are [1, 6]. Find mean absolute error.** ### Given Targets = [2,4]; predictions = [1,6]; n = 2. ### To find Find mean absolute error. ### Formula — which one and why **Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes. n = number of predictions. |e| means the nonnegative magnitude of error e. $$ \mathrm{MAE}=\frac1n\sum_i|\hat y_i-y_i| $$ **Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square. ### Solution — step by step The signed errors are 1 − 2 = −1 and 6 − 4 = 2. Absolute value keeps the size of a number without its sign, so their absolute errors are 1 and 2. Add them to obtain 3. There are two prediction–target pairs, so divide by 2. $$ \mathrm{MAE}=\frac{|1-2|+|6-4|}{2}=\frac{1+2}{2}=1.5 $$ **Final answer: MAE = 1.5.** Do not square the errors for MAE. --- ## MT2 · Binary cross entropy Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/14/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/14 Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### Binary cross-entropy: target 1, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.8 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.223143551)=0.223143551 $$ **Final answer:** Loss ≈ 0.223143551 nats. ### Binary cross-entropy: target 0, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 0; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ 1-p=1-0.8=0.2 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.2)\approx-1.609437912 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-1.609437912)=1.609437912 $$ **Final answer:** Loss ≈ 1.609437912 nats. ### Binary cross-entropy: target 1, predicted positive-class probability 0.5 **Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.5; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.5 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.693147181)=0.693147181 $$ **Final answer:** Loss ≈ 0.693147181 nats. --- ## MT2 · Categorical cross entropy Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/15/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/15 For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ \log_2q=\frac{\ln q}{\ln2} $$ $$ L=-\sum_i y_i\ln p_i=-\ln p_{true} $$ ## Cross-entropy for probability distributions **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities. $$ H(p,q)=-\sum_{i=1}^{K}p_i\log q_i $$ Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms. ### Logarithm base and units Natural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2: $$ \log_2 q=\frac{\ln q}{\ln 2} $$ For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities. ### A target distribution with two nonzero entries **Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.** ### Given A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms. ### To find Find cross-entropy in the requested units. ### Formula — which one and why **Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute. pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ H_2(p,q)=\frac{H(p,q)}{\ln2} $$ **Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai. ### Solution — step by step **Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded. $$ H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)] $$ **Step 2: calculate the logarithms.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: weight each negative logarithm by its target probability.** $$ -0.75\ln(0.5)\approx0.519860385 $$ $$ -0.25\ln(0.5)\approx0.173286795 $$ **Step 4: add the contributions, retaining calculator precision.** $$ H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats} $$ **Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1. $$ H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit} $$ **Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale. ### Cross-entropy, entropy and relative entropy These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p) $$ For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus. **Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.** ### Given For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats. ### To find Find entropy and KL divergence in nats. ### Formula — which one and why **Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p. p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p) $$ **Yaad rakhne ka cue:** KL = cross-entropy minus target ki own entropy. ### Solution — step by step **Step 1: calculate the target entropy.** $$ H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)] $$ $$ \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361 $$ $$ H(p)\approx0.215761554+0.346573590=0.562335145 $$ **Step 2: subtract target entropy from cross-entropy.** $$ D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036 $$ **Final answer:** H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ. ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. --- ## MT2 · Output delta for half squared loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/16/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/16 Delta is a short name for the loss derivative with respect to a neuron’s raw input z. For half squared loss, multiply prediction minus target by the activation’s derivative. The result is then multiplied by an incoming activation to obtain a weight gradient. $$ \delta^{out}=(o-y)f\prime(z) $$ ## Loss scaling: derive the gradient instead of guessing a factor ### 1. One error, three possible objectives Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square: ![Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge. $$ \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1 $$ Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below. | Stated loss over n outputs | Derivative with respect to one prediction aⱼ | | --- | --- | | SSE: sum of all squared errors | 2(aⱼ − yⱼ) | | Half SSE: half of that sum | aⱼ − yⱼ | | MSE: that sum divided by n | 2(aⱼ − yⱼ)/n | | Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n | ### 2. Why only the matching output appears Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor. Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages. ### 3. The activation contributes another factor If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore: $$ \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a) $$ For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question. ### 4. A numerical comparison For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042. The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used. ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ## Drill 11 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 12 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## MT2 · Sigmoid plus BCE; softmax plus CCE Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/17/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/17 Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect. $$ \delta^{out}=p-y $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### The multiclass version, entry by entry **Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.** ### Given Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0]. One-example categorical cross-entropy. ### To find Find all three logit derivatives. ### Formula — which one and why **Softmax with categorical cross-entropy.** For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target. zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry. $$ \frac{\partial L}{\partial z_i}=p_i-y_i $$ **Yaad rakhne ka cue:** Softmax + CCE together: p minus y. Extra sigmoid slope nahi. ### Solution — step by step A logit is the raw score before softmax. Subtract each target entry from its matching probability entry. $$ \frac{\partial L}{\partial z_1}=0.1-0=0.1 $$ $$ \frac{\partial L}{\partial z_2}=0.8-1=-0.2 $$ $$ \frac{\partial L}{\partial z_3}=0.1-0=0.1 $$ **Final answer: [0.1, −0.2, 0.1].** Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor. ### Sigmoid+BCE logit gradient: p=0.8, y=1 **Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 1; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(-1.25)(0.16)=-0.2 $$ **Final answer:** dL/dz = -0.2. No further sigmoid derivative is needed. ### Sigmoid+BCE logit gradient: p=0.8, y=0 **Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 0; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-0}{0.16}=5 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(5)(0.16)=0.8 $$ **Final answer:** dL/dz = 0.8. No further sigmoid derivative is needed. --- ## MT2 · Hidden delta, weight gradient and bias gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/18/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/18 A hidden neuron can influence several outputs. Add all those backward contributions before multiplying by its own activation derivative. For each connection, use the receiving neuron’s delta and the sending neuron’s activation. Keep the old weights unchanged until all gradients have been found. $$ \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out} $$ $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ ## More outputs and vectorized backpropagation **Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path. ### Why the hidden derivative is a sum If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives: $$ \frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2} $$ For any number of connected outputs, sum their contributions and then differentiate the hidden activation: $$ \delta_{h_j}=\left(\sum_k W_{2,kj}\delta_{o_k}\right)f'(z_{h_j}) $$ Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss. ![The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_branch_gradient.svg) Diagram: The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.. Labels: Two output paths both affect the hidden neuron | Two output paths both affect the hidden neuron | Example: hidden activation h = 0.6, using sigmoid in the hidden neuron. | h = 0.6 | Output 1 delta = 0.2 | Connecting weight = 0.4 | Output 2 delta = −0.1 | Connecting weight = 0.5 | Downstream sum = 0.2 × 0.4 + (−0.1) × 0.5 = 0.03 | Hidden delta = 0.03 × 0.6 × (1 − 0.6) = 0.0072 | Add the two downstream paths first. Then multiply by the hidden activation | derivative. The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative. Tap the diagram to enlarge. ### Matrix equations for one training sample Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂: $$ L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2 $$ $$ \delta_2=(\hat y-y)\odot f_2'(z_2) $$ $$ \frac{\partial L}{\partial W_2}=\delta_2a_1^T,\qquad\frac{\partial L}{\partial b_2}=\delta_2 $$ $$ \delta_1=(W_2^T\delta_2)\odot f_1'(z_1) $$ $$ \frac{\partial L}{\partial W_1}=\delta_1x^T,\qquad\frac{\partial L}{\partial b_1}=\delta_1 $$ Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation. | Quantity | Shape | Reason | | --- | --- | --- | | W₁ and its gradient | m × d | m receiving hidden neurons, d inputs | | W₂ and its gradient | k × m | k receiving output neurons, m hidden activations | | δ₂ | k × 1 | One derivative per output preactivation | | W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit | | δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ | ### How the loss changes the output delta For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target: $$ \delta_{\mathrm{output}}=\hat y-y $$ That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required. For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 11 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 12 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 13 A negative ReLU blocks this path ## Question **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2. ## Answer — step by step ### Given x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2. Hidden activation: ReLU; output activation: linear. Target y = 1; loss: half squared error. ### To find Find the gradients of w₁ and w₂ and explain any zero gradients. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output. $$ z_h=w_1x+b_1,\quad h=\max(0,z_h),\quad o=w_2h+b_2 $$ $$ L=\frac12(o-y)^2,\quad \delta_o=o-y $$ $$ \frac{\partial L}{\partial w_2}=\delta_o h,\quad \delta_h=\delta_o w_2\operatorname{ReLU}'(z_h),\quad \frac{\partial L}{\partial w_1}=\delta_h x $$ ### Step 1: hidden weighted sum and activation $$ z_h=(-0.4)\times2+0.1=-0.8+0.1=-0.7 $$ $$ h=\max(0,-0.7)=0 $$ Because the raw hidden score is negative, ReLU's derivative at this point is 0. ### Step 2: output prediction and output delta $$ o=0.5\times0+0.2=0.2 $$ $$ \delta_o=0.2-1=-0.8 $$ ### Step 3: output weight gradient $$ \frac{\partial L}{\partial w_2}=(-0.8)\times0=0 $$ This is zero because the input to that connection, h, is zero. ### Step 4: hidden delta and input weight gradient $$ \delta_h=(-0.8)\times0.5\times0=0 $$ $$ \frac{\partial L}{\partial w_1}=0\times2=0 $$ **Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero. --- ## MT2 · Gradient descent Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/19/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/19 A gradient tells us the local direction in which loss rises. Gradient descent subtracts a small multiple of that gradient. Learning rate η controls the size of this correction. After the first update, calculate the next gradient at the new parameter. $$ \theta_{new}=\theta-\eta g $$ $$ g=\frac{\partial J}{\partial\theta} $$ ## Reading an optimizer formula without getting lost An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated. ![Old value, current gradient and new value belong to different moments in the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. Old value, current gradient and new value belong to different moments in the update. Tap the diagram to enlarge. ### Old, current and new At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position. $$ \theta_t=\theta_{t-1}-\eta g_t $$ η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition. ### What a stored state means An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares. ### A weighted average from ordinary arithmetic If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square. $$ 0.9\times4=3.6,\qquad(1-0.9)\times9=0.9 $$ $$ s_{\mathrm{new}}=3.6+0.9=4.5 $$ The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum. ### Squared memory and signed direction For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g². ### Epsilon and hats ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second. ## The ordinary gradient descent update **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate. ### Explanation and working $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent. Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction. ![Calculate the slope first, multiply it by the learning rate, then subtract that correction.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: Calculate the slope first, multiply it by the learning rate, then subtract that correction.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. Calculate the slope first, multiply it by the learning rate, then subtract that correction. Tap the diagram to enlarge. **Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations. ### Given J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1. ### To find Differentiate J and perform two gradient-descent iterations. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ². **Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79. **Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061. **Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**. The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum. **Common error:** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients. ## Drill 16 GD variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ \theta_1=1-0.1(1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ \theta_2=0.9-0.1(0.9)=0.81 $$ **Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 17 GD variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ \theta_1=-0.5-0.1(-1.5)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ \theta_2=-0.35-0.1(-1.35)=-0.215 $$ **Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. --- ## MT2 · Batch and mini-batch gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/20/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/20 A batch is a group of training examples used together for one update. An epoch is one pass through the dataset. Averaging gradients and counting updates are different questions, so both receive complete worked examples below. $$ g_{batch}=\frac1B\sum_{i=1}^{B}g_i $$ $$ U_{epoch}=\left\lceil\frac NB\right\rceil $$ ## Model answer: batch, stochastic and mini-batch training ## Question **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration. ## Answer — step by step All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update. ![All methods visit the same six examples. Their grouping determines how often they update.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: All methods visit the same six examples. Their grouping determines how often they update.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge. A **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations. | Basis | Batch GD | Stochastic GD | Mini-batch GD | | --- | --- | --- | --- | | Examples in one gradient | Entire training dataset | One example | A smaller group of B examples | | Updates per epoch | One | N | Ceiling of N/B if the remainder is retained | | Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging | | Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation | ### Example to distinguish the three counts With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch. The basic parameter update is unchanged: $$ \theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g $$ For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples. ## A tiny dataset shows why the methods differ **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order. ### Given Model ŷ = wx; initial w = 1; η = 0.1. Samples in order: (1,2), (2,4). Mean half squared loss for the batch. ### To find Compare one batch update with a sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### One batch update **Step 1: predict sample 1 with the original weight.** $$ \hat y_1=1(1)=1 $$ **Step 2: calculate its error.** $$ e_1=1-2=-1 $$ **Step 3: multiply its error by its input.** $$ g_1=(-1)(1)=-1 $$ **Step 4: predict sample 2 with the same original weight.** $$ \hat y_2=1(2)=2 $$ **Step 5: calculate its error.** $$ e_2=2-4=-2 $$ **Step 6: multiply its error by its input.** $$ g_2=(-2)(2)=-4 $$ **Step 7: average the two gradients.** $$ g=\frac{-1+(-4)}2=-2.5 $$ **Step 8: update once.** $$ w_{\mathrm{new}}=1-0.1(-2.5)=1.25 $$ ![This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge. ### One sequential SGD epoch **Step 1: use the first sample gradient.** $$ g_1=-1 $$ **Step 2: update after the first sample.** $$ w_1=1-0.1(-1)=1.1 $$ **Step 3: predict sample 2 using the changed weight.** $$ \hat y_2=1.1(2)=2.2 $$ **Step 4: find sample 2 error.** $$ e_2=2.2-4=-1.8 $$ **Step 5: find the new sample 2 gradient.** $$ g_2=(-1.8)(2)=-3.6 $$ **Step 6: update after sample 2.** $$ w_2=1.1-0.1(-3.6)=1.46 $$ **Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples. Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim. ## Drill 14 Batch and sequential SGD ## Question **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch. ## Answer — step by step ### Given Model ŷ = wx; initial w = 0; η = 0.1. Samples in order: (x,y) = (1,1), then (2,2). Batch objective: mean half squared loss. ### To find Compare the weight after one batch update and one sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### Meaning and formulas For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample. $$ \hat y=wx,\quad L=\frac12(\hat y-y)^2,\quad g=(\hat y-y)x $$ $$ g_{\mathrm{batch}}=\frac{g_1+g_2}{2},\qquad w_{\mathrm{new}}=w-\eta g $$ **Step 1: Part A: one batch update from w = 0.** For sample (1,1), prediction = 0 × 1 = 0, so: $$ g_1=(0-1)\times1=-1 $$ For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so: $$ g_2=(0-2)\times2=-4 $$ Average these two gradients, then update once. $$ g_{\mathrm{batch}}=\frac{-1+(-4)}2=-2.5 $$ $$ w_{\mathrm{batch}}=0-0.1(-2.5)=0.25 $$ **Step 2: Part B: one SGD epoch, restarting at w = 0.** The first sample has gradient −1, just as above. $$ w_1=0-0.1(-1)=0.1 $$ For the second sample use this new weight, so prediction is: $$ \hat y_2=0.1\times2=0.2 $$ $$ g_2=(0.2-2)\times2=(-1.8)\times2=-3.6 $$ $$ w_2=0.1-0.1(-3.6)=0.1+0.36=0.46 $$ **Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates. ## Drill 15 Incomplete mini batch ## Question **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it. ## Answer — step by step ### Given N = 103 samples; B = 20 samples per mini-batch. E = 3 epochs. Compare both remainder policies: keep or drop the final 3 examples. ### To find Find total updates for full-batch GD, SGD, and both mini-batch policies. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step ### Meaning and formulas An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs. $$ U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NE $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained. ### Step 1: batch GD All 103 samples form one batch per epoch. $$ U_{\mathrm{batch}}=3\times1=3 $$ ### Step 2: SGD Each sample gets its own update. $$ U_{\mathrm{SGD}}=103\times3=309 $$ ### Step 3: mini-batches with the remainder $$ 103=5\times20+3 $$ The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give: $$ U_{\mathrm{keep}}=6\times3=18 $$ ### Step 4: mini-batches without the remainder Only the five complete batches are used each epoch. $$ U_{\mathrm{drop}}=5\times3=15 $$ **Final answer:** 3, 309, 18 and 15 updates respectively. --- ## MT2 · Momentum: signed velocity Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/21/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/21 Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ## Model answer: momentum and Nesterov ## Question **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ? ## Answer — step by step Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions. ![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge. ### Momentum formulas This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained. $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ### Nesterov's change Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look}}) $$ The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point. | Basis | Momentum | Nesterov | | --- | --- | --- | | Gradient evaluated at | Current parameter | Lookahead parameter | | Information retained | Previous velocity | Previous velocity | | Intuition | Continue useful movement | Inspect the slope where old momentum points | | Initial zero velocity | No movement contribution yet | First lookahead equals current parameter | Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently. ## Momentum remembers movement **Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions. ### Update equations and convention This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter: $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters. ![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg) Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients. Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration. **Step 1: Iteration 1: gradient, velocity, parameter.** $$ g_1=J'(1)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Iteration 2: use θ₁ and v₁.** $$ g_2=J'(0.9)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ | Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter | | --- | --- | --- | --- | --- | | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.9 | −0.18 | 0.72 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient. ### Properties and alternative notation Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective. Some texts store a positive gradient accumulator u instead: $$ u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t $$ With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions. ## Drill 18 Momentum variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ **Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 19 Momentum variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ v_2=0.9(0.15)-0.1(-1.35)=0.27 $$ $$ \theta_2=-0.35+(0.27)=-0.08 $$ **Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## MT2 · Nesterov: look-ahead gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/22/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/22 Nesterov first uses the old movement to choose a temporary look-ahead position. It measures the gradient there, calculates the new movement, then adds that movement to the original parameter. The temporary look-ahead position does not replace the original parameter in the final addition. $$ \theta_{look}=\theta+\beta v $$ $$ v_{new}=\beta v-\eta g(\theta_{look}) $$ $$ \theta_{new}=\theta+v_{new} $$ ## Nesterov looks before correcting **Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update. ### What changes relative to momentum Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured: $$ \theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look},t}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁. ![Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective. **Iteration 1 — step 1: locate the lookahead point.** $$ \theta_{\mathrm{look},1}=1+0.9(0)=1 $$ **Step 2: evaluate the gradient there.** $$ g_1=J'(1)=1 $$ **Step 3: calculate the new velocity and actual parameter.** $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Iteration 2 — step 1: use the previous actual parameter and velocity.** $$ \theta_{\mathrm{look},2}=0.9+0.9(-0.1)=0.81 $$ **Step 2: evaluate the gradient at 0.81, not at 0.9.** $$ g_2=J'(0.81)=0.81 $$ **Step 3: calculate the new velocity.** $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ **Step 4: update the old actual parameter.** $$ \theta_2=0.9+(-0.171)=0.729 $$ | Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ | | --- | --- | --- | --- | --- | --- | | 1 | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice. ## Drill 20 Nesterov variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=1+0.9(0)=1 $$ $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81 $$ $$ g_2=0.81-(0)=0.81 $$ $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ $$ \theta_2=0.9+(-0.171)=0.729 $$ **Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. ## Drill 21 Nesterov variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=-0.5+0.9(0)=-0.5 $$ $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=-0.35+0.9(0.15)=-0.215 $$ $$ g_2=-0.215-(1)=-1.215 $$ $$ v_2=0.9(0.15)-0.1(-1.215)=0.2565 $$ $$ \theta_2=-0.35+(0.2565)=-0.0935 $$ **Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. --- ## MT2 · AdaGrad Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/23/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/23 AdaGrad stores a running sum of squared gradients for each parameter. Squaring removes the sign only in this stored total. The unsquared gradient in the numerator still decides the direction of the update. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ ## AdaGrad adds every squared gradient **Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator. ### State, effective learning rate and update For one parameter θ, let gₜ be its current gradient and s₀ = 0: $$ s_t=s_{t-1}+g_t^2 $$ $$ \eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon} $$ $$ \theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t $$ s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign. ![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg) Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update. The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge. ### Complete numerical: supplied gradients **Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter. **Iteration 1 — step 1: update the accumulator.** $$ s_1=0+2^2=4 $$ **Step 2: calculate the effective learning rate.** $$ \eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05 $$ **Step 3: apply the signed gradient.** $$ \theta_1=1-0.05(2)=0.9 $$ **Iteration 2 — step 1: retain s₁ and add the new squared gradient.** $$ s_2=4+1^2=5 $$ **Step 2: calculate the new coefficient.** $$ \eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360 $$ **Step 3: update from θ₁.** $$ \theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640 $$ **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5. ### Advantage, limitation and epsilon convention Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes. ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical. ## Drill 22 AdaGrad variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_1=0+(2)^2=4 $$ $$ \sqrt{G_1}=\sqrt{4}=2 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** Add the new squared gradient to all previous squares. $$ G_2=4+(1)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=0.9-\frac{0.1\times(1)}{\sqrt{5}}=0.85527864 $$ ## Drill 23 AdaGrad variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Add the new squared gradient to all previous squares. $$ G_1=0+(-1)^2=1 $$ $$ \sqrt{G_1}=\sqrt{1}=1 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_2=1+(2)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=-0.4-\frac{0.1\times(2)}{\sqrt{5}}=-0.489442719 $$ --- ## MT2 · RMSProp Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/24/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/24 RMSProp remembers a weighted average of squared gradients. With decay 0.9, keep 90% of the old stored value and add 10% of the new squared gradient. Calculate this new stored value before taking its square root and updating the parameter. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ ## A moving average remembers the recent past **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients. $$ m_t=\beta m_{t-1}+(1-\beta)g_t,\qquad0\le\beta<1 $$ For m₀ = 0, expanding the recurrence gives: $$ m_t=(1-\beta)\sum_{k=1}^{t}\beta^{t-k}g_k $$ The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history. ### Signed gradients versus squared gradients An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form. **Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂. ### Given β = 0.9; initial m₀ = 0. Gradients in order: g₁ = 2, g₂ = −1. ### To find Find m₁ and m₂, preserving the sign of each gradient. ### Formula — which one and why **Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state. β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state. $$ m_t=\beta m_{t-1}+(1-\beta)g_t $$ **Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai. ### Solution — step by step **Step 1: find the recent-gradient coefficient.** $$ 1-0.9=0.1 $$ **Step 2: retain the old memory for step 1.** $$ 0.9(0)=0 $$ **Step 3: weight the first gradient.** $$ 0.1(2)=0.2 $$ **Step 4: add to obtain the first memory.** $$ m_1=0+0.2=0.2 $$ **Step 5: retain the first memory for step 2.** $$ 0.9(0.2)=0.18 $$ **Step 6: weight the second gradient.** $$ 0.1(-1)=-0.1 $$ **Step 7: add to obtain the second memory.** $$ m_2=0.18+(-0.1)=0.08 $$ **Final answer:** m₁ = 0.2; m₂ = 0.08. The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable. ## RMSProp remembers recent squared gradients **Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator. ### Recurrence and interpretation Using decay β and zero initial state: $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update. If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever. ![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg) Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient. Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β = 0.9. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step **Iteration 1 — step 1: calculate the squared-gradient average.** $$ s_1=0.9(0)+0.1(2^2)=0.4 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_1}=\sqrt{0.4}\approx0.632455532 $$ $$ \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766 $$ **Step 3: update the parameter.** $$ \theta_1=1-0.316227766\approx0.683772234 $$ **Iteration 2 — step 1: use s₁, not the original zero state.** $$ s_2=0.9(0.4)+0.1(1^2)=0.46 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_2}=\sqrt{0.46}\approx0.678232998 $$ $$ \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956 $$ **Step 3: update from θ₁, retaining calculator precision.** $$ \theta_2\approx0.683772234-0.147441956\approx0.536330278 $$ **Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46. ### Comparison with AdaGrad and Adam AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum. ## Drill 24 RMSProp variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(2)^2=0.4 $$ $$ \sqrt{s_1}=\sqrt{0.4}=0.632455532 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{0.4}}=0.683772234 $$ **Step 2: Iteration 2 with supplied gradient 1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.4)+0.1(1)^2=0.46 $$ $$ \sqrt{s_2}=\sqrt{0.46}=0.678232998 $$ $$ \theta_2=0.683772234-\frac{0.1\times(1)}{\sqrt{0.46}}=0.536330278 $$ ## Drill 25 RMSProp variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(-1)^2=0.1 $$ $$ \sqrt{s_1}=\sqrt{0.1}=0.316227766 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{0.1}}=-0.183772234 $$ **Step 2: Iteration 2 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.1)+0.1(2)^2=0.49 $$ $$ \sqrt{s_2}=\sqrt{0.49}=0.7 $$ $$ \theta_2=-0.183772234-\frac{0.1\times(2)}{\sqrt{0.49}}=-0.46948652 $$ --- ## MT2 · Adam: both memories Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/25/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/25 Adam keeps two separate running averages. The first remembers signed gradients and can be negative. The second remembers squared gradients and is nonnegative. These stored averages are not weights, and they are not extra gradients to multiply together. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ ## Optimizer notation: old value, memory, new value ### 1. A parameter and a stored state are different θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries. ![The time label tells you which value to use. Carry the previous memory into the current update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: The time label tells you which value to use. Carry the previous memory into the current update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. The time label tells you which value to use. Carry the previous memory into the current update. Tap the diagram to enlarge. At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values. ### 2. A supplied gradient versus a supplied objective If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead. For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64. ### 3. A squared gradient keeps size and loses sign Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square. ### 4. Square root and epsilon belong to the denominator If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero. ### 5. A table prevents mixing iterations Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1. The worked examples below use this exact order, with every state shown. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 26 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 27 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ --- ## MT2 · Adam: correction and update Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/26/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/26 Adam starts its two averages at zero. Early averages therefore contain less total weight on observed gradients. Bias correction compensates for that initialization. The power in βᵗ uses the current update number, and both averages must be corrected before the parameter update. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \widehat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ ## Adam bias correction: why the denominator is one minus beta to a power **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate. ### Explanation and working ### 1. What the moving average actually contains Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization. ![The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam_correction.svg) Diagram: The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.. Labels: Correct the early zero-initialized memory | Correct the early zero-initialized memory | Example: β₁ = 0.9, first gradient = 2, initial m = 0. | Raw memory | m₁ = 0.9 × 0 + 0.1 × | 2 = 0.2 | Correction factor | 1 − 0.9¹ = 0.1 | Corrected memory | m̂₁ = 0.2 ÷ 0.1 = 2 | The small raw value is partly caused by the starting zero. Bias correction | removes that effect. The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor. Tap the diagram to enlarge. $$ m_1=(1-\beta)g_1 $$ At the second step, substitute the first expression into the recurrence: $$ m_2=\beta(1-\beta)g_1+(1-\beta)g_2 $$ ### 2. Add the coefficients After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term: $$ \beta(1-\beta)+(1-\beta)=(1-\beta)(1+\beta)=1-\beta^2 $$ After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ The squared-gradient memory uses its own decay β₂ and the same reasoning: $$ \widehat s_t=\frac{s_t}{1-\beta_2^t} $$ ### 3. A constant-gradient example makes the purpose visible Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38. $$ \widehat m_1=\frac{0.2}{1-0.9}=2 $$ $$ \widehat m_2=\frac{0.38}{1-0.9^2}=\frac{0.38}{0.19}=2 $$ Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient. ### 4. Two betas mean two separate powers At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators. ### 5. Correct first, then combine The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows. This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 26 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 27 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ --- ## MT2 · L1 penalty and update Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/27/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/27 L1 adds a cost based on the absolute sizes of weights. Its derivative is positive for a positive weight and negative for a negative weight. The optimizer adds this penalty gradient to the data gradient before applying the usual update. $$ J=L+\lambda\sum_j|w_j| $$ $$ w_{new}=w-\eta\left[g_{data}+\lambda\operatorname{sign}(w)\right] $$ ## Model answer: regularization and the L1–L2 difference ## Question **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength. ## Answer — step by step Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost. ![Follow the stated penalty convention, including lambda and any half factor, when you form the objective.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: Follow the stated penalty convention, including lambda and any half factor, when you form the objective.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge. ### Objectives used in this course $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question. | Basis | L1 | L2 | | --- | --- | --- | | Quantity penalized | Absolute weight magnitude | Squared weight magnitude | | Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention | | Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes | | Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights | The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total. Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective. ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 29 L1 update on positive and negative weights ## Question **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero. ## Answer — step by step ### Given Old weights = [−0.5,0.3]. Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2. ### To find Find total gradients and the two updated weights. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formula The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j) $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed. **Step 1: Weight 1: old value −0.5.** $$ \lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1 $$ $$ g_{1,\mathrm{total}}=0.2+(-0.1)=0.1 $$ $$ w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52 $$ **Step 2: Weight 2: old value 0.3.** $$ \lambda\operatorname{sign}(0.3)=0.1(1)=0.1 $$ $$ g_{2,\mathrm{total}}=-0.1+0.1=0 $$ $$ w_{2,\mathrm{new}}=0.3-0.2(0)=0.3 $$ **Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero. ## Drill 28 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 41 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. --- ## MT2 · L2 penalty and update with a half Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/28/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/28 L2 adds a cost based on squared weights. Differentiating a square produces a factor of 2. If the objective includes a half, that half cancels the 2; if it does not, the 2 remains. Read the objective before choosing the gradient. $$ J=L+\frac\lambda2\sum_jw_j^2 $$ $$ w_{new}=w-\eta(g_{data}+\lambda w) $$ ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 30 L2 update ## Question **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw². ## Answer — step by step ### Given Old weights = [0.5,−0.2]; data gradients = [0.1,0.3]. λ = 0.2; η = 0.1; penalty = (λ/2)Σw². ### To find Find penalty gradients, total gradients and updated weights. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Formula and its reason The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda w_j $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ **Step 1: Weight 1.** $$ \lambda w_1=0.2\times0.5=0.1 $$ $$ g_{1,\mathrm{total}}=0.1+0.1=0.2 $$ $$ w_{1,\mathrm{new}}=0.5-0.1(0.2)=0.48 $$ **Step 2: Weight 2.** $$ \lambda w_2=0.2\times(-0.2)=-0.04 $$ $$ g_{2,\mathrm{total}}=0.3+(-0.04)=0.26 $$ $$ w_{2,\mathrm{new}}=-0.2-0.1(0.26)=-0.226 $$ **Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative. ## Drill 28 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 41 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. --- ## MT2 · Inverted dropout Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/29/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/29 Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated. $$ q=1-p $$ $$ a_{train}=\frac{m\odot a}{q} $$ $$ a_{test}=a $$ ## Model answer: dropout with training and inference explained ## Question **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling. ## Answer — step by step Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights. ![The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge. Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop. ### Inverted dropout during training $$ a'=\frac{ma}{q} $$ A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks. For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is: $$ 0.5(6)+0.5(0)=3 $$ ### Inference Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions. Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data. ## Dropout removes activations during training **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training. ### Explanation and working Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count. ![Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout.svg) Diagram: Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.. Labels: Dropout temporarily removes selected activations | Dropout temporarily removes selected activations | Example: keep probability q = 0.5. Inverted dropout during training. | Original | 0.8 | -0.4 | 0.6 | 0.2 | Mask | 1 | 0 | 1 | 0 | After mask ÷ q | 1.6 | 0 | 1.2 | 0 | The kept values double: 0.8 ÷ 0.5 = 1.6 and 0.6 ÷ 0.5 = 1.2. At inference, no | mask is used. Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge. Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions. $$ a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout} $$ $$ a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout} $$ **Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs. ### Given Activations = [0.8,−0.4,0.6,0.2]. Dropout rate = 50%, so keep probability q = 0.5. Chosen mask = [1,0,1,0]. ### To find Find training outputs under inverted dropout. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step **Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key. **Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer. Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition. ## Drill 31 Inverted dropout ## Question **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference. ## Answer — step by step ### Given Activations = [2,−1,0,3]. Keep probability q = 0.75; keep mask = [1,0,1,1]. ### To find Find training outputs and state the inference rule. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step ### Meaning and formula The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved. $$ a'_j=\frac{m_j a_j}{q} $$ The multiplication is entry by entry; there is no dot-product sum. ### Step 1: apply the mask $$ [1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3] $$ ### Step 2: divide each result by 0.75 $$ a'_1=\frac2{0.75}=2.666666667 $$ $$ a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0 $$ $$ a'_4=\frac3{0.75}=4 $$ **Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero. ### During inference Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training. ## Drill 42 Mixed dropout and a momentum update ## Question **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1. ## Answer — step by step ### Given Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5. Linear-output weights = [0.2,0.3]; bias = 0; target = 1. Half squared loss; update only the output weights. Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1. ### To find Find dropped/scaled activations, prediction, gradients, new velocities and new output weights. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Formulas used Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients. $$ a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j $$ The signed-velocity momentum convention is: $$ v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}} $$ ### Step 1: apply inverted dropout $$ a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0 $$ ### Step 2: prediction and error $$ o=0.2(2)+0.3(0)+0=0.4 $$ $$ o-y=0.4-1=-0.6 $$ ### Step 3: current weight gradients $$ g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0 $$ ### Step 4: remembered movement plus new correction $$ v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21 $$ $$ v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09 $$ ### Step 5: add velocities to old weights $$ w_{1,\mathrm{new}}=0.2+0.21=0.41 $$ $$ w_{2,\mathrm{new}}=0.3-0.09=0.21 $$ **Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero. --- ## MT2 · Convolution output dimensions Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/30/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/30 This formula counts how many legal filter positions fit along one image dimension. Padding adds space at both ends, the kernel occupies K positions, and stride controls how far the next window moves. The floor keeps only complete valid positions. $$ N_{out}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ ## Reading image shapes before doing CNN arithmetic **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension. ### Explanation and working ### Height, width and channels The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors. ![Count height, width and channels separately. Three channels are stacked at the same spatial positions.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_shape.svg) Diagram: Count height, width and channels separately. Three channels are stacked at the same spatial positions.. Labels: Read height, width and channels separately | Read height, width and channels separately | The text uses 4 × 5 × 3: four rows, five columns and three channels. | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 3 stacked channels | H × W × C | H = 4 rows | W = 5 columns | C = 3 channels | Total entries = 60 | A filter spans the incoming channels. Its spatial width is a different quantity. Count height, width and channels separately. Three channels are stacked at the same spatial positions. Tap the diagram to enlarge. ### One position versus the whole map A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels. ### Why filter depth matches input channels A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels. ### Reading a complete shape calculation The spatial-size formula computes height and width. It does not compute the channel count. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position. ### Shape, parameters and computation are three different counts **Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights. In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count. ## Stride padding and floor **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output. ### Explanation and working Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension. ![The border adds to both sides of the input. The stride changes the next starting position of the filter window.](https://deep-learning-study-room.vercel.app/assets/figures/v_stride_padding.svg) Diagram: The border adds to both sides of the input. The stride changes the next starting position of the filter window.. Labels: Padding adds a border. Stride moves the window. | Padding adds a border. Stride moves the window. | Example: a 3 × 3 input becomes 5 × 5 after one zero border. | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 3 | 0 | 0 | 4 | 5 | 6 | 0 | 0 | 7 | 8 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | Padding P = 1 | 0 | 1 | 2 | 3 | 4 | Stride S = 2 | Starting positions: 0, 2, … | For each spatial dimension: add 2P, subtract K, divide by S, take floor, then add | 1. The border adds to both sides of the input. The stride changes the next starting position of the filter window. Tap the diagram to enlarge. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: $\left\lfloor 2.5\right\rfloor$ = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels. **Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1. ### Output dimension with a fractional intermediate result **Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.** ### Given N = 6; K = 3; S = 2; P = 0; dilation = 1. ### To find The integer output side length. ### Formula — which one and why The formula counts complete kernel placements. Flooring removes an incomplete final placement. N is input side; P is padding on each side; K is kernel side; S is stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1. ### Solution — step by step **Step 1: include padding and subtract kernel size.** $$ 6+2(0)-3=3 $$ **Step 2: divide by the stride.** $$ \frac32=1.5 $$ **Step 3: take the floor.** $$ \lfloor1.5\rfloor=1 $$ **Step 4: count the first placement.** $$ N_{\mathrm{out}}=1+1=2 $$ **Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter. ### Padding on both sides **Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.** ### Given N = 5; K = 3; S = 1; P = 1 on each side. ### To find Padded input and output dimensions. ### Formula — which one and why Padding adds P entries on each of two opposite sides; the output counts complete kernel placements. N = unpadded side; K = kernel side; S = stride; P = per-side padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\lfloor(N+2P-K)/S\rfloor+1 $$ **Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain. ### Solution — step by step **Step 1: count both padding edges.** $$ 2P=2(1)=2 $$ **Step 2: calculate the padded input side.** $$ N+2P=5+2=7 $$ **Step 3: subtract kernel width.** $$ 7-3=4 $$ **Step 4: divide by stride and take the floor.** $$ \lfloor4/1\rfloor=4 $$ **Step 5: include the first placement.** $$ N_{out}=4+1=5 $$ **Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter. “Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case. ## Even-sized kernels: SAME padding is not always symmetric **Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides. For a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention. **Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho. ### Example 1: one entry on every side **Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)** ### Given Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter. ### To find The padded input dimensions and the convolution output dimensions. ### Formula — which one and why The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel. ### Solution — step by step **Step 1: count the two padding borders.** $$ 2P=2(1)=2 $$ **Step 2: find the padded side length.** $$ N_{\mathrm{padded}}=5+2=7 $$ The padded input is 7 × 7 × 1. **Step 3: subtract the kernel size.** $$ 7-2=5 $$ **Step 4: divide by stride, take the floor and count the first placement.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5}{1}\right\rfloor+1=5+1=6 $$ **Step 5: write height, width and channels.** Both spatial dimensions equal 6. There is one filter, so there is one output channel. **Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel. ### Example 2: preserve the size with SAME padding **Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)** ### Given Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found. ### To find Padding above, below, left and right; padded-input shape; final output shape. ### Formula — which one and why For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+P_{\mathrm{before}}+P_{\mathrm{after}}-K}{S}\right\rfloor+1 $$ At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1. $$ P_{\mathrm{total}}=P_{\mathrm{before}}+P_{\mathrm{after}}=K-1 $$ ### Solution — step by step **Step 1: calculate total padding along the height.** $$ P_{\mathrm{total}}=2-1=1 $$ Only one extra row in total is needed. **Step 2: place the row using the stated convention.** Padding above = 0; padding below = 1. **Step 3: apply the same calculation to width.** Only one extra column in total is needed. Padding left = 0; padding right = 1. **Step 4: find the padded-input shape.** $$ H_{\mathrm{padded}}=5+0+1=6 $$ $$ W_{\mathrm{padded}}=5+0+1=6 $$ The padded input is 6 × 6 × 1. **Step 5: count the output positions along each axis.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5+0+1-2}{1}\right\rfloor+1=4+1=5 $$ **Step 6: include the output channel.** One filter creates one output channel. **Final answer:** add **0 above, 1 below, 0 left and 1 right**; padded input = **6 × 6 × 1**; output = **5 × 5 × 1**. The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula. ## Drill 33 Padding stride and channels ## Question **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total. ## Answer — step by step ### Given Input shape = 31 × 31 × 3. Filter size = 5 × 5; number of filters = 8. Stride S = 2; padding P = 1 per side; one bias per filter. ### To find Find output shape, weights, biases and total parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning of each given value The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels. ### Formulas used $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ \text{weights}=K^2C_{\mathrm{in}}C_{\mathrm{out}},\quad \text{biases}=C_{\mathrm{out}} $$ $$ \text{parameters}=\text{weights}+\text{biases} $$ ### Step 1: find the output height and width Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33. $$ N_{\mathrm{out}}=\left\lfloor\frac{31+2(1)-5}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{28}{2}\right\rfloor+1=14+1=15 $$ The height uses the same calculation. **Output shape = 15 × 15 × 8.** ### Step 2: count weights in one filter Each filter covers 5 × 5 positions in each of 3 channels. $$ 5\times5\times3=75\text{ weights per filter} $$ ### Step 3: count all filters and their biases $$ \text{weights}=75\times8=600 $$ $$ \text{biases}=8,\qquad \text{total}=600+8=608 $$ **Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights. --- ## MT2 · Convolution value Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/31/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/31 One convolution output is a sum of matched input–kernel products. Move the window and repeat that exact rule for the next output. With several input channels, add contributions from every channel and add the filter bias once. $$ Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o $$ ## One filter over multiple channels: all contributions go into one cell **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels. ### Explanation and working ### 1. Separate spatial size from depth Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3. ![Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.](https://deep-learning-study-room.vercel.app/assets/figures/v_channel_products.svg) Diagram: Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.. Labels: A 1 × 1 filter still mixes all three channels | A 1 × 1 filter still mixes all three channels | Use input [2, 5, 1], weights [0.1, −0.2, 0.5] and one bias 0.3. | Channel 1 | 2 × 0.1 = 0.2 | Channel 2 | 5 × (−0.2) = −1 | Channel 3 | 1 × 0.5 = 0.5 | One filter → one output value | 0.2 + (−1) + 0.5 + 0.3 = 0 | One spatial position can contain several channel values. The one-by-one filter | mixes them. Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example. Tap the diagram to enlarge. $$ z=0.1(2)+(-0.2)(5)+0.5(1)+0.3 $$ $$ z=0.2-1+0.5+0.3=0 $$ The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels. ### 2. A larger filter has one spatial grid per input channel A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias. The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified. ### 3. What changes when there are several filters? Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth. For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters. ### 4. One-by-one does not mean one weight A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed. This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately. ## Faculty example: a complete binary-image convolution **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)** ### Given Image X = $\begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}$ . Filter K = $\begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}$ . These are the values in the Unit 4 lecture example. ### To find The output shape and all nine feature-map values. ### Formula — which one and why The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window. ![Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.](https://deep-learning-study-room.vercel.app/assets/figures/u4_binary_windows.svg) Diagram: Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.. Labels: u4 binary windows | One filter, nine window positions | Input: 5 × 5 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | Shared filter: 3 × 3 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | → | Output: 3 × 3 | 4 | 3 | 4 | 2 | 4 | 3 | 2 | 3 | 4 | Orange window → orange output cell = 4. Move one cell at a time. Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions. Tap the diagram to enlarge. ### Solution — step by step **Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3. **Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**. **Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**. **Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**. **Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**. **Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**. **Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**. **Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**. **Final answer:** Y = $\begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}$ . All nine outputs use the same nine filter weights. They do not introduce nine different filters. **Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain. ## Faculty example: combine three RGB channel sums **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)** ### Given Red patch = $\begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}$ ; red kernel = $\begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}$ . Green patch = $\begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}$ ; green kernel = $\begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}$ . Blue patch = $\begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}$ ; blue kernel = $\begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}$ . Bias = 1. These are three slices of **one** full-depth filter, not three independent output filters. ### To find The contribution of each input channel and the final scalar output at this position. ### Formula — which one and why Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions. $$ z=s_R+s_G+s_B+b $$ The quantities s\_R, s\_G and s\_B are intermediate weighted sums, not separate final output channels. ### Solution — step by step **Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1. **Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**. **Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1). **Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**. **Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1. **Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**. **Step 7: add the channel contributions.** 308 −498 +164 = −26. **Step 8: add one filter bias.** −26 +1 = **−25**. **Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages. **Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo. ## The convolution calculation **Definition.** A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention. ### Explanation and working In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it. **Question.** Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias. ### Given Input X = $\begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}$ . Kernel K = $\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$ . Stride = 1; padding = 0; bias = 0; no kernel flip. ### To find Find the 2 × 2 output map. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_cd39cabe48.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 0 | 0 | −1 | Matrix 2 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 Given matrices for this question Tap the diagram to enlarge. $$ X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix} $$ **Step 1. Top-left window.** 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4. **Step 2. Move right by one.** 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4. **Step 3. Move down and return left.** 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4. **Step 4. Bottom-right window.** 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is $\begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}$ . If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested. ![Four windows produce four output cells](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: Four windows produce four output cells. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. Four windows produce four output cells Tap the diagram to enlarge. ## Drill 32 Convolution with a negative coefficient ## Question **Question.** Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | --- | --- | | 1 | −1 | | 0 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel K are printed with the question. Stride S = 1; padding P = 0; bias = 0. Use the kernel without flipping it. ### To find Calculate all nine output cells. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−2) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(-1)+(1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 3 = 3** **2 + 3 = 5** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(-1)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(3)=5 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 2 = 2** **(−1) + 2 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(-1)+(2)=1 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(-1)+(1)=0 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −3 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−3) = −2** Row 2**0 + 0 = 0** **(−2) + 0 = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(-1)=-2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(-2)+(0)=-2 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + (−2) = 1** Row 2**0 + 1 = 1** **1 + 1 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(1)+(1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | −1 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + (−1) = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(1)+(0)=1 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 2 = 2** **1 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(2)=3 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 3 = 3** **(−1) + 3 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(-1)+(3)=2 $$ **Final output:** $\begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ## Drill 40 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. --- ## MT2 · Max, min and average pooling Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/32/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/32 Pooling reduces each window to one value. Max pooling keeps the largest entry; average pooling adds all included entries and divides by their count. Pooling is performed independently in each channel. $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{max}=\max_{(i,j)\in W}x_{ij} $$ $$ y_{avg}=\frac1{|W|}\sum_{(i,j)\in W}x_{ij} $$ ## Pooling needs comparisons or averages **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases. ### Explanation and working Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged. **Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding: ### Given The 4 × 4 input map is printed below. Window = 2 × 2; stride = 2; padding = 0. ### To find Find the pooled output by the operation(s) requested. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step $$ A=\begin{bmatrix}21&11&15&10\\12&9&10&11\\11&13&18&12\\7&4&15&12\end{bmatrix} $$ ### Top-left window **Step 1: select the four entries.** [21, 11, 12, 9] **Step 2: choose the largest entry.** $$ y_{\max}=21 $$ **Step 3: add all entries.** $$ 21+11+12+9=53 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{53}4=13.25 $$ ### Top-right window **Step 1: select the four entries.** [15, 10, 10, 11] **Step 2: choose the largest entry.** $$ y_{\max}=15 $$ **Step 3: add all entries.** $$ 15+10+10+11=46 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{46}4=11.5 $$ ### Bottom-left window **Step 1: select the four entries.** [11, 13, 7, 4] **Step 2: choose the largest entry.** $$ y_{\max}=13 $$ **Step 3: add all entries.** $$ 11+13+7+4=35 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{35}4=8.75 $$ ### Bottom-right window **Step 1: select the four entries.** [18, 12, 15, 12] **Step 2: choose the largest entry.** $$ y_{\max}=18 $$ **Step 3: add all entries.** $$ 18+12+15+12=57 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{57}4=14.25 $$ Outputs are max $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ and average $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . ![Each pooling window reduces several values to one](https://deep-learning-study-room.vercel.app/assets/figures/pooling.svg) Diagram: Each pooling window reduces several values to one. Labels: INPUT 4 × 4 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 | MAX POOL | 21 | 15 | 13 | 18 | AVERAGE POOL | 13.25 | 11.5 | 8.75 | 14.25 | 2 × 2 windows | stride 2 Each pooling window reduces several values to one Tap the diagram to enlarge. Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance. ## Max, min and average pooling on the same lecture grid **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases. **Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)** ### Given X = $\begin{bmatrix}12 & 20 & 30 & 0\\8 & 12 & 2 & 0\\34 & 70 & 37 & 4\\112 & 100 & 25 & 12\end{bmatrix}$ . Window = 2 × 2; stride = 2; padding = 0. ### To find The shared output shape and each of the four cells for each pooling method. ### Formula — which one and why Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap. $$ y_{\max}=\max(x_1,x_2,x_3,x_4) $$ $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{\mathrm{avg}}=\frac{x_1+x_2+x_3+x_4}{4} $$ ![The four colours identify four separate pooling windows. Each window creates one cell in each output map.](https://deep-learning-study-room.vercel.app/assets/figures/u4_pooling_windows.svg) Diagram: The four colours identify four separate pooling windows. Each window creates one cell in each output map.. Labels: u4 pooling windows | Same four windows, three different summaries | Input: 4 × 4 | 12 | 20 | 30 | 0 | 8 | 12 | 2 | 0 | 34 | 70 | 37 | 4 | 112 | 100 | 25 | 12 | Max | 20 | 30 | 112 | 37 | Min | 8 | 0 | 34 | 4 | Average | 13 | 8 | 79 | 19.5 | Bottom-right average: (37 + 4 + 25 + 12) ÷ 4 = 19.5 | Pooling changes values and shape; ordinary pooling learns no parameters. The four colours identify four separate pooling windows. Each window creates one cell in each output map. Tap the diagram to enlarge. ### Solution — step by step **Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2. **Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13. **Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8. **Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79. **Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20. **Step 6: put each result back in its window's position.** Max = $\begin{bmatrix}20 & 30\\112 & 37\end{bmatrix}$ . Min = $\begin{bmatrix}8 & 0\\34 & 4\end{bmatrix}$ . Average = $\begin{bmatrix}13 & 8\\79 & 19.5\end{bmatrix}$ . **Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation. ### Why the choice matters Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour. Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter. **Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide. ## Drill 34 Max and average pooling ## Question **Question.** Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_563e4ee8f7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | A | 1 | 4 | 2 | 3 | 0 | 2 | 5 | 1 | 6 | 3 | 2 | 0 | 1 | 7 | 4 | 8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input A is printed with the question. Window size = 2 × 2; stride = 2; no padding. ### To find Find the complete max-pooling and average-pooling outputs. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 4, 0, 2] **Step 2: add all selected entries.** 1 + 4 + 0 + 2 = 7 **Step 3: divide by the 4 entries in this window.** 7/4 = **1.75**. Maximum = **4** ### Pooling window (1,2) **Step 1: select the entries.** [2, 3, 5, 1] **Step 2: add all selected entries.** 2 + 3 + 5 + 1 = 11 **Step 3: divide by the 4 entries in this window.** 11/4 = **2.75**. Maximum = **5** ### Pooling window (2,1) **Step 1: select the entries.** [6, 3, 1, 7] **Step 2: add all selected entries.** 6 + 3 + 1 + 7 = 17 **Step 3: divide by the 4 entries in this window.** 17/4 = **4.25**. Maximum = **7** ### Pooling window (2,2) **Step 1: select the entries.** [2, 0, 4, 8] **Step 2: add all selected entries.** 2 + 0 + 4 + 8 = 14 **Step 3: divide by the 4 entries in this window.** 14/4 = **3.5**. Maximum = **8** **Average output:** $\begin{bmatrix}1.75 & 2.75\\4.25 & 3.5\end{bmatrix}$ . **Max output:** $\begin{bmatrix}4 & 5\\7 & 8\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. ## Drill 40 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. --- ## MT2 · Convolution parameter count Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/33/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/33 A filter has one learned weight for each spatial position in each input channel. Count the weights in one filter, add its one bias, then multiply by the number of filters. Sliding a filter to many positions reuses these weights; it does not create new parameters. $$ P_{conv}=(K_hK_wC_{in}+1)C_{out} $$ ## Output size and parameter count are different **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters. ### Explanation and working **Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias. ### Given Input = 7 × 7 × 3; two 3 × 3 filters. Stride = 2; padding = 1; include biases. ### To find Find the output shape and trainable parameter count. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step **Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2. ![Output size counts produced values; parameter count counts adjustable weights and biases.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Output size counts produced values; parameter count counts adjustable weights and biases.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Output size counts produced values; parameter count counts adjustable weights and biases. Tap the diagram to enlarge. **Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters. ## Model answer: shape, activations and parameters ## Question **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them. ## Answer — step by step Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions. ![Keep output shape, activation count and parameter count as three separate quantities.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Keep output shape, activation count and parameter count as three separate quantities.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Keep output shape, activation count and parameter count as three separate quantities. Tap the diagram to enlarge. ### Spatial output shape For an ordinary convolution with dilation 1: $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters. ### Output activations $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ This is the number of output values for one example, excluding any batch dimension. ### Trainable parameters $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions. ### Small example A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels. ## Faculty CNN table: shapes, activations and 62958 parameters **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)** ### Given Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question. ### To find Output shape and number of output values for each layer, its trainable parameters, and the total parameters. ### Formula — which one and why Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ A counts produced activations. P\_conv and P\_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here. ### Solution — step by step **Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0. **Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272. **Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**. **Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight. **Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600. **Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**. **Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters. **Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations. **Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations. **Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation. **Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**. | Layer | Output shape | Activations | Parameters | | --- | --- | --- | --- | | Input | 32 × 32 × 3 | 3,072 | 0 | | Conv1 | 28 × 28 × 8 | 6,272 | 608 | | Pool1 | 14 × 14 × 8 | 1,568 | 0 | | Conv2 | 10 × 10 × 16 | 1,600 | 3,216 | | Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 | | Dense 120 | 120 | 120 | 48,120 | | Dense 84 | 84 | 84 | 10,164 | | Dense 10 → softmax | 10 | 10 | 850 + 0 | **Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network. **Yaad rakho:** Shape aage carry karo; weights aur activations ko alag columns mein rakho. ## Parameters are different from activation values **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters. ### Explanation and working An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights. ![The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations. Tap the diagram to enlarge. $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation. ### One filter, then all filters **Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.** ### Given Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter. ### To find Weights, biases, total trainable parameters and output channels. ### Formula — which one and why Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights. Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters. $$ P=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply. ### Solution — step by step **Step 1: count weights in one filter.** $$ 3(3)(3)=27 $$ **Step 2: include its one bias.** $$ 27+1=28 $$ **Step 3: count all weights.** $$ 27(16)=432 $$ **Step 4: count all biases.** $$ 1(16)=16 $$ **Step 5: add and state channel count.** $$ P=432+16=448 $$ Each filter produces one channel, so output channels = 16. **Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels. $$ P_{\mathrm{pool}}=0,\quad P_{\mathrm{flatten}}=0,\quad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values. ## Drill 35 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. --- ## MT2 · Flatten and dense parameter count Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/34/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/34 Flattening rearranges all activation values into a list. It learns no new numbers. A dense layer then connects every value in that list to every receiving neuron; each receiving neuron also has one bias. $$ N_{flat}=HWC $$ $$ P_{dense}=(N_{in}+1)N_{out} $$ ## Counting parameters in a dense network **Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases. ### Deriving the parameter count For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters: $$ P_{\mathrm{dense}}=mn+n=(m+1)n $$ Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram. ![Count all input-to-neuron connections and then one bias per receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/v_dense_count.svg) Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters. Count all input-to-neuron connections and then one bias per receiving neuron. Tap the diagram to enlarge. ### Complete numerical **Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters. ### Given Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs. Every hidden/output neuron has one bias. ### To find Count weights and biases at each receiving layer, then the total. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step **Step 1: identify the consecutive layer widths.** $$ 10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4 $$ There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters. **Step 2: count weights and biases for each transformation.** | Transformation | Weights | Biases | Layer total | | --- | --- | --- | --- | | Input → hidden 1 | 10 × 5 = 50 | 5 | 55 | | Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 | | Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 | | Hidden 3 → output | 5 × 4 = 20 | 4 | 24 | **Step 3: add the layer totals.** $$ P=55+30+30+24=139 $$ Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139. **Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network. ## One complete CNN pipeline **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron. ### Given Input = 128 × 128 × 3. Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2. Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2. Flatten → dense 240 → dense 120 → dense 5. One bias per learned filter/dense neuron. ### To find Find every output shape and layer parameter count, then the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step **Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448. ![Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.](https://deep-learning-study-room.vercel.app/assets/figures/v_cnn_pipeline.svg) Diagram: Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.. Labels: Follow the complete CNN question, one layer at a time | Follow the complete CNN question, one layer at a time | Start with 128 × 128 × 3. Shape and learned-parameter count are separate. | LAYER | OUTPUT SHAPE / LENGTH | PARAMETERS | 1. Convolution 1 | 126 × 126 × 16 | 448 | 2. Pooling 1 | 63 × 63 × 16 | 0 | 3. Convolution 2 | 61 × 61 × 32 | 4,640 | 4. Pooling 2 | 30 × 30 × 32 | 0 | 5. Flatten | 28,800 | 0 | 6. Dense 240 | 240 | 6,912,240 | 7. Dense 120 | 120 | 28,920 | 8. Classifier | 5 | 605 | Total learned parameters = 6,946,853 | Each output is the next layer’s input. Pooling and flattening add no learned | parameters. Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count. Tap the diagram to enlarge. **Step 2. Pool 1 size.** $\left\lfloor\frac{126 - 2}{2}\right\rfloor+1$ = 63; output 63 × 63 × 16. Parameters = 0. **Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640. **Step 4. Pool 2 size.** $\left\lfloor\frac{61 - 2}{2}\right\rfloor+1$ = $\left\lfloor 29.5\right\rfloor$ + 1 = 30; output 30 × 30 × 32. Parameters = 0. **Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0. **Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240. **Step 7. Dense 120.** (240 + 1) × 120 = 28,920. **Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer. **Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer. ## Drill 35 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. ## Drill 43 Mixed CNN dimensions softmax and loss ## Question **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE. ## Answer — step by step ### Given Input = 8 × 8 × 3. Conv: kernel 3, stride 1, padding 1, 2 filters. Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs. Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]). ### To find Find shapes, parameter counts, probabilities and CCE. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Formulas used Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}} $$ Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.367879441, 1, 0.135335283]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Correct class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**. --- ## MT2 · Convolution multiply-accumulate count Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/35/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/35 A parameter count counts learned numbers. A multiply-accumulate count counts how many times input–weight products are used. One filter is reused at every output position, so computation includes output height and width even though its parameter count does not. $$ MAC=H_{out}W_{out}K_hK_wC_{in}C_{out} $$ ## Why one by one convolution saves work **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts. ![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg) Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications. Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge. **Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.** ### Given Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size. ### To find Direct MACs, total two-stage bottleneck MACs and percentage saved. ### Formula — which one and why Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs. ### Solution — step by step **Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400 $$ **Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels. $$ \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448 $$ **Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels. $$ \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200 $$ **Step 4: add the two bottleneck costs.** $$ \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648 $$ **Step 5: calculate the saving.** $$ \mathrm{saving}=120422400-12443648=107978752 $$ $$ \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\% $$ **Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption. ### A second bottleneck with different dimensions **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1. ### To find Direct MACs, bottleneck MACs and saving. ### Formula — which one and why Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers. ### Solution — step by step **Step 1: direct spatial convolution.** $$ 8(8)(8)(3)(3)(16)=73728 $$ **Step 2: 1 × 1 reduction.** $$ 8(8)(4)(1)(1)(16)=4096 $$ **Step 3: reduced-depth 3 × 3 convolution.** $$ 8(8)(8)(3)(3)(4)=18432 $$ **Step 4: add the stages and subtract from the direct cost.** $$ 4096+18432=22528 $$ $$ 73728-22528=51200 $$ **Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values. ## The faculty's 480-channel bottleneck numerical ![Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.](https://deep-learning-study-room.vercel.app/assets/figures/u4_bottleneck_480.svg) Diagram: Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.. Labels: u4 bottleneck 480 | Same output shape, different computation | Input | 14 × 14 × 480 | → | Direct 5 × 5 | 48 filters | → | Output | 14 × 14 × 48 | 112,896,000 MACs | 1 × 1 reduction | 480 → 16 channels | → | Reduced 5 × 5 | 16 → 48 channels | → | Output | 14 × 14 × 48 | 1,505,280 + 3,763,200 = 5,268,480 MACs | The lower path also starts from 14 × 14 × 480. Count both layers. Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers. Tap the diagram to enlarge. **Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.** ### Given Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters. ### To find MACs for the direct path, both bottleneck stages, their total and percentage saved. ### Formula — which one and why $$ M=H_oW_oC_oK^2C_i $$ M is the MAC count. H\_oW\_oC\_o counts output values; K²C\_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck. $$ \mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}} $$ ### Solution — step by step **Step 1: direct output count.** 14 × 14 × 48 = 9,408. **Step 2: products per direct output.** 5 × 5 × 480 = 12,000. **Step 3: direct MACs.** $$ 9408(12000)=112896000 $$ **Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values. $$ 14(14)(16)(480)=1505280 $$ **Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values. $$ 9408(400)=3763200 $$ **Step 6: add both bottleneck costs.** $$ 1505280+3763200=5268480 $$ **Step 7: calculate the saving.** $$ 112896000-5268480=107627520 $$ $$ 100\frac{107627520}{112896000}\approx95.333333\% $$ **Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone. ## Drill 37 Bottleneck cost with new values ## Question **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first. ## Answer — step by step ### Given Input = 16 × 16 × 64. Direct branch: 32 same-padded 3 × 3 filters. Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer. Spatial size remains 16 × 16; exclude bias additions. ### To find Find both MAC counts, absolute saving and percentage saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Meaning and formula A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ ### Step 1: direct 3 × 3 convolution Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 reduction from 64 to 8 channels $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ ### Step 3: 3 × 3 convolution now receives only 8 channels $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: add the two bottleneck stages $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: calculate the saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32. ### Bottleneck computation on an 8 × 8 input **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8 in both stages. Input channels = 16; output filters = 8; main kernel = 3 × 3. Bottleneck uses 4 filters of 1 × 1. ### To find Direct MACs, two-stage bottleneck MACs and savings. ### Formula — which one and why Each output scalar requires one multiply-accumulate per incoming kernel entry. A bottleneck reduces the input depth of the expensive spatial convolution. One MAC is one multiply-accumulate pair. If counting multiplication and addition separately, the operation count is approximately twice the MAC count. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K_hK_wC_{\mathrm{in}} $$ **Yaad rakhne ka cue:** Spatial kernel se pehle channels kam; dono stages ka cost add. ### Solution — step by step **Step 1: count output positions per channel.** $$ 8(8)=64 $$ **Step 2: count direct work per output scalar.** $$ 3(3)(16)=144 $$ **Step 3: count the full direct layer.** $$ 64(8)(144)=73728 $$ **Step 4: count the 1 × 1 reduction.** $$ 64(4)(1)(1)(16)=4096 $$ **Step 5: count the reduced-depth 3 × 3 layer.** $$ 64(8)(3)(3)(4)=18432 $$ **Step 6: sum both bottleneck stages.** $$ 4096+18432=22528 $$ **Step 7: subtract to obtain savings.** $$ 73728-22528=51200 $$ **Final answer:** Direct = 73,728 MACs; bottleneck = 22,528 MACs; savings = 51,200 MACs. --- ## MT2 · Concatenation and residual addition Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/36/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/36 Concatenation and addition combine branches differently. Concatenation places channel groups next to each other and increases the channel count. Residual addition adds matching entries, so the two full shapes must be compatible. $$ C_{concat}=\sum_b C_b $$ $$ y_{res}=F(x)+x $$ ## Inception concatenation and residual addition **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths. ### Explanation and working Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations. ![Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.](https://deep-learning-study-room.vercel.app/assets/figures/v_join_add.svg) Diagram: Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.. Labels: Concatenation and addition produce different shapes | Concatenation and addition produce different shapes | Use height and width 14 × 14 throughout. | CONCATENATE four branches | Channels: 16 + 32 + 8 + 8 | Result: 14 × 14 × 64 | Stack all 64 channels. | ADD two matching tensors | 14 × 14 × 64 + 14 × 14 × 64 | Result: 14 × 14 × 64 | Add each matching pair. | A 14 × 14 × 32 shortcut needs a projection before addition to a 14 × 14 × 64 | branch. Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape. Tap the diagram to enlarge. Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64. A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition. ## ResNet adds a shortcut **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x. $$ y=F(x)+x $$ If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch. ![The two paths must have equal output shape before elementwise addition.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge. ### Effect on gradient propagation For the identity-shortcut formulation, the derivative includes a direct identity contribution. $$ \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I $$ This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients. ### Projection when dimensions differ If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride. $$ y=F(x)+W_sx $$ An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition. **Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum. ### Given x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3]. ### To find Find the three components of x+F(x). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. ### Solution — step by step **Step 1: add the first matching pair.** $$ 1+0.1=1.1 $$ **Step 2: add the second matching pair.** $$ 2+(-0.2)=1.8 $$ **Step 3: add the third matching pair.** $$ 3+0.3=3.3 $$ **Step 4: assemble in the original order.** $$ y=[1.1,1.8,3.3] $$ The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum. ## Drill 44 Inception concatenation and parameter count ## Question **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters. ## Answer — step by step ### Given Input = 10 × 10 × 8. Branch A: 1 × 1 convolution, 4 filters. Branch B: 3 × 3 same convolution, 6 filters. Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters. ### To find Find each branch shape and parameters, then the concatenated shape and total parameters. ### Formula — which one and why **Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches. C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions. $$ C_{\mathrm{out}}=\sum_k C_k $$ $$ P_{\mathrm{total}}=\sum_k P_k $$ **Yaad rakhne ka cue:** Channels jodo; height-width same rakho. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values. $$ C_{\mathrm{combined}}=C_A+C_B+C_C $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Assume one bias per filter. Pooling has no learned parameters. **Step 1: Branch A.** Its 4 filters produce shape 10 × 10 × 4. $$ P_A=(1\times1\times8+1)\times4=9\times4=36 $$ **Step 2: Branch B.** Its 6 filters produce shape 10 × 10 × 6. $$ P_B=(3\times3\times8+1)\times6=73\times6=438 $$ **Step 3: Branch C.** Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2. $$ P_C=0+(1\times1\times8+1)\times2=18 $$ **Step 4: Join the shapes and add the parameters.** $$ C_{\mathrm{combined}}=4+6+2=12 $$ $$ P_{\mathrm{total}}=36+438+18=492 $$ **Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches. ## Drill 45 Projection shortcut ## Question **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters. ## Answer — step by step ### Given Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16. Shortcut kernel = 1 × 1; stride 2; padding 0. Use one bias per projection filter. ### To find Choose the number of filters and find projection output shape and parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ ### Step 1: choose 16 filters One filter creates one channel, so 16 filters give the required 16 output channels. ### Step 2: use the specified 1 × 1 kernel and stride 2 $$ N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8 $$ The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1. ### Step 3: count weights and biases $$ \text{weights}=1\times1\times8\times16=128 $$ $$ \text{biases}=16,\qquad \text{total}=128+16=144 $$ **Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid. ## Drill 38 Residual addition ## Question **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block. ## Answer — step by step ### Given x = [2,−3,1]; F(x) = [−1,1,0.5]. ### To find Find x+F(x), apply ReLU, and compare with ReLU(F(x)). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join. $$ H(x)=x+F(x),\qquad a_j=\max(0,H_j) $$ ### Step 1: add each pair $$ H_1=2+(-1)=1 $$ $$ H_2=-3+1=-2 $$ $$ H_3=1+0.5=1.5 $$ The residual output before ReLU is **[1, −2, 1.5]**. ### Step 2: apply ReLU $$ [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5] $$ ### Step 3: compare with the plain branch Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**. Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation. --- ## MT2 · Batch normalization and learned scale/shift Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/46/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/46 Batch normalization first uses batch statistics, then applies a learned scale and shift. Epsilon belongs inside the square root. Inference ordinarily uses stored running statistics. $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ ## Batch normalization inside a residual network **Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training. For one feature with m values in a training batch: $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ The batch mean μ\_B is the centre; batch variance σ²\_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch. **Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.** ### Given x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5. ### To find Batch mean, variance, normalized values and scale-and-shift outputs. ### Formula — which one and why Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale. ### Solution — step by step **Step 1: mean.** $$ \mu_B=\frac{2+4}{2}=3 $$ **Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1. **Step 3: squared deviations and variance.** $$ \sigma_B^2=\frac{(-1)^2+1^2}{2}=\frac{1+1}{2}=1 $$ **Step 4: stable denominator.** $$ d=\sqrt{1+0.00001}\approx1.00000499999 $$ **Step 5: normalize separately.** $$ \hat{x}_1=\frac{-1}{d}\approx-0.99999500004 $$ $$ \hat{x}_2=\frac{1}{d}\approx0.99999500004 $$ **Step 6: scale the first normalized value, then shift it.** $$ y_1=2(-0.99999500004)+0.5\approx-1.49999000008 $$ **Step 7: scale the second normalized value, then shift it.** $$ y_2=2(0.99999500004)+0.5\approx2.49999000008 $$ **Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here. **Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo. --- ## MT2 · Global average pooling Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/47/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/47 Global average pooling takes one mean per complete spatial channel. The worked example compares the resulting dense classifier with a classifier after flattening. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ ## Global average pooling: one average for each channel ![Average each channel separately: these are the exact values used in the worked example.](https://deep-learning-study-room.vercel.app/assets/figures/u4_gap.svg) Diagram: Average each channel separately: these are the exact values used in the worked example.. Labels: u4 gap | Global average pooling keeps channels separate | Channel A | 1 | 3 | 5 | 7 | → | 4 | 16 ÷ 4 | Channel B | 2 | 4 | 6 | 8 | → | 5 | 20 ÷ 4 | 2 × 2 × 2 input → two channel means → [4, 5] Average each channel separately: these are the exact values used in the worked example. Tap the diagram to enlarge. **Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier. **Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = $\begin{bmatrix}1 & 3\\5 & 7\end{bmatrix}$ and channel B = $\begin{bmatrix}2 & 4\\6 & 8\end{bmatrix}$ . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.** ### Given H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs. ### To find The GAP result and dense-head parameter count. ### Formula — which one and why Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias. $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Solution — step by step **Step 1: add channel A.** 1 + 3 + 5 + 7 = 16. **Step 2: average channel A.** 16 ÷ 4 = 4. **Step 3: add channel B.** 2 + 4 + 6 + 8 = 20. **Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**. **Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights. **Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero. **Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters. **Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao. --- ## MT2 · Top-k error percentage Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/formulas/48/ Reader: https://deep-learning-study-room.vercel.app/#mt2/formulas/48 Top-k error counts examples whose correct class is absent from the highest-scoring k predictions. Success counts must not be added twice. $$ E_k=\frac{N-C_k}{N}\times100\% $$ ## Top-1 and top-5 error: what an ImageNet result means **Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models. For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures. **Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.** ### Given Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes. ### To find The percentage of mistakes under each rule. ### Formula — which one and why Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage. $$ E_k=\frac{N-C_k}{N}\times100\% $$ N means total evaluated images; C\_k means images whose correct class appears among the top k predictions. ### Solution — step by step **Step 1: top-1 mistakes.** 100 − 78 = 22. **Step 2: top-1 error percentage.** $$ E_1=\frac{22}{100}\times100\%=22\% $$ **Step 3: top-5 mistakes.** 100 − 92 = 8. **Step 4: top-5 error percentage.** $$ E_5=\frac{8}{100}\times100\%=8\% $$ **Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions. **Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success. --- ## MT2 · Examination schedule and scope Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c1-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/1/mt2-c1-s0 The supplied timetable schedules Deep Learning MT2 for **Tuesday, 13 October 2026, 3:00–4:00 pm**, for **10 marks**. **Faculty update received on 12 October:** prepare **Unit 3 (optimization and regularization) and Unit 4 (CNNs)** for theory and numericals. The message highlights possible Unit 3 numericals and two Unit 4 numericals. The 2-mark questions can combine theory and a numerical or be theory only. **Three 3-mark choices, as clarified:** either **Numerical + Numerical + Theory**, or **one mixed Theory-and-Numerical question + one Numerical + one Theory question**. Their exact order and numerical topics are not specified. The common M2 layout already supplied is **Q1(a,b), 2 marks each, compulsory; answer any two of Q2–Q4, 3 marks each**. Our mocks use this layout. Their exact topic allocation is practice design, not an official prediction. Maths, losses, gradients and the neuron model remain prerequisite explanations. **RNN is not named in this faculty update and is now kept in the TEE route, outside this focused MT2 reading path.** --- ## MT2 · Recommended reading order Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c1-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/1/mt2-c1-s1 Study **Optimization algorithms → Regularization and generalization → CNN calculations → CNN architectures**. If a symbol or step is unfamiliar, use the Mathematics revision and Neural-network foundations chapters as short prerequisite references, then return to the Unit 3–4 lesson. For every numerical, reproduce: **Given → To find → formula and why → numbered working → final answer**. For theory, learn the technical definition, mechanism, labelled architecture or relevant formula, advantages/limitations and a concrete example where appropriate. **Hinglish cue:** Unit 3 mein gradient aur update ka order; Unit 4 mein window, shape aur parameter count alag rakho. Theory dono units ki karni hai. --- ## MT2 · Preparation sessions before 13 October Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c1-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/1/mt2-c1-s2 | Session | What to cover | What to write on paper | | --- | --- | --- | | 1 | Essential prerequisite revision only | Derivative of the stated loss; meaning of gradient and learning rate; neuron/CNN notation | | 2 | GD, batch/SGD/mini-batch, momentum and Nesterov | Two iterations, update counts, old versus new state, lookahead gradient | | 3 | AdaGrad, RMSProp and Adam | Accumulators, bias correction, sign changes and complete two-step working | | 4 | Regularization and generalization | L1/L2, dropout, early stopping, underfitting/overfitting, training/validation/test; theory and numerical examples | | 5 | Convolution, stride, padding and pooling | Every window product/sum, multi-channel bias, floor and SAME-padding cases, max/min/average pooling | | 6 | CNN shapes, parameters and architectures | Full layer table, flatten/dense count, bottlenecks, residual/concatenation, architecture and transfer-learning theory | | 7 | Mock A | Two Unit 4 numerical choices and Unit 3 theory; full answers directly below each question | | 8 | Mock B and final revision | Mixed Unit 3/4 numerical choices; review both units' theory and redo incorrect calculations | These are short study blocks, not eight full days. The exam is on 13 October; spend the remaining time on the Unit 3–4 blocks you have not completed. Do not spend this MT2 preparation period on the extra RNN material retained in TEE. --- ## MT2 · Mock-paper conventions Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c1-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/1/mt2-c1-s3 Both practice papers have **60 minutes and 10 attempted marks**. Q1(a,b) contributes 4 compulsory marks; any two 3-mark questions contribute 6 marks. All five answers are provided so every alternative can be studied. Mock A practises theory-only Q1 plus **Numerical + Numerical + Theory** among Q2–Q4, with both long numericals from Unit 4. Mock B practises a theory/numerical Q1 plus **Theory-and-Numerical mixed + Numerical + Theory** among Q2–Q4. Its mixed question is from Unit 3; the other two questions are from Unit 4. These match the two clarified combinations; exact topic assignment is practice design. ### How the supplied papers ask questions Use the Unit 3 and Unit 4 historical questions under **Solved Questions** for longer numerical variations and theory preparation. Historical 5/10-mark questions retain their original marks; they are not predictions of MT2 marks. The new faculty message takes priority over the older provisional course-policy scope inference. For Unit 3, be ready to differentiate the supplied objective, initialize optimizer state, compute the requested iterations and explain the mechanism. For Unit 4, be ready to compute feature maps, pool them, track tensor dimensions, count parameters, compare computational cost and explain the named architectures. Prepare theory from both units even when practising numericals. --- ## MT2 · Writing technical answers Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c1-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/1/mt2-c1-s4 For theory, start with the definition, state the mathematical model or labelled architecture, explain its working, and add the relevant advantage, limitation or comparison. For a numerical, show given values, formula, intermediate states and final result. In iterative algorithms, use one row or clearly labelled block per iteration. --- ## MT2 · Before maths: what are we trying to learn? Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s0 ### Machine ko ek number do. Imagine a tiny machine that estimates a bill. You tell it how many notebooks you bought. This information going INTO the machine is called the INPUT. Abhi bas naam samjho: input = andar diya hua number. ![Start at the three notebooks on the left. Their count is the input; the machine turns that quantity into a bill estimate.](https://deep-learning-study-room.vercel.app/assets/figures/notebooks_input.png) ### Visible text and connections in this diagram INPUT MODEL OUTPUT 3 notebooks Multiply by 10 Estimated bill 3 × 10 Total Rs 30 3 notebooks × Rs 10 per notebook = Rs 30 Input = quantity. Weight = price per notebook. Output = estimated bill. 7 8 9 ÷ 4 5 6 × 1 2 3 − 0 . = + Green arrow: three notebooks to calculator Green arrow: calculator to receipt Diagram: Start at the three notebooks on the left. Their count is the input; the machine turns that quantity into a bill estimate. Start at the three notebooks on the left. Their count is the input; the machine turns that quantity into a bill estimate. Tap the diagram to enlarge. **Calculation:** Input = 3 notebooks ### Machine se ek answer niklega. Suppose the machine says the bill is 30 rupees. This answer coming OUT is called the OUTPUT. Input aur output alag jobs hain: 3 was the quantity; 30 is the predicted bill. ![Input and output name two different jobs: quantity enters and the estimated bill leaves.](https://deep-learning-study-room.vercel.app/assets/figures/v_output.svg) Diagram: Input and output name two different jobs: quantity enters and the estimated bill leaves.. Labels: Follow the number through the machine | Follow the number through the machine | The units change: notebook quantity goes in; estimated rupees come out. | INPUT | 3 notebooks | MODEL | Multiply by 10 | OUTPUT | Rs 30 | 3 is the quantity. 30 is the bill prediction. Input and output name two different jobs: quantity enters and the estimated bill leaves. Tap the diagram to enlarge. **Calculation:** Input: 3 notebooks → Output: 30 rupees ### Multiply kyun karte hain? If each notebook costs 10 rupees, three notebooks cost 10 + 10 + 10 = 30. Multiplication is just the shorter calculation: 3 × 10. In this simple model the multiplying number is called a WEIGHT. ![Point to each Rs 10 notebook contribution, then add the three equal amounts.](https://deep-learning-study-room.vercel.app/assets/figures/v_multiply.svg) Diagram: Point to each Rs 10 notebook contribution, then add the three equal amounts.. Labels: Multiplication collects equal groups | Multiplication collects equal groups | Three notebooks. Each contributes the same Rs 10. | Rs 10 | + | Rs 10 | + | Rs 10 | = Rs 30 | 10 + 10 + 10 = 30. The shorter way to write it is 3 × 10 = 30. Point to each Rs 10 notebook contribution, then add the three equal amounts. Tap the diagram to enlarge. **Calculation:** 3 × 10 = 30 ### Weight badla, answer badla. Keep the input 3. Change only the weight from 10 to 8. The output becomes 3 × 8 = 24. A weight controls how strongly an input affects the answer. Neural-network weights can also be negative or fractions. ![Only the multiplier changes. Compare both outputs while keeping the three notebooks fixed.](https://deep-learning-study-room.vercel.app/assets/figures/v_weight.svg) Diagram: Only the multiplier changes. Compare both outputs while keeping the three notebooks fixed.. Labels: Same input. Change only the weight. | Same input. Change only the weight. | The multiplier controls the prediction. | Weight = 10 | 3 notebooks × 10 | Prediction = Rs 30 | Weight = 8 | 3 notebooks × 8 | Prediction = Rs 24 | The quantity stays 3 in both calculations. Only the multiplier changes. Compare both outputs while keeping the three notebooks fixed. Tap the diagram to enlarge. **Calculation:** Old: 3 × 10 = 30. New: 3 × 8 = 24. ### Correct answer ko target kehte hain. The real bill is 30, but the machine predicted 24. The correct answer supplied for a training example is its TARGET. Our prediction is 6 too small because 30 − 24 = 6. ![The prediction is the machine’s result. The target is the real bill used to judge that result.](https://deep-learning-study-room.vercel.app/assets/figures/v_target.svg) Diagram: The prediction is the machine’s result. The target is the real bill used to judge that result.. Labels: Prediction and target have different jobs | Prediction and target have different jobs | The target is the known correct answer for this training example. | Prediction | Machine says Rs 24 | Target | Actual bill is Rs 30 | Compare | 30 − 24 = 6 rupees too low | Prediction may change during learning. The correct target stays 30. The prediction is the machine’s result. The target is the real bill used to judge that result. Tap the diagram to enlarge. **Calculation:** Target = 30. Prediction = 24. Difference = 6. ### Learning ka matlab weight improve karna. For this tiny example, changing the weight from 8 to 10 changes the prediction from 24 to the correct 30. Learning means using examples to adjust weights and reduce prediction error. Later, gradient descent will calculate HOW to adjust them. ![The weight changes from 8 to 10. The same input now produces a prediction closer to the fixed target.](https://deep-learning-study-room.vercel.app/assets/figures/v_learning.svg) Diagram: The weight changes from 8 to 10. The same input now produces a prediction closer to the fixed target.. Labels: Learning changes the adjustable number | Learning changes the adjustable number | Here we show a useful change; later, gradient descent calculates the | correction. | Before | 3 × 8 = 24 | Adjust weight | 8 becomes 10 | After | 3 × 10 = 30 | Input 3 and target 30 stay fixed. The weight is what changes. The weight changes from 8 to 10. The same input now produces a prediction closer to the fixed target. Tap the diagram to enlarge. **Calculation:** 3 × 8 = 24 → adjust weight → 3 × 10 = 30 ### Bias ek extra add hone wala number hai. Now suppose there is a fixed packing fee of 2 rupees. First calculate the notebook cost: 3 × 10 = 30. Then add 2. That extra adjustable number is called a BIAS. In a neural network, it lets the result shift independently of the input. ![The packing fee is added after the notebook cost. It is not multiplied by the notebook quantity.](https://deep-learning-study-room.vercel.app/assets/figures/v_bias.svg) Diagram: The packing fee is added after the notebook cost. It is not multiplied by the notebook quantity.. Labels: A bias adds a separate amount | A bias adds a separate amount | A fixed packing fee is a useful picture of an added offset. | Notebook cost | 3 × 10 = 30 | Add packing | 30 + 2 | New output | Rs 32 | Multiply first. Add the bias once. 3 × 10 + 2 = 32. The packing fee is added after the notebook cost. It is not multiplied by the notebook quantity. Tap the diagram to enlarge. **Calculation:** 3 × 10 + 2 = 30 + 2 = 32 ### Letters numbers ke naam hain. Instead of writing input, weight and bias every time, books use short names. x means input; w means weight; b means bias. So x × w + b says exactly the calculation you already did. A letter is a named box holding a number. ![Read each letter as a named box holding a number, then follow the same arithmetic.](https://deep-learning-study-room.vercel.app/assets/figures/v_symbols.svg) Diagram: Read each letter as a named box holding a number, then follow the same arithmetic.. Labels: Letters are labels for numbers | Letters are labels for numbers | Read the meaning first, then put the number in its place. | x = input | x holds 3 | w = weight | w holds 10 | b = bias | b holds 2 | x × w + b = 3 × 10 + 2 = 32 | A letter does not add a new operation. It names a value. Read each letter as a named box holding a number, then follow the same arithmetic. Tap the diagram to enlarge. **Calculation:** x = 3, w = 10, b = 2 → x × w + b = 32 ### Deep learning mein aage kya hoga? A neural network connects many small calculating units called neurons. They multiply inputs by weights, add a bias and apply an activation rule. Several layers can learn more complex relationships. First we will learn the small calculation; phir layers jodenge. ![Each circle is a small calculation. Outputs from one layer become the next layer’s inputs.](https://deep-learning-study-room.vercel.app/assets/figures/v_layers.svg) Diagram: Each circle is a small calculation. Outputs from one layer become the next layer’s inputs.. Labels: A deep network connects several calculation stages | A deep network connects several calculation stages | Numbers from one layer become inputs to the next layer. | 1 | 2 | 1 | 2 | 3 | 1 | 2 | 1 | Inputs | Hidden layer 1 | Hidden layer 2 | Output | Predictions are calculated forward | Circle numbers identify neurons, not computed values. Each connection has a | weight; each receiving neuron has a bias. Each circle is a small calculation. Outputs from one layer become the next layer’s inputs. Tap the diagram to enlarge. **Calculation:** Input → calculations in layers → prediction **Question.** A toy model has input 4, weight 5 and bias 2. Find its output using input × weight + bias. ### Given A toy model has input 4, weight 5 and bias 2. ### To find Find the preactivation and the requested neuron output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. ### Solution — step by step **Step 1:** multiply input by weight: 4 × 5 = 20. **Step 2:** add the bias: 20 + 2 = 22. Final output = 22. The bias is added once, after multiplication. --- ## MT2 · Negative numbers are direction Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s1 ### 1. The minus sign can describe a number Zero is the reference point. Positive numbers such as 1, 2 and 3 lie to its right on a number line. Negative numbers such as −1, −2 and −3 lie to its left. The number −3 is three units below zero; it is not an instruction to subtract until another number is supplied. ![Read left as smaller and right as larger. Follow one signed movement at a time.](https://deep-learning-study-room.vercel.app/assets/figures/v_negative.svg) Diagram: Read left as smaller and right as larger. Follow one signed movement at a time.. Labels: Negative numbers are positions and directions | Negative numbers are positions and directions | Left means smaller. Right means larger. | -3 | -2 | -1 | 0 | 1 | 2 | 3 | 0 + (−2) = −2 | −2 + 2 = 0 | Subtracting a negative reverses its direction: 5 − (−2) = 5 + 2 = 7. Read left as smaller and right as larger. Follow one signed movement at a time. Tap the diagram to enlarge. In 5 − 2, the minus sign is an operation: subtract 2 from 5. In 5 + (−2), the sign inside the brackets belongs to the number negative 2. Both expressions finish at 3. Brackets keep the signed number together. ### 2. Adding a negative moves left Start at 5. Adding positive 2 moves two places right, to 7. Adding negative 2 moves two places left, to 3. $$ \begin{aligned}5+2&=7\\5+(-2)&=3\end{aligned} $$ If both numbers are negative, keep moving left. Start at −3 and add −2: the answer is −5. $$ (-3)+(-2)=-5 $$ ### 3. Subtracting reverses that movement Subtracting positive 2 moves left. Subtracting negative 2 reverses a leftward movement, so it moves right. A money example is removing a debt of 2: your balance increases by 2. $$ 5-(-2)=5+2=7 $$ The shortcut “two negatives make a positive” is not a rule for every calculation. Negative 3 plus negative 2 is still −5. First identify the operation: addition, subtraction, multiplication or division. ### 4. Multiplication has its own sign rule For multiplication and division, equal signs give a positive result and different signs give a negative result. Calculate the ordinary size, then attach the correct sign. | Signs being multiplied | Complete calculation | Result sign | | --- | --- | --- | | Positive × positive | 3 × 2 = 6 | Positive | | Negative × positive | (−3) × 2 = −6 | Negative | | Positive × negative | 3 × (−2) = −6 | Negative | | Negative × negative | (−3) × (−2) = 6 | Positive | Division uses the same sign pattern: (−6) ÷ 2 = −3 and (−6) ÷ (−2) = 3. The divisor must be nonzero. ### 5. Fully worked arithmetic **Question.** Calculate −3 + 2, −3 − 2 and −3 − (−2). **Answer — step by step.** ### Given Expressions: −3+2; −3−2; −3−(−2). ### To find Evaluate the three signed-number expressions. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step **Step 1: evaluate the first expression.** start at −3 and move two places right. The answer is −1. $$ -3+2=-1 $$ **Step 2: evaluate the second expression.** start at −3 and move two places left. The answer is −5. $$ -3-2=-5 $$ **Step 3: evaluate the third expression.** replace subtract-negative-2 with add-positive-2. The answer is −1. $$ \begin{aligned}-3-(-2)&=-3+2\\&=-1\end{aligned} $$ Later, a weight update subtracts a correction that can itself be negative. The third expression explains exactly why that update can increase a weight. --- ## MT2 · Fractions decimals and percentages Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s2 ### 1. A fraction tells us to divide The upper number is the **numerator**. The lower number is the **denominator**. In one-quarter, split one whole into four equal parts and take one part. The fraction bar means division. $$ \frac14=1\div4=0.25 $$ ![One shaded part out of four is the same amount as 0.25 or 25 percent.](https://deep-learning-study-room.vercel.app/assets/figures/v_fractions.svg) Diagram: One shaded part out of four is the same amount as 0.25 or 25 percent.. Labels: One whole, split into four equal parts | One whole, split into four equal parts | Shade one of the four parts. | 1 out of 4 | 25% | 1/4 = 1 ÷ 4 = 0.25 = 25%. These are three ways to name the same amount. One shaded part out of four is the same amount as 0.25 or 25 percent. Tap the diagram to enlarge. A decimal and a fraction can name the same amount. One-half is 0.5, one-tenth is 0.1, and one-hundredth is 0.01. The denominator cannot be zero: division by zero is undefined. $$ \begin{aligned}\frac12&=0.5\\\frac1{10}&=0.1\\\frac1{100}&=0.01\end{aligned} $$ ### 2. Why small decimals become smaller when multiplied Multiplying by 0.1 means taking one-tenth. Multiplying by 0.2 means taking two-tenths. For 0.2 × 0.3, express both numbers as fractions. Multiply the numerators and multiply the denominators. $$ \begin{aligned}0.2\times0.3&=\frac2{10}\times\frac3{10}\\&=\frac6{100}\\&=0.06\end{aligned} $$ This also explains a calculation used later: 0.1 × (−0.6) is one-tenth of negative 0.6, which is −0.06. The negative sign is kept because the factors have different signs. ### 3. Dividing by one-half asks how many halves fit Two halves fit inside 1. Four halves fit inside 2. Therefore dividing 2 by one-half gives 4, even though division by a number greater than 1 would make 2 smaller. $$ \begin{aligned}2\div0.5&=2\div\frac12\\&=2\times2\\&=4\end{aligned} $$ The **reciprocal** flips a nonzero fraction: the reciprocal of 1/2 is 2/1. Dividing by a fraction means multiplying by its reciprocal. For decimals, multiplying both the numerator and denominator by the same nonzero number preserves their ratio. $$ \begin{aligned}\frac{0.8}{0.5}&=\frac{0.8\times10}{0.5\times10}\\&=\frac85\\&=1.6\end{aligned} $$ ### 4. Percent means out of one hundred To turn a percentage into a decimal, divide by 100. To turn a decimal into a percentage, multiply by 100. $$ \begin{aligned}20\%&=\frac{20}{100}=0.2\\50\%&=\frac{50}{100}=0.5\end{aligned} $$ If 20% is removed, 80% remains. As decimals: 1 − 0.2 = 0.8. The number 1 represents the whole amount, or 100%. This same arithmetic will be used for dropout later. ### 5. A mean is a fair-share amount For 2, 4 and 9, first add all three numbers: 2 + 4 + 9 = 15. Divide by the number of values, which is 3. The mean is 5. Do not divide by 9 just because it is the largest value. $$ \text{mean}=\frac{\text{sum}}{\text{count}} $$ Here the sum is 15 and the count is 3. Apply those two quantities to the formula. $$ \begin{aligned}\text{mean}&=\frac{2+4+9}{3}\\&=\frac{15}{3}\\&=5\end{aligned} $$ A **weighted mean** gives different shares of importance to the values. With shares 0.9 and 0.1, the first value receives 90% of the importance and the second receives 10%. The shares add to 1. **Question.** Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively. **Answer — step by step.** ### Given Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively. ### To find Find the weighted mean. ### Formula — which one and why **Weighted sum.** Multiply each value by its own coefficient and then add. For a weighted mean, divide by the sum of the weights unless that sum is already 1. xᵢ = value i; aᵢ = its weight. Each input keeps its own matching coefficient. $$ \bar x_w=\frac{\sum_i a_ix_i}{\sum_i a_i} $$ **Yaad rakhne ka cue:** Har number apne weight ke saath multiply; phir sab add. ### Solution — step by step **Step 1:** multiply the first value by its weight: 0.9 × 10 = 9. **Step 2:** multiply the second value by its weight: 0.1 × 20 = 2. **Step 3:** add the contributions: 9 + 2 = 11. The weighted mean is **11**. It is closer to 10 because 10 received most of the importance. $$ \begin{aligned}0.9(10)+0.1(20)&=9+2\\&=11\end{aligned} $$ If importance weights do not sum to 1, divide their weighted sum by the sum of the weights to obtain the weighted mean. A neuron's general weighted sum does not require weights to sum to 1 and is not automatically a mean. --- ## MT2 · Brackets before everything else Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s3 ### 1. Brackets tell us what must stay together An expression is a written calculation. Brackets group part of that calculation. In 1 ÷ (1 + 3), the whole quantity 1 + 3 is the divisor, so calculate it before dividing. ![Follow the bracketed calculation from left to right. The bottom line shows why removing brackets changes the answer.](https://deep-learning-study-room.vercel.app/assets/figures/v_brackets.svg) Diagram: Follow the bracketed calculation from left to right. The bottom line shows why removing brackets changes the answer.. Labels: Calculate inside the bracket first | Calculate inside the bracket first | Compare the two different expressions. | With brackets | 1 ÷ (1 + 3) | Finish inside | 1 ÷ 4 | Result | 0.25 | Without brackets: 1 ÷ 1 + 3 = 1 + 3 = 4. The expression has changed. Follow the bracketed calculation from left to right. The bottom line shows why removing brackets changes the answer. Tap the diagram to enlarge. $$ \frac1{1+3}=\frac14=0.25 $$ Removing the brackets changes the calculation. In 1 ÷ 1 + 3, division comes before addition: 1 ÷ 1 = 1, then 1 + 3 = 4. $$ 1\div1+3=1+3=4 $$ ### 2. Use this order for the expressions in this course First calculate inside brackets. Then evaluate powers and roots. Then do multiplication and division, reading left to right. Finally do addition and subtraction, again reading left to right. Multiplication does not have priority over division; they share a level. For 8 ÷ 2 × 3, calculate 8 ÷ 2 = 4 and then 4 × 3 = 12. Do not change it into 8 ÷ (2 × 3). ### 3. A full example with brackets and a power **Question.** Evaluate (2 + 3)² ÷ 5. **Answer — step by step.** ### Given Evaluate (2 + 3)² ÷ 5. ### To find Evaluate each expression and state its value. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step **Step 1:** finish the brackets: 2 + 3 = 5. The expression becomes 5² ÷ 5. **Step 2:** square the bracket result: 5² = 25. The expression becomes 25 ÷ 5. **Step 3:** divide: 25 ÷ 5 = **5**. $$ \begin{aligned}\frac{(2+3)^2}{5}&=\frac{5^2}{5}\\&=\frac{25}{5}\\&=5\end{aligned} $$ ### 4. A full example without grouping the addition **Question.** Evaluate 2 + 3² × 4. **Answer — step by step.** ### Given Evaluate 2 + 3² × 4. ### To find Evaluate each expression and state its value. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step **Step 1:** calculate the power: 3² = 9. Keep the other numbers where they are: 2 + 9 × 4. **Step 2:** multiply: 9 × 4 = 36. The expression becomes 2 + 36. **Step 3:** add: 2 + 36 = **38**. $$ \begin{aligned}2+3^2\times4&=2+9\times4\\&=2+36\\&=38\end{aligned} $$ ### 5. A fraction bar groups the whole denominator In a printed fraction, everything below the bar belongs to the denominator. When entering that fraction in a single calculator line, put brackets around the complete denominator. A printed fraction with 1 above and 1 + 3 below must be entered as 1 ÷ (1 + 3). --- ## MT2 · Powers roots and very small numbers Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s4 ### 1. A power counts repeated multiplication In 2³, 2 is the **base** and 3 is the **exponent**. For a positive whole-number exponent, multiply that many copies of the base. It does not mean 2 × 3. $$ 2^3=2\times2\times2=8 $$ The exponent 2 is called a **square**: 3² = 3 × 3 = 9. Three rows of three cells make nine cells. ![Count the three rows of three cells. Squaring builds nine; the square root returns three.](https://deep-learning-study-room.vercel.app/assets/figures/v_powers.svg) Diagram: Count the three rows of three cells. Squaring builds nine; the square root returns three.. Labels: A square and its square root | A square and its square root | Three rows of three cells make nine cells. | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 3² = 3 × 3 = 9 | √9 = 3 | The root returns the side length. | (−2)² = (−2) × (−2) = 4. Keep the minus sign inside the brackets. Count the three rows of three cells. Squaring builds nine; the square root returns three. Tap the diagram to enlarge. ### 2. A negative number must stay inside its brackets Squaring negative 2 means multiplying negative 2 by negative 2. The equal signs of the factors give a positive result. $$ (-2)^2=(-2)(-2)=4 $$ Without brackets, −2² means first calculate 2², then apply the minus sign: −(4) = −4. Enter brackets when the entire negative number is the base. For decimals, use the same rule. Negative 0.3 squared is positive 0.09 because 3/10 multiplied by 3/10 is 9/100. $$ \begin{aligned}(-0.3)^2&=(-0.3)(-0.3)\\&=0.09\end{aligned} $$ ### 3. A square root asks which nonnegative number was squared Because 3 × 3 = 9, the principal square root of 9 is 3. Although (−3)² also equals 9, the symbol √9 by itself means the nonnegative root, 3. $$ \begin{aligned}\sqrt9&=3\\\sqrt{0.09}&=0.3\end{aligned} $$ The **absolute value** of a number is its distance from zero: |−4| = 4 and |4| = 4. Squaring and taking a square root returns this nonnegative size. $$ \sqrt{(-4)^2}=\sqrt{16}=4 $$ ### 4. Zero and negative exponents follow a pattern Start with powers of 10. Each time the exponent decreases by 1, divide the previous value by 10: 1000 → 100 → 10 → 1 → 0.1 → 0.01. This explains both the zero power and the negative powers. | Power | Calculation | Value | | --- | --- | --- | | 10³ | 10 × 10 × 10 | 1000 | | 10² | 1000 ÷ 10 | 100 | | 10¹ | 100 ÷ 10 | 10 | | 10⁰ | 10 ÷ 10 | 1 | | 10⁻¹ | 1 ÷ 10 | 0.1 | | 10⁻² | 0.1 ÷ 10 | 0.01 | | 10⁻³ | 0.01 ÷ 10 | 0.001 | For a nonzero base, power zero is 1. A negative exponent means the reciprocal of the corresponding positive power. It does not mean a negative answer. $$ \begin{aligned}10^{-3}&=\frac1{10^3}\\&=\frac1{1000}\\&=0.001\end{aligned} $$ ### 5. Tiny scientific notation is still an ordinary number The notation 1 × 10⁻⁸ equals 0.00000001. Calculators may display it as 1E−8. In optimizer formulas, ε, pronounced “epsilon”, often names a small positive constant added to a denominator. Read its supplied value before calculating. **Question.** Calculate 1 − 0.9². ### Given Expression: 1−0.9². ### To find Evaluate the power first, then the subtraction. ### Formula — which one and why **A square before subtraction.** The exponent applies to 0.9. Squaring is repeated multiplication; perform it before subtracting the result from 1. q denotes the number being squared. Parentheses would be required to square the whole difference. $$ q^2=q\times q $$ $$ 1-q^2=1-(q\times q) $$ **Yaad rakhne ka cue:** Jis number pe power hai, sirf uska square; subtraction baad mein. ### Solution — step by step **Step 1: square only the 0.9.** $$ 0.9^2=0.9(0.9)=0.81 $$ **Step 2: subtract the result from 1.** $$ 1-0.81=0.19 $$ **Final answer:** 0.19. The square acts on 0.9, not on the entire difference. $$ \begin{aligned}1-0.9^2&=1-0.81\\&=0.19\end{aligned} $$ --- ## MT2 · Variables functions and symbols Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s5 These notes assume familiarity with algebra and basic calculus. The revision here covers the notation used in neural-network equations and the conventions that affect numerical answers. | Notation | Technical meaning | | --- | --- | | x, y, ŷ | Input feature vector, target, model prediction | | W, b, θ | Weight matrix, bias vector, complete trainable parameter vector | | z, a | Preactivation (affine input) and activation (layer output) | | L, J | Per-example loss and aggregate training objective | | ∂L/∂w | Partial derivative with respect to one parameter | | ∇θJ | Gradient: vector of partial derivatives with respect to θ | | η, λ, ε | Learning rate, regularization coefficient, numerical stabilizer | | Wᵀ, u⊙v | Transpose and elementwise multiplication | | θₜ, W⁽ˡ⁾ | Update index t and layer index l; neither denotes a power | ### Conventions used in numerical solutions Vectors are columns unless a question specifies otherwise. A weight matrix has one row per receiving neuron and one column per incoming feature. Write matrix dimensions before multiplying. A bias is added once per receiving neuron or output feature map. Keep at least six significant digits during intermediate calculations and round the final result as requested. The loss reduction matters: a sum, mean and half-sum produce different gradient factors. The location of ε also matters: a denominator √s + ε differs from √(s + ε). The sign of a gradient belongs to the derivative, not to the loss value. For example, w = 0.4, η = 0.1 and gradient −2 give the update below. $$ w_{\mathrm{new}}=0.4-0.1(-2)=0.6 $$ In later calculations, state the optimizer convention, initial state and iteration number before substituting values. This prevents mixing momentum velocity with Adam's second-moment accumulator. --- ## MT2 · Exponentials and natural logarithms Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s6 **Definitions.** The exponential function exp(x) = eˣ uses Euler's constant e. The natural logarithm ln is its inverse on positive real values. These functions occur in sigmoid, tanh, softmax and cross-entropy. $$ e^{-x}=\frac1{e^x},\qquad\ln(e^x)=x $$ $$ e^{\ln x}=x\ (x>0),\qquad\ln(ab)=\ln a+\ln b\ (a,b>0) $$ For positive constants and valid domains, exponent and log rules simplify expressions before a calculator is needed. For example, e^(−ln 2) = 1/2. Use natural ln for the loss formulas, not the calculator's base-10 log. ### Derivatives used in neural networks $$ \frac{d}{dz}e^{-z}=-e^{-z},\qquad\frac{d}{dp}[-\ln p]=-\frac1p $$ The minus sign in the first expression comes from differentiating the inner function −z. Applying the chain rule correctly here gives the sigmoid derivative derived in the activation section. ### Brief integration and trigonometry reminder Integration reverses differentiation up to an additive constant. The basic relevant identities are: $$ \int x^n\,dx=\frac{x^{n+1}}{n+1}+C\quad(n\ne-1) $$ $$ \int\frac1x\,dx=\ln|x|+C,\qquad\int e^x\,dx=e^x+C $$ For angles in radians, d(sin x)/dx = cos x and d(cos x)/dx = −sin x. The identity sin²x + cos²x = 1 concerns ordinary trigonometric functions. Hyperbolic tanh is a different function; using tan instead of tanh gives an incorrect recurrent-network answer. $$ \tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad\frac{d}{dz}\tanh z=1-\tanh^2z $$ The supplied MT2 calculation families mainly require derivatives, matrix products, logarithms and elementary statistics. Extended integration or trigonometric exercises are not needed for those families. ### Calculator values for activation calculations At z = 2, sigmoid = 1/(1 + e⁻²) ≈ 0.880797. At z = 1, tanh ≈ 0.761594. Keep unrounded values when a later gradient uses these activations. --- ## MT2 · Calculator routine Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s7 You reported a Casio ES-series calculator. The exact model string is not confirmed. The functions below are mathematical entries; use the keys labelled on your model. On fx-991 ES/fx-991 ES PLUS models, eˣ is normally the SHIFT function of ln. Do not confuse eˣ with the × 10ˣ scientific-notation key. Use the function labels on your own calculator for these entries. ![Calculate the exponential, build the full denominator, then take its reciprocal.](https://deep-learning-study-room.vercel.app/assets/figures/v_calculator.svg) Diagram: Calculate the exponential, build the full denominator, then take its reciprocal.. Labels: Build the calculator expression in pieces | Build the calculator expression in pieces | Example: sigmoid at a raw score of 2. | Exponential | e⁻² ≈ 0.135335 | Denominator | 1 + 0.135335 | Reciprocal | 1 ÷ 1.135335 ≈ | 0.880797 | Enter 1 ÷ (1 + e^(−2)). The entire denominator needs brackets. Calculate the exponential, build the full denominator, then take its reciprocal. Tap the diagram to enlarge. 1. Use ordinary computation mode. Enter complete brackets around a denominator or negative exponent. 2. For sigmoid at z = 2, enter 1 ÷ (1 + e^(−2)). Result: 0.880797078. Keep the full calculator value; write about six decimals during intermediate work. 3. For tanh at z = 1, use the hyperbolic tanh function if available. A universal alternative is (e² − 1) ÷ (e² + 1), giving 0.761594156. Ordinary tan is a different function; do not use it. 4. For cross-entropy with true probability 0.8, enter −ln(0.8), giving 0.223143551. 5. For softmax [2.3, 4.5, 3.2], subtract 4.5 first, then calculate e⁻²·², 1, e⁻¹·³. Add them, then divide each exponential by the same total. 6. If a result displays as a fraction, use your model’s fraction/decimal conversion key, often S⇔D. Do not reset the calculator unnecessarily. **Five-minute warm-up:** compute sigmoid(0) = 0.5; sigmoid(−2) ≈ 0.119203; tanh(0) = 0; −ln(0.5) ≈ 0.693147; √0.25 = 0.5. A different result signals an entry mistake, not a reason to memorize a wrong value. --- ## MT2 · A complete arithmetic calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s8 Evaluate 0.4 − 0.1 × (−2). The multiplication has priority over the subtraction. ![The negative product stays in brackets until the final subtraction is simplified.](https://deep-learning-study-room.vercel.app/assets/figures/v_arithmetic.svg) Diagram: The negative product stays in brackets until the final subtraction is simplified.. Labels: Carry a negative result as one bracketed value | Carry a negative result as one bracketed value | Example: 0.4 − 0.1 × (−2). | Multiply first | 0.1 × (−2) = −0.2 | Replace product | 0.4 − (−0.2) | Subtract negative | 0.4 + 0.2 = 0.6 | The two minus signs have separate jobs before they simplify. The negative product stays in brackets until the final subtraction is simplified. Tap the diagram to enlarge. **Step 1:** multiply 0.1 by −2, giving −0.2. **Step 2:** substitute that entire result, including its sign: 0.4 − (−0.2). **Step 3:** subtracting a negative means adding its positive value: 0.4 + 0.2 = 0.6. The answer is **0.6**. This calculation will later appear inside a weight update; the arithmetic is exactly the same. --- ## MT2 · Loss: how one wrong prediction becomes a number Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c2-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/2/mt2-c2-s9 **Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative. $$ e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2 $$ For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective. ### Derivatives and their scaling When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_j}=2e_j,\qquad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_j}=e_j,\qquad \frac{\partial\mathrm{MSE}}{\partial\hat y_j}=\frac{2}{n}e_j $$ The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight. **Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.** ### Given Predictions = [0.2,0.7,0.4]. Targets = [0,1,1]; n = 3 scalar predictions. ### To find Find SSE, half SSE and MSE, then all three prediction-gradient lists. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step **Step 1: calculate the three signed prediction errors.** $$ e=[0.2-0,\ 0.7-1,\ 0.4-1]=[0.2,-0.3,-0.6] $$ **Step 2: square every error.** $$ e\odot e=[0.04,0.09,0.36] $$ **Step 3: add the three squared errors.** $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ **Step 4: apply the requested loss divisors.** $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245,\qquad \mathrm{MSE}=\frac{0.49}{3}\approx0.163333333 $$ **Step 5: multiply every error by 2 for the SSE gradient.** $$ \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2] $$ **Step 6: use each error itself for the half-SSE gradient.** $$ \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6] $$ **Step 7: multiply every error by 2/3 for the MSE gradient.** $$ \nabla_{\hat y}\mathrm{MSE}=\frac23[0.2,-0.3,-0.6]=[0.133333333,-0.2,-0.4] $$ The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate. --- ## MT2 · A vector is a list and a matrix is a table Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s0 The input [2, 3] is a vector containing two features. A matrix is a rectangle of numbers. “2 by 3” means two rows and three columns. A column vector with two entries has shape 2 by 1. Always write rows before columns. ![Count rows downward and columns across. The shape describes the arrangement of entries.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Count rows downward and columns across. The shape describes the arrangement of entries.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Count rows downward and columns across. The shape describes the arrangement of entries. Tap the diagram to enlarge. In this course, unless a question’s diagram explicitly uses another convention, **each row of a weight matrix belongs to one receiving neuron**. Thus W has shape number of output neurons by number of input features, and the input is a column vector. $$ W=\begin{bmatrix}0.11&0.21\\0.12&0.08\end{bmatrix},\quad x=\begin{bmatrix}2\\3\end{bmatrix},\quad Wx=\begin{bmatrix}0.11(2)+0.21(3)\\0.12(2)+0.08(3)\end{bmatrix}=\begin{bmatrix}0.85\\0.48\end{bmatrix} $$ **Do it slowly:** row 1 says 0.11 × 2 = 0.22 and 0.21 × 3 = 0.63, so its answer is 0.85. Row 2 says 0.12 × 2 = 0.24 and 0.08 × 3 = 0.24, so its answer is 0.48. Matrix multiplication is several weighted sums written together. **Shape check:** (2 by 2) × (2 by 1) gives (2 by 1). The inside numbers must match; the outside numbers are the answer shape. This check catches many transposed-weight errors before arithmetic begins. Transpose swaps rows and columns. Elementwise multiplication does not do row-by-column sums: [2, 3] ⊙ [4, 5] = [8, 15]. Dot product does add: [2, 3] · [4, 5] = 8 + 15 = 23. A training dropout mask uses elementwise multiplication, not a dot product. **Practice.** Multiply W = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by x = [2, 1] as a column. Answer: first row 1 × 2 + 2 × 1 = 4; second row 3 × 2 + 4 × 1 = 10. With b = [1, −1], Wx + b = [5, 9]. --- ## MT2 · Matrix multiplication: every row has a job Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s1 ### Row, column, vector and matrix — read these first A **row** runs horizontally, from left to right. A **column** runs vertically, from top to bottom. A **matrix** is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows. A **vector** is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors. ### Transpose: rows become columns The superscript T means **transpose**, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m. $$ [A^T]_{ij}=A_{ji} $$ The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1. **Yaad rakhne ka cue:** transpose = row ko column bana; number aur sign ko haath mat laga. ### Transpose example 1 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}2&1&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 1 row(s) and 3 column(s), so its size is 1 × 3. **Step 2: move original row 1 into new column 1.** Copy [2, 1, 3] down column 1, in the same order. **Step 3: write the transposed matrix.** $$ A^T=\begin{bmatrix}2\\1\\3\end{bmatrix} $$ **Step 4: state its new size.** The result has 3 row(s) and 1 column(s): 3 × 1. **Final answer:** Aᵀ has size 3 × 1. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 1 rows × 3 columns A: 1 rows × 3 columns | | | | | --- | --- | --- | | 2 | 1 | 3 | Read each colored row left to right. Aᵀ: 3 rows × 1 columns Aᵀ: 3 rows × 1 columns | | | --- | | 2 | | 1 | | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 2 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 3 column(s), so its size is 2 × 3. **Step 2: move original row 1 into new column 1.** Copy [1, 2, 3] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4, 5, 6] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix} $$ **Step 5: state its new size.** The result has 3 row(s) and 2 column(s): 3 × 2. **Final answer:** Aᵀ has size 3 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 3 columns A: 2 rows × 3 columns | | | | | --- | --- | --- | | 1 | 2 | 3 | | 4 | 5 | 6 | Read each colored row left to right. Aᵀ: 3 rows × 2 columns Aᵀ: 3 rows × 2 columns | | | | --- | --- | | 1 | 4 | | 2 | 5 | | 3 | 6 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 3 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&-2\\0&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 2 column(s), so its size is 2 × 2. **Step 2: move original row 1 into new column 1.** Copy [1, -2] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [0, 3] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix} $$ **Step 5: state its new size.** The result has 2 row(s) and 2 column(s): 2 × 2. **Final answer:** Aᵀ has size 2 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 2 columns A: 2 rows × 2 columns | | | | --- | --- | | 1 | −2 | | 0 | 3 | Read each colored row left to right. Aᵀ: 2 rows × 2 columns Aᵀ: 2 rows × 2 columns | | | | --- | --- | | 1 | 0 | | −2 | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 4 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}-1\\4\\2\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 3 row(s) and 1 column(s), so its size is 3 × 1. **Step 2: move original row 1 into new column 1.** Copy [-1] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4] down column 2, in the same order. **Step 4: move original row 3 into new column 3.** Copy [2] down column 3, in the same order. **Step 5: write the transposed matrix.** $$ A^T=\begin{bmatrix}-1&4&2\end{bmatrix} $$ **Step 6: state its new size.** The result has 1 row(s) and 3 column(s): 1 × 3. **Final answer:** Aᵀ has size 1 × 3. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 3 rows × 1 columns A: 3 rows × 1 columns | | | --- | | −1 | | 4 | | 2 | Read each colored row left to right. Aᵀ: 1 rows × 3 columns Aᵀ: 1 rows × 3 columns | | | | | --- | --- | --- | | −1 | 4 | 2 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Dot product: multiply matching entries, then add A **dot product** turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed. $$ w^Tx=w_1x_1+w_2x_2+w_3x_3 $$ This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products. ### Dot-product example 1 **Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.** ### Given Weights w = [2, 3]. Matching inputs x = [4, 1]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (2)(4)=8 $$ **Step 2: multiply matching pair 2.** $$ (3)(1)=3 $$ **Step 3: add the products.** $$ w^Tx=(8)+(3)=11 $$ **Final answer:** wᵀx = 11. ### Dot-product example 2 **Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.** ### Given Weights w = [1, -2]. Matching inputs x = [2, 3]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (1)(2)=2 $$ **Step 2: multiply matching pair 2.** $$ (-2)(3)=-6 $$ **Step 3: add the products.** $$ w^Tx=(2)+(-6)=-4 $$ **Final answer:** wᵀx = -4. ### Dot-product example 3 **Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.** ### Given Weights w = [0.5, 1, -1]. Matching inputs x = [4, 2, 3]. Both lists contain 3 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (0.5)(4)=2 $$ **Step 2: multiply matching pair 2.** $$ (1)(2)=2 $$ **Step 3: multiply matching pair 3.** $$ (-1)(3)=-3 $$ **Step 4: add the products.** $$ w^Tx=(2)+(2)+(-3)=1 $$ **Final answer:** wᵀx = 1. ### Matrix multiplication: each row makes one output To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B. $$ C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj} $$ For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once. **Yaad rakhne ka cue:** multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column. **Definition.** A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation. $$ z=Wx+b $$ For d input features and m receiving neurons, the dimensions are: $$ W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1} $$ The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product. ![Each row of the weight matrix calculates one receiving neuron's preactivation.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge. **Question.** Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b. ### Given Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ. ### To find Find both components of z = Wx+b. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. ### Solution — step by step **Step 1: verify compatible dimensions.** W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column. **Step 2: calculate the first receiving row products.** $$ 1(2)=2 $$ $$ (-2)(1)=-2 $$ **Step 3: sum row 1 and add its bias.** $$ z_1=2+(-2)+0.1=0.1 $$ **Step 4: calculate the second receiving row products.** $$ 0.5(2)=1 $$ $$ 3(1)=3 $$ **Step 5: sum row 2 and add its bias.** $$ z_2=1+3-0.2=3.8 $$ $$ z_1=1(2)+(-2)(1)+0.1=0.1 $$ Row 1 produces output component z1 1. Select weight row 1 1. Select weight row 1 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 2 | −2 | First × first; second × second. Sum the products**0** + Bias for this row**0.1** = z1**0.1** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z_2=0.5(2)+3(1)-0.2=3.8 $$ Row 2 produces output component z2 1. Select weight row 2 1. Select weight row 2 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 1 | 3 | First × first; second × second. Sum the products**4** + Bias for this row**−0.2** = z2**3.8** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z=\begin{bmatrix}0.1\\3.8\end{bmatrix} $$ An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W. ### Two output columns **Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.** ### Given $$ A=\begin{bmatrix}1&2\\3&4\end{bmatrix} $$ $$ B=\begin{bmatrix}2&0\\1&3\end{bmatrix} $$ ### To find The 2 × 2 matrix C = AB. ### Formula — which one and why Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication. Cᵢⱼ is the entry in output row i, column j. $$ C_{ij}=\sum_k A_{ik}B_{kj} $$ **Yaad rakhne ka cue:** Output ki position dekho: wahi row, wahi column. ### Solution — step by step **Step 1: check the dimensions.** (2 × 2)(2 × 2) is valid; output size = 2 × 2. **Step 2: row 1 with column 1.** $$ C_{11}=1(2)+2(1)=2+2=4 $$ **Step 3: row 1 with column 2.** $$ C_{12}=1(0)+2(3)=0+6=6 $$ **Step 4: row 2 with column 1.** $$ C_{21}=3(2)+4(1)=6+4=10 $$ **Step 5: row 2 with column 2.** $$ C_{22}=3(0)+4(3)=0+12=12 $$ **Step 6: assemble the output by position.** $$ C=\begin{bmatrix}4&6\\10&12\end{bmatrix} $$ **Final answer:** C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2. ### Two products that must not be confused **Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.** ### Given a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5. ### To find The 2 × 1 entry-by-entry product and the 2 × 2 outer product. ### Formula — which one and why ⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons. ⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row. $$ (a\odot b)_i=a_i b_i $$ $$ (ab^T)_{ij}=a_i b_j $$ **Yaad rakhne ka cue:** ⊙ mein same jagah; outer product mein har pair. ### Solution — step by step **Step 1: multiply matching entries.** $$ a\odot b=\begin{bmatrix}8\\15\end{bmatrix} $$ **Step 2: transpose b.** $$ b^T=\begin{bmatrix}4&5\end{bmatrix} $$ **Step 3: form row 1 of the outer product.** $$ 2(4)=8,\qquad2(5)=10 $$ **Step 4: form row 2 of the outer product.** $$ 3(4)=12,\qquad3(5)=15 $$ **Step 5: write both rows.** $$ ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix} $$ **Final answer:** Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ . --- ## MT2 · A graph and a slope Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s2 A graph shows how one value changes with another. Put the weight w on the horizontal axis and the loss L on the vertical axis. A slope is change in height divided by change in horizontal position. If L rises by 6 when w rises by 2, the average slope is 3. A derivative is the slope for an extremely small change at one location. A positive derivative says increasing the weight initially increases loss. A negative derivative says increasing the weight initially decreases loss. Gradient descent moves against that local slope. **Gradient batata hai loss kis taraf badhega; hum ulte taraf chhota step lete hain.** $$ L(w)=w^2,\qquad \frac{dL}{dw}=2w $$ At w = 3, slope = 6. With learning rate 0.1, the update is 3 − 0.1 × 6 = 2.4. Loss falls from 9 to 5.76. At w = −3, slope = −6, so the update is −3 − 0.1 × (−6) = −2.4. Both moves approach zero. ![A slope tells you the local direction of increasing loss](https://deep-learning-study-room.vercel.app/assets/figures/gradient.svg) Diagram: A slope tells you the local direction of increasing loss. Labels: w | Loss | Negative slope | increase w | Positive slope | decrease w | New weight = old weight − learning rate × gradient A slope tells you the local direction of increasing loss Tap the diagram to enlarge. --- ## MT2 · Derivatives without a calculus course Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s3 **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector. $$ \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix} $$ The rules needed for the supplied neural-network numericals are: | Function or operation | Derivative | | --- | --- | | Constant c | 0 | | xⁿ | nxⁿ⁻¹ | | c f(x) | c f′(x) | | f(x) + g(x) | f′(x) + g′(x) | | f(x)g(x) | f′(x)g(x) + f(x)g′(x) | | eˣ | eˣ | | ln x, x > 0 | 1/x | | f(g(x)) | f′(g(x))g′(x) | ### A parameter derivative is not an input derivative For z = wx + b, the variable named in the denominator determines which derivative is required. $$ \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1 $$ At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities. ### Gradient direction For a small parameter displacement Δθ, first-order Taylor expansion gives: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large. ### Differentiate before substituting: 3w^2+5 **Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.** ### Given J(w) = 3w^2+5; w = 2. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(2)=3(2)^2+5=17 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=3(2)w^{2-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(2)=3(2)(2)^{1}=12 $$ **Final answer:** J = 17; derivative = 12. ### Differentiate before substituting: 0.7w^3+0 **Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.** ### Given J(w) = 0.7w^3+0; w = 1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(1)=0.7(1)^3+0=0.7 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=0.7(3)w^{3-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(1)=0.7(3)(1)^{2}=2.1 $$ **Final answer:** J = 0.7; derivative = 2.1. ### Differentiate before substituting: 2w^1+4 **Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.** ### Given J(w) = 2w^1+4; w = -1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(-1)=2(-1)^1+4=2 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=2(1)w^{1-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(-1)=2(1)(-1)^{0}=2 $$ **Final answer:** J = 2; derivative = 2. --- ## MT2 · Derivatives: from an ordinary change to a local slope Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s4 ### 1. Function value and change are different numbers Let L(w) = w². At w = 3, loss is 9. If we increase w to 4, loss becomes 16. The loss change is 16 − 9 = 7 and the weight change is 4 − 3 = 1. Their ratio is 7. This is an average slope over a whole interval. ![Start at weight 3 each time. As the increase becomes smaller, the slope estimate approaches 6.](https://deep-learning-study-room.vercel.app/assets/figures/v_slope_limit.svg) Diagram: Start at weight 3 each time. As the increase becomes smaller, the slope estimate approaches 6.. Labels: Shrink the interval to see the local slope | Shrink the interval to see the local slope | Use the same starting point every time: w = 3, L = 9. | Average slope = change in loss ÷ change in weight | Weight increase | New loss | Slope estimate | 1 | 16 | (16 − 9) ÷ 1 = 7 | 0.1 | 9.61 | 0.61 ÷ 0.1 = 6.1 | 0.01 | 9.0601 | 0.0601 ÷ 0.01 = 6.01 | 7 → 6.1 → 6.01 → … → 6. The derivative at w = 3 is 6, not the loss value 9. Start at weight 3 each time. As the increase becomes smaller, the slope estimate approaches 6. Tap the diagram to enlarge. $$ \text{average slope}=\frac{\text{new loss}-\text{old loss}}{\text{new weight}-\text{old weight}} $$ ### 2. Shrink the interval For the same starting weight 3, use an increase of 0.1. The new weight is 3.1 and its square is 9.61. The slope estimate is (9.61 − 9)/0.1 = 6.1. With an increase of 0.01, it is (9.0601 − 9)/0.01 = 6.01. Smaller intervals approach 6. | Increase in w | New w | New loss w² | Change in loss | Change ratio | | --- | --- | --- | --- | --- | | 1 | 4 | 16 | 7 | 7 | | 0.1 | 3.1 | 9.61 | 0.61 | 6.1 | | 0.01 | 3.01 | 9.0601 | 0.0601 | 6.01 | The derivative is the limiting local slope as that interval approaches zero. We do not substitute a literal zero into a division by the interval. ### 3. Why the square has derivative 2w Call a small nonzero change Δw, read “delta w”. Expanding the new square gives: $$ (w+\Delta w)^2=w^2+2w\Delta w+(\Delta w)^2 $$ Subtract the old square w² and divide by the change Δw: $$ \frac{(w+\Delta w)^2-w^2}{\Delta w}=2w+\Delta w $$ As Δw approaches zero, this approaches 2w. That is the origin of the rule d(w²)/dw = 2w. At w = 3 it gives 6, matching the table's trend. ### 4. Reading derivative notation In dL/dw, the top names the quantity whose response we study, and the bottom names what we vary. Read it as “the derivative of L with respect to w”. The notation is not asking you to divide the current loss 9 by the current weight 3; that would give 3 and answer a different question. For a function of several variables, ∂L/∂w is a partial derivative: change w while holding the other independent variables fixed. The gradient groups those partial derivatives for all parameters. ### 5. Why subtract the gradient? If the derivative is positive, increasing w locally increases loss, so we try a small decrease. If it is negative, increasing w locally decreases loss, so we try a small increase. The subtraction in gradient descent automatically handles both signs. $$ w_{\mathrm{new}}=w-\eta g $$ This is local information. A very large step can cross a minimum and make loss worse. The learning rate η keeps control of the step; the derivative does not promise that any step size is safe. --- ## MT2 · Partial derivatives hold other things still Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s5 For z = wx + b, the derivative with respect to w is x, because x is held fixed. The derivative with respect to b is 1. The derivative with respect to x is w. This is the entire local maths of a neuron’s weighted sum. ![Choose one changing parameter. Everything else is held fixed while you find its effect.](https://deep-learning-study-room.vercel.app/assets/figures/v_partial.svg) Diagram: Choose one changing parameter. Everything else is held fixed while you find its effect.. Labels: Change one parameter. Hold the others fixed. | Change one parameter. Hold the others fixed. | For z = wx + b, choose which parameter you are changing. | Change w; keep x and b fixed | ∂z/∂w = x | If x = 3, the slope is 3. | Change b; keep x and w fixed | ∂z/∂b = 1 | Adding 1 to b adds 1 to z. | “Partial” identifies one chosen variable while the other variables stay fixed. Choose one changing parameter. Everything else is held fixed while you find its effect. Tap the diagram to enlarge. $$ \frac{\partial z}{\partial w}=x,\qquad \frac{\partial z}{\partial b}=1,\qquad \frac{\partial z}{\partial x}=w $$ **Question:** z = 3w + b, w = 2, b = 1. Find z, ∂z/∂w and ∂z/∂b. ### Given z = 3w+b; w = 2; b = 1. ### To find Find z, ∂z/∂w and ∂z/∂b. ### Formula — which one and why **Partial derivatives.** Differentiate only with respect to the named variable; all other independent inputs and parameters remain fixed. ∂z/∂w reads “rate of change of z when w changes”. For z = wx+b, x is the coefficient of w, while b has coefficient 1. $$ \frac{\partial z}{\partial w}=x $$ $$ \frac{\partial z}{\partial b}=1 $$ **Yaad rakhne ka cue:** Neeche jis letter ka naam hai, sirf usko change samjho. ### Solution — step by step **Step 1: calculate the function value.** $$ z=3(2)+1=6+1=7 $$ **Step 2: differentiate with respect to w while b is fixed.** $$ \frac{\partial z}{\partial w}=3+0=3 $$ **Step 3: differentiate with respect to b while w is fixed.** $$ \frac{\partial z}{\partial b}=0+1=1 $$ **Final answer:** z = 7; weight derivative = 3; bias derivative = 1. A function value and its rate of change are different quantities. --- ## MT2 · The chain rule is multiplying linked effects Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s6 **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions. ### First understand why the rates multiply Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency. The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w. **Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add. ### Chain-rule example 1: two linear links **Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.** ### Given w = 1; u = 3w+1; v = 2u. ### To find The two function values and the rate dv/dw. ### Formula — which one and why v depends on w through u. Both links must be included, so multiply the two local derivatives. du/dw is the slope of 3w+1. dv/du is the slope of 2u. $$ \frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw} $$ **Yaad rakhne ka cue:** Andar ka slope × bahar ka slope. ### Solution — step by step **Step 1: calculate the intermediate value.** $$ u=3(1)+1=4 $$ **Step 2: calculate the final value.** $$ v=2(4)=8 $$ **Step 3: differentiate the first link.** $$ \frac{du}{dw}=3 $$ **Step 4: differentiate the second link.** $$ \frac{dv}{du}=2 $$ **Step 5: multiply along the path.** $$ \frac{dv}{dw}=2(3)=6 $$ **Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different. ### 2. One output: calculate both parameter gradients **Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1. ### Given Input x = 2; target y = 4. Old weight w = 1; old bias b = 0. Linear prediction z = wx+b; loss L = ½(z−y)². Learning rate η = 0.1; update both parameters once. ### To find Find the loss gradient for w and for b, then w\_new and b\_new. Finally evaluate the changed prediction and loss. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ![The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.](https://deep-learning-study-room.vercel.app/assets/figures/chain_linear_revision.svg) Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1). The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient. Tap the diagram to enlarge. **Step 1 — evaluate the original model.** Use the given parameters before applying any update. $$ z=wx+b=1(2)+0=2 $$ $$ e=z-y=2-4=-2 $$ $$ L=\frac12e^2=\frac12(-2)^2=2 $$ Forward pass: three distinct quantities Prediction z**2**1 × 2 + 0 → Error e**−2**Prediction 2 − target 4 → Half squared loss**2**½ × (−2)² The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide. **Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it. $$ \frac{\partial L}{\partial e}=\frac12(2e)=e $$ $$ \frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1 $$ $$ \delta_z=\frac{\partial L}{\partial z}=e(1)=-2 $$ The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection. **Step 3 — differentiate the neuron's affine expression.** When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1. $$ \frac{\partial z}{\partial w}=x=2 $$ $$ \frac{\partial z}{\partial b}=1 $$ **Step 4 — multiply along each parameter's path.** $$ g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4 $$ $$ g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2 $$ Backward pass: a separate derivative for each parameter Shared loss derivative δz**−2**Both parameter paths start here. Weight path: multiply by x = 2**−4**∂L/∂w = (−2) × 2 Bias path: multiply by 1**−2**∂L/∂b = (−2) × 1 Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate. **Step 5 — update both parameters simultaneously.** Gradient descent subtracts the learning rate times the corresponding gradient. $$ w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4 $$ $$ b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2 $$ One simultaneous parameter update Weight**1 → 1.4**Subtract 0.1 × (−4) Bias**0 → 0.2**Subtract 0.1 × (−2) Both updates use gradients from the same original forward pass. Re-evaluate the model afterward. Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure. **Step 6 — perform a new forward pass to evaluate the update.** $$ z_{\mathrm{new}}=1.4(2)+0.2=3 $$ $$ L_{\mathrm{new}}=\frac12(3-4)^2=0.5 $$ **Final answer:** weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3. ### 3. Two outputs: why backward contributions are added A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward. **Question.** A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed. ### Given h = wx+b; outputs o₁ = 2h and o₂ = −h. x = 2; w = 0.5; b = 0.1; targets = [1,0]. Loss = half the sum of the two squared output errors. Output multipliers 2 and −1 stay fixed. ### To find Find total loss and its derivatives with respect to h, w and b. ### Formula — which one and why **Chain rule with more than one path.** The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b. oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h} $$ $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h} $$ **Yaad rakhne ka cue:** Har raaste ka effect pehle multiply, junction par sab add. ### Solution — step by step ![The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.](https://deep-learning-study-room.vercel.app/assets/figures/chain_branch_revision.svg) Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5 The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias. Tap the diagram to enlarge. **Step 1 — forward values.** $$ h=0.5(2)+0.1=1.1 $$ $$ o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1 $$ $$ L_1=\frac12(2.2-1)^2=0.72 $$ $$ L_2=\frac12(-1.1-0)^2=0.605 $$ $$ L=L_1+L_2=0.72+0.605=1.325 $$ **Step 2 — output derivatives.** Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output. $$ \frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2 $$ $$ \frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1 $$ **Step 3 — calculate the contribution of each complete path.** The derivative of 2h with respect to h is 2; the derivative of −h is −1. $$ \frac{\partial L_1}{\partial h}=(1.2)(2)=2.4 $$ $$ \frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1 $$ **Step 4 — add the two contributions at the shared hidden value.** $$ \frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h} $$ $$ \frac{\partial L}{\partial h}=2.4+1.1=3.5 $$ Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path. **Step 5 — continue to the hidden weight and bias.** Since h = wx + b, its derivatives with respect to w and b are x and 1. $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7 $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5 $$ **Final answer:** L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation. ### Chain-rule example 4: a square after a linear link **Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.** ### Given w = 1; z = 3w+1; L = z². ### To find z, L and the derivative of L with respect to w. ### Formula — which one and why The square acts on z, not directly on w. Differentiate the square first and include the change of z with w. dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ $$ \frac{d(z^2)}{dz}=2z $$ **Yaad rakhne ka cue:** Square ka 2z; andar ka 3; dono multiply. ### Solution — step by step **Step 1: evaluate the inner expression.** $$ z=3(1)+1=4 $$ **Step 2: evaluate the loss expression.** $$ L=4^2=16 $$ **Step 3: differentiate the square and use z = 4.** $$ \frac{dL}{dz}=2z=2(4)=8 $$ **Step 4: differentiate the inner expression.** $$ \frac{dz}{dw}=3 $$ **Step 5: multiply, because the changes occur in sequence.** $$ \frac{dL}{dw}=8(3)=24 $$ **Final answer:** z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3. --- ## MT2 · The derivative of prediction error Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s7 For the half squared loss, use prediction minus target. Let target be y and prediction be ŷ. ![The half and the averaging factor belong to the loss definition, so they also affect its derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: The half and the averaging factor belong to the loss definition, so they also affect its derivative.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. The half and the averaging factor belong to the loss definition, so they also affect its derivative. Tap the diagram to enlarge. $$ L=\frac12(\hat y-y)^2,\qquad \frac{\partial L}{\partial \hat y}=\hat y-y $$ The 2 from differentiating a square cancels the half. Writing the loss as ½(y − ŷ)² gives the same derivative: the derivative of the inside y − ŷ adds a minus sign. This is why “target minus prediction” without the extra minus gives the wrong gradient. If the question uses plain squared error, its derivative is 2(ŷ − y). If it averages errors over n values, also divide by n. Read the stated loss before using a memorized gradient. --- ## MT2 · Loss scaling: derive the gradient instead of guessing a factor Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s8 ### 1. One error, three possible objectives Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square: ![Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge. $$ \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1 $$ Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below. | Stated loss over n outputs | Derivative with respect to one prediction aⱼ | | --- | --- | | SSE: sum of all squared errors | 2(aⱼ − yⱼ) | | Half SSE: half of that sum | aⱼ − yⱼ | | MSE: that sum divided by n | 2(aⱼ − yⱼ)/n | | Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n | ### 2. Why only the matching output appears Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor. Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages. ### 3. The activation contributes another factor If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore: $$ \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a) $$ For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question. ### 4. A numerical comparison For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042. The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used. --- ## MT2 · Probability mean and variance in plain language Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s9 **Definitions.** A probability assigns a value between 0 and 1 to an event. A discrete probability distribution assigns probabilities to mutually exclusive outcomes whose probabilities sum to 1. Expectation is a probability-weighted average. Variance measures expected squared deviation from the mean. $$ \sum_i p_i=1,\qquad\mathbb E[X]=\sum_i p_i x_i $$ $$ \operatorname{Var}(X)=\mathbb E[(X-\mu)^2]=\mathbb E[X^2]-\mu^2 $$ ### Conditional probability and classification P(A|B) is the probability of A conditional on B, defined when P(B) > 0. Bayes' rule relates it to the reverse conditional probability. $$ P(A\mid B)=\frac{P(A\cap B)}{P(B)}=\frac{P(B\mid A)P(A)}{P(B)} $$ For classification, the network estimates P(class | input). A softmax output is normalized across mutually exclusive classes. Independent sigmoid outputs are used for multiple labels that can be present together. ### Empirical mean, variance and standardization For N observed values, use the stated variance convention. These notes use division by N for the population-style variance in the numerical; an unbiased sample-variance estimate uses N − 1. $$ \mu=\frac1N\sum_i x_i,\qquad\sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2},\qquad x_{\mathrm{std}}=\frac{x-\mu}{\sigma} $$ For values [3, 3, 7, 7], μ = 5, squared deviations are [4, 4, 4, 4], σ² = 16/4 = 4 and σ = 2. Thus a value 7 standardizes to (7 − 5)/2 = 1: one standard deviation above the mean. For preprocessing, fit μ and σ on training data and reuse them for validation/test data; σ must be nonzero. ### Likelihood and cross-entropy If the correct class has predicted probability p, its negative-log loss is −ln p. For p = 0.8, this is approximately 0.223144. For p = 0.1, it is approximately 2.302585. Increasing correct-class probability lowers this loss. Over independent examples, maximizing the product of correct-class probabilities is equivalent to minimizing the sum of negative log probabilities. ### Population mean and spread: [1, 2, 3] **Question. For the population [1, 2, 3], find mean, population variance and standard deviation.** ### Given Values = [1, 2, 3]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 1+2+3=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (1-2)=-1 $$ $$ (-1)^2=1 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (3-2)=1 $$ $$ (1)^2=1 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{1+0+1}{3}\approx0.666666667 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.666666667}\approx0.816496581 $$ **Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581. ### Population mean and spread: [2, 2, 2] **Question. For the population [2, 2, 2], find mean, population variance and standard deviation.** ### Given Values = [2, 2, 2]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 2+2+2=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{0+0+0}{3}\approx0.000000000 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.000000000}\approx0.000000000 $$ **Final answer:** μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation. --- ## MT2 · Mean, variance and standard deviation from ordinary arithmetic Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s10/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s10 **Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i,\qquad \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2,\qquad \sigma=\sqrt{\sigma^2} $$ These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable. **Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].** ### Given Observations = [3,3,7,7]; N = 4. Use population variance (divide by N). ### To find Find the population mean μ, variance σ² and standard deviation σ. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3. --- ## MT2 · Standardization: what the answer 1 actually means Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s11/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s11 **Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units. $$ z=\frac{x-\mu_{\mathrm{train}}}{\sigma_{\mathrm{train}}},\qquad \sigma_{\mathrm{train}}>0 $$ Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1]. **Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.** ### Given Training observations = [3,3,7,7]. New observation = 9. Estimate the population standard deviation from the training values. ### To find Find the training mean and standard deviation; standardize all four training values and the new observation. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. **Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units. x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Step 8: standardize each original training observation.** $$ \frac{3-5}{2}=-1 $$ $$ \frac{7-5}{2}=1 $$ The repeated observations give [-1, −1, 1, 1]. **Step 9: subtract the training mean from the new observation.** $$ 9-5=4 $$ **Step 10: divide by the training standard deviation.** $$ z_{\mathrm{new}}=\frac42=2 $$ **Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy. ### Standardization with x = 3 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.** ### Given x = 3; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=3-5=-2 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{-2}{2}=-1 $$ **Step 3: interpret sign and size.** The value is 1 standard-deviation unit(s) below the training mean. **Final answer:** z = -1. Use the same training statistics when transforming a new test input. ### Standardization with x = 5 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.** ### Given x = 5; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=5-5=0 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{0}{2}=0 $$ **Step 3: interpret sign and size.** The value is 0 standard-deviation unit(s) away from the training mean. **Final answer:** z = 0. Use the same training statistics when transforming a new test input. ### Standardization with x = 9 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.** ### Given x = 9; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=9-5=4 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{4}{2}=2 $$ **Step 3: interpret sign and size.** The value is 2 standard-deviation unit(s) above the training mean. **Final answer:** z = 2. Use the same training statistics when transforming a new test input. --- ## MT2 · Chain rule: the complete calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c3-s12/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/3/mt2-c3-s12 When one value depends on another, a change passes through both calculations. The total derivative multiplies the two local derivatives. It does not add them. Here w is the number being changed, z is an intermediate value, and L is the final value. ![Follow w to z to loss. The local slopes on that dependency path multiply.](https://deep-learning-study-room.vercel.app/assets/figures/v_chain_rule.svg) Diagram: Follow w to z to loss. The local slopes on that dependency path multiply.. Labels: Linked changes multiply along a path | Linked changes multiply along a path | Example: z = 3w + 1 and L = z², evaluated at w = 1. | w = 1 | dz/dw = 3 | z = 4 | dL/dz = 2z = 8 | L = 16 | dL/dw = 8 × 3 = 24 | w affects z; z affects L. Multiply the two local slopes. Follow w to z to loss. The local slopes on that dependency path multiply. Tap the diagram to enlarge. Let z = 3w + 1 and L = z². We will calculate the derivative of L with respect to w at w = 1. **Step 1: find the intermediate value.** Replace w by 1: z = 3 × 1 + 1 = 4. **Step 2: differentiate the last calculation.** For L = z², the power rule gives dL/dz = 2z. At z = 4, this is 2 × 4 = 8. This measures how L responds to z. **Step 3: differentiate the first calculation.** For z = 3w + 1, the derivative of 3w is 3 and the derivative of the constant 1 is 0. Therefore dz/dw = 3. This measures how z responds to w. **Step 4: connect the two effects.** Multiply 8 by 3. The final derivative is 24. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}=(2z)(3)=8\times3=24 $$ The function value is L = 4² = 16. The derivative is 24. These have different meanings: 16 is the current value of L; 24 is its local rate of change per unit change in w. --- ## MT2 · What does learning from data mean Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s0 **Definition.** Machine learning develops models from data so that they can perform a task on new examples. A model is a calculation rule with adjustable numbers. Deep learning uses neural networks with multiple trainable layers; each layer transforms the previous layer's outputs. ### Start with one complete training example A **sample** means one example in the dataset. Suppose one example says: an input number is 2 and the required answer is 4. We write the input as x = 2 and the correct answer, or **target**, as y = 4. These letters are labels for the two numbers. An **input feature** is one recorded property used to make a prediction. One house might have area, number of rooms and age as three features. The three values together describe one sample. An **input vector** is just an ordered list of those feature values. Order matters: the first model weight must always multiply the first feature. The model's calculated answer is the **prediction**, written ŷ, read “y hat”. The hat helps us distinguish the calculated answer ŷ from the required answer y. A prediction is a probability only when the task and output function give it that interpretation. ### The simplest model: multiply, then add In the following model, x is the input, w is a weight and b is a bias. A **weight** multiplies the input. A **bias** is added once after multiplication. Both w and b are adjustable model numbers, called **parameters**. $$ \hat y=wx+b $$ The model first multiplies w and x; then it adds b. For several inputs it uses one matching weight per input, adds their products, and then adds the bias. **Yaad rakhne ka cue:** x diya hua input; y sahi answer; ŷ model ka answer; w aur b training mein badlenge. ![Input numbers are multiplied by weights, their contributions are added with a bias, and an activation gives the neuron output. The next lesson explains each part.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Input numbers are multiplied by weights, their contributions are added with a bias, and an activation gives the neuron output. The next lesson explains each part.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Input numbers are multiplied by weights, their contributions are added with a bias, and an activation gives the neuron output. The next lesson explains each part. Tap the diagram to enlarge. ### Model example 1: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 2, w = 1 and b = 0, calculate its prediction.** ### Given Input x = 2. Weight w = 1; bias b = 0. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(1)(2)=2 $$ **Step 2: add the bias once.** $$ \hat y=2+(0)=2 $$ **Final answer:** ŷ = 2. This is a prediction, not a loss or a learned weight. ### Model example 2: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 2, w = 1.5 and b = 1, calculate its prediction.** ### Given Input x = 2. Weight w = 1.5; bias b = 1. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(1.5)(2)=3 $$ **Step 2: add the bias once.** $$ \hat y=3+(1)=4 $$ **Final answer:** ŷ = 4. This is a prediction, not a loss or a learned weight. ### Model example 3: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 3, w = 1.5 and b = 1, calculate its prediction.** ### Given Input x = 3. Weight w = 1.5; bias b = 1. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(1.5)(3)=4.5 $$ **Step 2: add the bias once.** $$ \hat y=4.5+(1)=5.5 $$ **Final answer:** ŷ = 5.5. This is a prediction, not a loss or a learned weight. ### Model example 4: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 2, w = -0.5 and b = 3, calculate its prediction.** ### Given Input x = 2. Weight w = -0.5; bias b = 3. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(-0.5)(2)=-1 $$ **Step 2: add the bias once.** $$ \hat y=-1+(3)=2 $$ **Final answer:** ŷ = 2. This is a prediction, not a loss or a learned weight. ### How the model learns An initial model can predict 2 when the target is 4. A **loss function** measures the discrepancy. Training calculates how changing each parameter changes the loss; that rate is a **gradient**. An **optimizer** uses the gradient to adjust the parameters. The learning rate controls the size of this adjustment. The loss, derivative and complete update are explained with full examples in the next lessons. The learning rate is a **hyperparameter**: a setting chosen for training. A learned weight is a **parameter**: an adjustable number inside the model. During ordinary **inference**, the trained weights stay fixed and the model calculates an output for a new input. For many training samples, the objective normally averages the individual losses. θ (theta) means the collection of adjustable parameters; N is the number of training examples; Lᵢ is the loss for example i. $$ J(\theta)=\frac1N\sum_{i=1}^{N}L_i(\theta) $$ Σ is an instruction to add the listed terms. For three losses it means L₁ + L₂ + L₃, followed by division by 3. Regularization may add a penalty; the regularization chapter explains that separate term. ### Why several layers help A **layer** contains several neurons that calculate outputs from the same incoming values. In an image model, earlier layers can respond to simple patterns such as edges; later layers combine earlier responses into more complex features. This hierarchy is learned from data. A deep network is not guaranteed to learn useful features merely because it has many layers; training data, architecture and optimization still matter. **Generalization** means performing well on new examples. Training data adjusts weights; validation data helps choose settings and stopping time; held-out test data evaluates the chosen model. Small training loss alone does not establish good generalization. **Yaad rakhne ka cue:** training = answer nikalo → loss nikalo → gradients nikalo → parameters update karo; inference = trained model se answer nikalo. ### Average training loss **Question. Three training examples have losses 2, 0.5 and 0.5. Calculate their mean training loss.** ### Given L₁ = 2; L₂ = 0.5; L₃ = 0.5; N = 3. ### To find The mean loss J. ### Formula — which one and why The question asks for the mean, so add all sample losses and divide by the number of samples. Lᵢ means loss of example i. N counts examples. $$ J=\frac{L_1+L_2+L_3}{N} $$ **Yaad rakhne ka cue:** Mean = total loss ÷ examples. ### Solution — step by step **Step 1: add the losses.** $$ 2+0.5+0.5=3 $$ **Step 2: divide by the number of examples.** $$ J=\frac33=1 $$ **Final answer:** Mean training loss = 1. --- ## MT2 · Your first complete training step Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s1 **Question.** A linear model is defined by $\hat y=wx$ . For the training example x = 2, y = 4, use initial weight w₀ = 1, learning rate η = 0.1 and half squared loss. Perform two gradient-descent updates and report the prediction and loss after each update. ### Given Model: ŷ = wx; bias is absent and is not updated. x = 2; y = 4; initial w₀ = 1; η = 0.1. Loss = half squared error; perform two updates. ### To find Find the weight, prediction and loss after each update. ### Formula — which one and why **Chain rule with only a trainable weight.** The weight changes the prediction and the prediction changes the half squared loss. Multiply both rates, then subtract the scaled gradient. The stated model has no bias parameter. ŷ = prediction; y = target; x = input; w = weight; η = learning rate; g = loss derivative with respect to w. $$ \hat y=wx $$ $$ L=\frac12(\hat y-y)^2 $$ $$ g=(\hat y-y)x $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Prediction minus target, phir input se multiply; naya weight = purana − ηg. ### Solution — step by step ### Model, objective and gradient The input and target remain fixed; only the weight is trainable. There is no bias parameter in this question. $$ \hat y=wx,\qquad L(w)=\frac12(wx-y)^2,\qquad w_t=w_{t-1}-\eta g_t $$ Apply the chain rule through the prediction: $$ g_t=\left.\frac{\partial L}{\partial w}\right|_{w_{t-1}}=\frac{\partial L}{\partial\hat y}\frac{\partial\hat y}{\partial w}=(\hat y-y)x $$ The factor 1/2 in the objective cancels the factor 2 from differentiating the square. The remaining factor x comes from differentiating wx with respect to w. ![Forward evaluation, gradient calculation and a parameter update for the same training example.](https://deep-learning-study-room.vercel.app/assets/figures/first_training.png) ### Visible text and connections in this diagram HOW ONE WEIGHT LEARNS 1. PREDICT Input x = 2 Weight w = 1 Prediction = 1 × 2 = 2 Target y = 4 Prediction is 2 too small. 2. MEASURE AND CORRECT Loss = ½ × (2 − 4)² = 2 Gradient = (2 − 4) × 2 = −4 New weight = 1 − 0.1 × (−4) = 1.4 Learning rate = 0.1 3. PREDICT AGAIN Same input x = 2 New weight w = 1.4 New prediction = 1.4 × 2 = 2.8 Target stays 4 Loss falls from 2 to 0.72. Predict row: input to weight to prediction Measure and correct row: loss to gradient to new weight Predict again row: same input to new weight to new prediction Diagram: Forward evaluation, gradient calculation and a parameter update for the same training example. Forward evaluation, gradient calculation and a parameter update for the same training example. Tap the diagram to enlarge. ### Iteration 1 **Step 1: forward evaluation at the current parameter.** $$ \hat y_0=w_0x=1(2)=2,\qquad e_0=\hat y_0-y=2-4=-2 $$ $$ L_0=\frac12(-2)^2=2 $$ **Step 2: gradient at the current parameter.** $$ g_1=(\hat y_0-y)x=(-2)(2)=-4 $$ **Step 3: parameter update.** $$ w_1=w_0-\eta g_1=1-0.1(-4)=1.4 $$ **Step 4: evaluate the updated model.** $$ \hat y_1=1.4(2)=2.8,\qquad L_1=\frac12(2.8-4)^2=0.72 $$ ### Iteration 2 Use w₁ = 1.4 and recompute the gradient; reusing the previous gradient would not be gradient descent on this objective. $$ g_2=(\hat y_1-y)x=(2.8-4)(2)=-2.4 $$ $$ w_2=w_1-\eta g_2=1.4-0.1(-2.4)=1.64 $$ $$ \hat y_2=1.64(2)=3.28,\qquad L_2=\frac12(3.28-4)^2=0.2592 $$ | Model state | Weight | Prediction | Half squared loss | | --- | --- | --- | --- | | Initial | 1 | 2 | 2 | | After update 1 | 1.4 | 2.8 | 0.72 | | After update 2 | 1.64 | 3.28 | 0.2592 | **Interpretation.** Both gradients are negative, so the updates increase the weight toward the optimum w = y/x = 2. The loss decreases in these iterations. Gradient descent does not guarantee a decrease for every learning rate; step size matters. --- ## MT2 · Classification regression and learning types Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s2 **Definition.** A learning problem is specified by its data, desired output and objective. Classification estimates a discrete label or class distribution; regression estimates numerical responses. The supervision type describes the feedback available during training. ### Explanation and working **Regression** predicts a numerical quantity, such as temperature, price or a continuous rating. **Classification** predicts a category, such as defective/non-defective or cat/dog/bird. Binary classification has two classes. Multiclass classification chooses one of several classes. Multilabel classification can choose several labels together, such as “beach” and “sunset” for the same image. ![Look at the required answer: a numerical quantity is regression; a category is classification.](https://deep-learning-study-room.vercel.app/assets/figures/v_tasks.svg) Diagram: Look at the required answer: a numerical quantity is regression; a category is classification.. Labels: What kind of answer do you need? | What kind of answer do you need? | The output decides whether this is regression or classification. | REGRESSION: a number | Input: house features | Output: predicted price | Example: Rs 50 lakh | CLASSIFICATION: a category | Input: an image | Output: cat / dog / bird | Example: dog | Supervised training supplies correct input–target pairs for these tasks. Look at the required answer: a numerical quantity is regression; a category is classification. Tap the diagram to enlarge. **Supervised learning** uses known input–target pairs. Most examination numericals in this course use it. **Unsupervised learning** searches for patterns without supplied target labels, for example grouping similar samples. **Reinforcement learning** learns actions using rewards from an environment. You only need these distinctions as vocabulary here; detailed algorithms from last semester are not prerequisites for this course. --- ## MT2 · Training validation and test data Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s3 **Definition.** Dataset partitioning separates parameter estimation, model selection and final performance assessment. This separation is essential to evaluate generalization without contaminating the evaluation with selection decisions. ### Explanation and working Training data changes the weights. Validation data helps choose settings such as layer size, learning rate and the stopping epoch. Test data is held back for the final estimate of performance on unseen samples. A possible split is 70% training, 15% validation, 15% testing, but the best split depends on the task and dataset size. ![The three splits have different jobs. Only the training part is used to fit preprocessing and model parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_splits.svg) Diagram: The three splits have different jobs. Only the training part is used to fit preprocessing and model parameters.. Labels: Give each data split one job | Give each data split one job | Example: 1,000 independent samples split 70% / 15% / 15%. | 700 train | 150 validate | 150 test | Training | Adjust weights. | Validation | Choose settings. | Test | Final evaluation. | Fit preprocessing on training data. Reuse the same rule for validation and test. The three splits have different jobs. Only the training part is used to fit preprocessing and model parameters. Tap the diagram to enlarge. For 1,000 independent samples, this split gives 700 training, 150 validation and 150 testing. Avoid putting duplicates or near-identical views of the same subject across splits. For time series, later observations normally belong after earlier observations rather than being mixed randomly without considering time. **Data leakage** occurs when information that would not be available at prediction time enters training or model selection. Fitting a normalization rule to the whole dataset before splitting can leak information. Estimate preprocessing quantities from training data and apply the same rule to validation and test data. **Normalization example.** To map a training range 0 to 100 into 0 to 1, divide by 100. A value 40 becomes 0.4. Standardization instead subtracts the training mean and divides by the training standard deviation. Scaling can make gradients easier to optimize; it does not create labels or guarantee accuracy. --- ## MT2 · Parameters and hyperparameters Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s4 **Definition.** Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure. ### Explanation and working Weights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights. ![Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters.svg) Diagram: Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately.. Labels: What is learned, and what is chosen? | What is learned, and what is chosen? | Two different kinds of quantities appear in training. | PARAMETERS | Weights and biases | Changed by training | Example: w = 0.5 → 0.6 | HYPERPARAMETERS | Learning rate, batch size | Chosen or tuned separately | Example: learning rate 0.1 | Optimizer memories are algorithm state; they are not extra network connection | weights. Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately. Tap the diagram to enlarge. An **epoch** is one complete pass through the training dataset. A **batch** is the group used to calculate one update. An **iteration** here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update. --- ## MT2 · What one artificial neuron does Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s5 **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output. ### What goes into a neuron? An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive. The **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**. **Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output. ### Mathematical model and notation $$ z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b $$ $$ a=f(z) $$ | Symbol | Meaning and role | | --- | --- | | xᵢ | Input component i; the neuron receives d such components | | wᵢ | Trainable weight multiplying that input component | | b | Trainable bias added once to this neuron's total | | z | Preactivation: the result before applying the activation function | | f | Chosen activation function, such as sigmoid or ReLU | | a | Activation: the neuron's output after applying f | The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs. ### Why both a bias and an activation are needed The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin. The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values. ![Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron. Tap the diagram to enlarge. ### Worked numerical **Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output. ### Given Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5]. Bias = 0.1; activation = sigmoid. ### To find Find the preactivation z, then the sigmoid output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. ### From one neuron to a layer With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron: $$ z=Wx+b,\qquad a=f(z) $$ Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together. ### Neuron example 2: ReLU output **Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2]. Weights = [3]; bias = 1. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(3)=6 $$ **Step 2: add the input contributions.** $$ s=(6)=6 $$ **Step 3: add the bias once.** $$ z=6+(1)=7 $$ **Step 4: apply the stated activation.** $$ a=\max(0,7)=7 $$ **Final answer:** z = 7; a = 7. ### Neuron example 3: ReLU output **Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.** ### Given Inputs = [2, 1]. Weights = [-1, 0.5]; bias = 0.2. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(-1)=-2 $$ **Step 2: multiply input 2 by its own weight.** $$ (1)(0.5)=0.5 $$ **Step 3: add the input contributions.** $$ s=(-2)+(0.5)=-1.5 $$ **Step 4: add the bias once.** $$ z=-1.5+(0.2)=-1.3 $$ **Step 5: apply the stated activation.** $$ a=\max(0,-1.3)=0 $$ **Final answer:** z = -1.3; a = 0. ### Neuron example 4: sigmoid output **Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.** ### Given Inputs = [1, 2]. Weights = [0.2, -0.1]; bias = 0. Activation = sigmoid. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (1)(0.2)=0.2 $$ **Step 2: multiply input 2 by its own weight.** $$ (2)(-0.1)=-0.2 $$ **Step 3: add the input contributions.** $$ s=(0.2)+(-0.2)=0 $$ **Step 4: add the bias once.** $$ z=0+(0)=0 $$ **Step 5: apply the stated activation.** $$ a=\frac1{1+e^{0}}=\frac1{1+1}=0.5 $$ **Final answer:** z = 0; a = 0.5. --- ## MT2 · Model answer: explain an artificial neuron Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s6 ## Question **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example. ## Answer — step by step An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers. ![Follow each input through its weight, add one bias at the receiving neuron, then apply activation.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Follow each input through its weight, add one bias at the receiving neuron, then apply activation. Tap the diagram to enlarge. ### Formula and symbols $$ z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z) $$ | Part | Meaning | What it does | | --- | --- | --- | | x₁, x₂, … | Input features | Supply the information for this calculation | | w₁, w₂, … | Weights | Control the contribution of each matching input | | b | Bias | Shifts the weighted sum independently of the inputs | | z | Net input | The number before activation | | f | Activation rule | Transforms z, often nonlinearly | | a | Activated output | The number sent onward | ### Worked ReLU neuron example **Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU. ### To find Preactivation z and activated output a. ### Formula — which one and why Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total. z means before activation; a means after activation. $$ z=w_1x_1+w_2x_2+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Product, product, sum, bias, phir activation. ### Solution — step by step **Step 1: multiply the first pair.** $$ 0.5(2)=1 $$ **Step 2: multiply the second pair.** $$ (-0.2)(1)=-0.2 $$ **Step 3: add both contributions.** $$ 1+(-0.2)=0.8 $$ **Step 4: add the bias once.** $$ z=0.8+0.1=0.9 $$ **Step 5: apply ReLU.** $$ a=\max(0,0.9)=0.9 $$ **Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged. The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once. --- ## MT2 · Activation functions and how to calculate them Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s7 ### What an activation function does, and why it is needed A neuron first produces the weighted sum z. The activation function f is a rule that transforms that sum into the output a = f(z). Applying an activation does not change the stored weights; it calculates a new output from the current sum. If every layer only multiplies and adds, several layers still combine into a single affine transformation. A nonlinear activation lets the network represent more complicated relationships. The output activation must also suit the task: sigmoid for a binary-class probability, softmax for mutually exclusive class probabilities, and often a linear output for unrestricted regression. The **activation value** says what the neuron outputs. The **activation derivative** says how quickly that output changes when z changes, which is needed during backpropagation. For sigmoid at z = 0, the value is 0.5 but the derivative is 0.25. They serve different purposes. **Yaad rakhne ka cue:** z = activation se pehle ka sum; a = uske baad ka answer; f′ = answer kitni tezi se badlega. **Definition.** An activation function maps a neuron's preactivation to its output. Nonlinear activations allow a multilayer network to represent nonlinear functions; the output activation also encodes the prediction type. Without a nonlinear activation, a composition of affine layers remains affine. ### Explanation and working $$ \sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ Sigmoid converts any finite real input into a value strictly between 0 and 1. At zero it is 0.5. It is useful for a binary-class probability or independent multilabel outputs. At very positive or negative inputs it saturates, so its derivative becomes small. Its derivative is at most 0.25. ![Trace the output of each curve as the same raw input moves from negative to positive.](https://deep-learning-study-room.vercel.app/assets/figures/v_activations.svg) Diagram: Trace the output of each curve as the same raw input moves from negative to positive.. Labels: See what each activation does | See what each activation does | The horizontal axis is the raw score z; the vertical axis is the output. | Sigmoid | 0 | z | 0 < output < 1 | Tanh | 0 | z | −1 < output < 1 | ReLU | 0 | z | Negative → 0; positive → z | The shape of an activation changes both the output and the gradient passed | backward. Trace the output of each curve as the same raw input moves from negative to positive. Tap the diagram to enlarge. $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1},\quad \tanh'(z)=1-\tanh^2(z) $$ Tanh lies between −1 and +1 and is zero-centered. It is common in recurrent hidden states. It also saturates. “Zero-centered” does not mean “no vanishing gradients”. $$ \operatorname{ReLU}(z)=\max(0,z),\quad f'(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ ReLU returns the input if positive and zero if negative. It is cheap to compute and has derivative 1 on its positive side. Units can become inactive on all relevant inputs, known as dying ReLU. At z = 0 the mathematical derivative is undefined; a convention such as 0 is used in implementations and this course’s exercises when needed. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\az&z<0\end{cases},\qquad a=0.01\ \text{unless specified} $$ The small negative slope a lets some gradient through for negative inputs. At z = −2 and a = 0.01, output = −0.02. Do not take the absolute value. PReLU learns the negative slope; Leaky ReLU normally fixes it. $$ p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Softmax acts on a group of class scores, called logits. All probabilities are positive and sum to 1. Subtracting the same maximum c from all logits leaves the result unchanged and avoids very large exponentials. It is suited to mutually exclusive multiclass output. **Full softmax example.** For [−1, 0, 1], exponentials are [0.367879, 1, 2.718282]. Sum = 4.086161. Divide each by that same sum: [0.090031, 0.244728, 0.665241]. Check their sum is approximately 1. Three separate sigmoids would give [0.268941, 0.5, 0.731059], which are different and do not sum to 1. ### Derivation of the sigmoid derivative Write sigmoid as a negative power. The outer derivative contributes −1, and differentiating the inner exponential contributes a second negative sign: $$ \sigma(z)=(1+e^{-z})^{-1} $$ $$ \sigma'(z)=-(1+e^{-z})^{-2}(-e^{-z})=\frac{e^{-z}}{(1+e^{-z})^2} $$ Separate the expression into two factors: $$ \sigma'(z)=\frac{1}{1+e^{-z}}\frac{e^{-z}}{1+e^{-z}}=\sigma(z)[1-\sigma(z)] $$ The second factor is 1−σ(z) because subtracting 1/(1+e⁻ᶻ) from 1 leaves e⁻ᶻ/(1+e⁻ᶻ). Once a forward pass has supplied σ(z), its derivative requires no further exponential evaluation. ### Derivation of the tanh derivative Use the quotient rule on the exponential definition. Here u = e²ᶻ−1 and v = e²ᶻ+1, with u′ = v′ = 2e²ᶻ. $$ \frac{d}{dz}\tanh z=\frac{2e^{2z}(e^{2z}+1)-2e^{2z}(e^{2z}-1)}{(e^{2z}+1)^2}=\frac{4e^{2z}}{(e^{2z}+1)^2} $$ The identity 1−tanh²(z) gives the same expression, since the numerator becomes (e²ᶻ+1)²−(e²ᶻ−1)² = 4e²ᶻ. Therefore the backward-pass factor is 1−a² when a = tanh(z). ### Why nonlinear activations are required For two affine layers without a nonlinear activation: $$ h=W_1x+b_1,\qquad \hat y=W_2h+b_2=W_2W_1x+(W_2b_1+b_2) $$ The composition is still one affine transformation. A nonlinear activation between layers changes the family of functions the network can represent. This is the mathematical reason for using nonlinear hidden activations; merely adding affine layers does not provide this expressive advantage. ### sigmoid example 1 **Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.** ### Given z = -2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(-2)=2 $$ **Step 2: calculate the exponential.** $$ e^{2}\approx7.389056099 $$ **Step 3: add one in the denominator.** $$ 1+7.389056099\approx8.389056099 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: use the activated output in the slope.** $$ f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585 $$ **Final answer:** a ≈ 0.119202922. Derivative ≈ 0.104993585. ### sigmoid example 2 **Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.** ### Given z = 0; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: add one in the denominator.** $$ 1+1.000000000\approx2.000000000 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: use the activated output in the slope.** $$ f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000 $$ **Final answer:** a ≈ 0.500000000. Derivative ≈ 0.250000000. ### sigmoid example 3 **Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.** ### Given z = 2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(2)=-2 $$ **Step 2: calculate the exponential.** $$ e^{-2}\approx0.135335283 $$ **Step 3: add one in the denominator.** $$ 1+0.135335283\approx1.135335283 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: use the activated output in the slope.** $$ f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585 $$ **Final answer:** a ≈ 0.880797078. Derivative ≈ 0.104993585. ### tanh example 1 **Question. For preactivation z = -2, calculate the tanh output and state its derivative.** ### Given z = -2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(-2)=-4 $$ **Step 2: calculate the exponential.** $$ e^{-4}\approx0.018315639 $$ **Step 3: calculate numerator and denominator separately.** $$ 0.018315639-1\approx-0.981684361 $$ $$ 0.018315639+1\approx1.018315639 $$ **Step 4: divide.** $$ a=\frac{-0.981684361}{1.018315639}\approx-0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(-2)=1-(-0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ -0.964027580. Derivative ≈ 0.070650825. ### tanh example 2 **Question. For preactivation z = 0, calculate the tanh output and state its derivative.** ### Given z = 0; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: calculate numerator and denominator separately.** $$ 1.000000000-1\approx0.000000000 $$ $$ 1.000000000+1\approx2.000000000 $$ **Step 4: divide.** $$ a=\frac{0.000000000}{2.000000000}\approx0.000000000 $$ **Step 5: square the activated output for its slope.** $$ f\prime(0)=1-(0.000000000)^2\approx1.000000000 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 1.000000000. ### tanh example 3 **Question. For preactivation z = 2, calculate the tanh output and state its derivative.** ### Given z = 2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(2)=4 $$ **Step 2: calculate the exponential.** $$ e^{4}\approx54.598150033 $$ **Step 3: calculate numerator and denominator separately.** $$ 54.598150033-1\approx53.598150033 $$ $$ 54.598150033+1\approx55.598150033 $$ **Step 4: divide.** $$ a=\frac{53.598150033}{55.598150033}\approx0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(2)=1-(0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ 0.964027580. Derivative ≈ 0.070650825. ### ReLU example 1 **Question. For preactivation z = -2, calculate the ReLU output and state its derivative.** ### Given z = -2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 0.000000000. ### ReLU example 2 **Question. For preactivation z = 0, calculate the ReLU output and state its derivative.** ### Given z = 0; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### ReLU example 3 **Question. For preactivation z = 2, calculate the ReLU output and state its derivative.** ### Given z = 2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ### Leaky ReLU example 1 **Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = -2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=-0.2 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0.1 $$ **Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000. ### Leaky ReLU example 2 **Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 0; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### Leaky ReLU example 3 **Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ### Softmax for [0, 0, 0] **Question. Find stable softmax probabilities for logits [0, 0, 0].** ### Given Logits = [0, 0, 0]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}0&0&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{0}\approx1.000000000 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{0}\approx1.000000000 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=1.000000000+1.000000000+1.000000000\approx3.000000000 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding. ### Softmax for [1, 2, 3] **Question. Find stable softmax probabilities for logits [1, 2, 3].** ### Given Logits = [1, 2, 3]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}-2&-1&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{-2}\approx0.135335283 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{-1}\approx0.367879441 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=0.135335283+0.367879441+1.000000000\approx1.503214724 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{0.135335283}{1.503214724}\approx0.090030573 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{0.367879441}{1.503214724}\approx0.244728471 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{1.503214724}\approx0.665240956 $$ **Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding. --- ## MT2 · Model answer: why a network needs nonlinear activation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s8 ## Question **Question.** Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations. ## Answer — step by step An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary. ![Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.](https://deep-learning-study-room.vercel.app/assets/figures/v_nonlinear.svg) Diagram: Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.. Labels: Two affine layers still make one affine rule | Two affine layers still make one affine rule | Without a nonlinear activation, these two stages can be combined. | Layer 1 and layer 2 | h = 2x + 1 | y = 3h + 4 | Substitute and simplify | y = 3(2x + 1) + 4 | y = 6x + 7 | A nonlinear activation between layers prevents this simple affine collapse. Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation. Tap the diagram to enlarge. ### Why two linear layers are still linear in the input For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second: $$ y=3(2x+1)+4=6x+3+4=6x+7 $$ This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent. | Activation | Output range | Useful property | Main limitation | | --- | --- | --- | --- | | Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small | | Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients | | ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient | Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score. **Exam conclusion:** select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction. --- ## MT2 · Choosing a loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s9 **Definition.** A loss function quantifies the discrepancy between a prediction and its target. The training objective aggregates per-example losses and may add regularization. The loss must be compatible with the task and output parameterization. ### Explanation and working Activation produces the prediction. Loss measures how wrong that prediction is. An optimizer uses gradients to update parameters. These are three different roles. ![First identify what the prediction means. Then choose the loss paired with that task and output.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_choice.svg) Diagram: First identify what the prediction means. Then choose the loss paired with that task and output.. Labels: Match the loss to what the output means | Match the loss to what the output means | A probability and an unrestricted numerical prediction need different | interpretations. | Predict a numerical quantity | Output: a price or temperature | Typical loss: MSE or MAE | Predict a class probability | Binary: sigmoid + BCE | Multiclass: softmax + CCE | The loss measures the mistake. The activation determines the output | transformation. First identify what the prediction means. Then choose the loss paired with that task and output. Tap the diagram to enlarge. $$ \operatorname{MSE}=\frac1n\sum_{i=1}^n(\hat y_i-y_i)^2,\qquad \operatorname{MAE}=\frac1n\sum_{i=1}^n|\hat y_i-y_i| $$ **Question.** Targets are [2, 4], predictions are [1, 6]. Find MSE and MAE. ### Given Targets = [2,4]; predictions = [1,6]; n = 2. ### To find Find mean squared error and mean absolute error. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. **Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes. n = number of predictions. |e| means the nonnegative magnitude of error e. $$ \mathrm{MAE}=\frac1n\sum_i|\hat y_i-y_i| $$ **Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square. ### Solution — step by step **Step 1: calculate prediction minus target.** $$ e_1=1-2=-1 $$ $$ e_2=6-4=2 $$ **Step 2: square both errors.** $$ e_1^2=(-1)^2=1 $$ $$ e_2^2=2^2=4 $$ **Step 3: add squares and divide by two observations.** $$ \mathrm{MSE}=\frac{1+4}{2}=2.5 $$ **Step 4: take the absolute value of each original error.** $$ |-1|=1,\qquad |2|=2 $$ **Step 5: add absolute errors and divide by two.** $$ \mathrm{MAE}=\frac{1+2}{2}=1.5 $$ **Final answer:** MSE = 2.5; MAE = 1.5. Squaring gives larger errors more influence; MAE is not differentiable at zero error. $$ L_{\mathrm{BCE}}=-[y\ln p+(1-y)\ln(1-p)],\qquad L_{\mathrm{CCE}}=-\sum_k y_k\ln p_k $$ For binary y = 1, BCE becomes −ln(p). For y = 0, it becomes −ln(1 − p). For one-hot multiclass labels, CCE is simply minus the natural log of the predicted probability assigned to the correct class. **Question.** True class is Mild, encoded [0,1,0], and probabilities are [0.1,0.8,0.1]. Find CCE. ### Given True class = Mild; one-hot target = [0,1,0]. Predicted probabilities = [0.1,0.8,0.1]. ### To find Find the categorical cross-entropy. ### Formula — which one and why **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step **Step 1: write one weighted log term for each class.** $$ L=-[0\ln(0.1)+1\ln(0.8)+0\ln(0.1)] $$ **Step 2: remove terms with zero target weight.** $$ L=-\ln(0.8) $$ **Step 3: evaluate the natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 4: apply the outer minus sign.** $$ L\approx-(-0.223143551)=0.223143551 $$ **Final answer:** CCE ≈ 0.223144. Only the correct-class probability contributes for this one-hot target. | Task | Typical output | Typical loss | | --- | --- | --- | | Unbounded numerical regression | One linear output | MSE or MAE | | Binary classification | One sigmoid output | Binary cross-entropy | | Exactly one of K classes | K softmax outputs | Categorical cross-entropy | | Several independent labels | K sigmoid outputs | Binary cross-entropy per label | A continuous rating restricted to 0–10 can use a linear output with a stated range policy, or 10 × sigmoid with an appropriate regression loss. If ratings are treated as discrete ordered categories, modelling choices differ. State the interpretation instead of selecting a loss merely from the number 10. ### Squared-error scaling in backpropagation For n included scalar predictions, let eᵢ = ŷᵢ−yᵢ. Three frequently used objectives are: $$ \mathrm{SSE}=\sum_i e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_i e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ Differentiating with respect to one prediction produces different factors: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_i}=2e_i,\quad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_i}=e_i,\quad \frac{\partial\mathrm{MSE}}{\partial\hat y_i}=\frac2n e_i $$ Use the exact objective in the question. The teacher's two-output backpropagation example uses half SSE. Do not automatically insert the MSE factor into that example. ### Why cross-entropy uses the negative logarithm For a one-hot target, the likelihood of the observed class is the probability assigned to it. Maximizing that likelihood is equivalent to minimizing its negative logarithm. For independent training examples, multiplying probabilities becomes adding their logarithms. A low correct-class probability therefore produces a large loss. ### Sigmoid with binary cross-entropy: output delta Let p = σ(z). First differentiate the loss with respect to p, holding y constant: $$ \frac{\partial L}{\partial p}=-\frac yp+\frac{1-y}{1-p}=\frac{p-y}{p(1-p)} $$ Then multiply by the sigmoid derivative: $$ \frac{\partial L}{\partial z}=\frac{\partial L}{\partial p}\frac{\partial p}{\partial z}=\frac{p-y}{p(1-p)}p(1-p)=p-y $$ Thus the logit delta is p−y for this paired output and loss. With sigmoid and half squared error it is instead (p−y)p(1−p). Confusing these two cases changes every subsequent gradient. ### Softmax with categorical cross-entropy: output delta For a normalized target vector, the derivative of softmax component pᵢ with respect to logit zⱼ is pᵢ times (δᵢⱼ−pⱼ), where δᵢⱼ is 1 if i = j and 0 otherwise. Therefore: $$ \frac{\partial L}{\partial z_j}=-\sum_i\frac{y_i}{p_i}p_i(\delta_{ij}-p_j)=-y_j+p_j\sum_i y_i=p_j-y_j $$ This result assumes the target components sum to 1. For a mean loss over a batch, include the corresponding averaging factor as well. ### Three squared-error conventions: [1, 3] **Question. Predictions are [1, 3] and targets are [2, 3]. Find SSE, half SSE and MSE.** ### Given Predictions = [1, 3]. Targets = [2, 3]. Number of scalar predictions n = 2. ### To find SSE, half SSE and MSE separately. ### Formula — which one and why All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention. ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Square sab mein; half mein 2, mean mein n se divide. ### Solution — step by step **Step 1: calculate error 1.** $$ e_1=1-2=-1 $$ **Step 2: square error 1.** $$ e_1^2=(-1)^2=1 $$ **Step 3: calculate error 2.** $$ e_2=3-3=0 $$ **Step 4: square error 2.** $$ e_2^2=(0)^2=0 $$ **Step 5: add squared errors.** $$ \mathrm{SSE}=1+0=1 $$ **Step 6: divide by 2 for half SSE.** $$ L_{\mathrm{half}}=\frac{1}{2}=0.5 $$ **Step 7: divide by the number of predictions for MSE.** $$ \mathrm{MSE}=\frac{1}{2}\approx0.500000000 $$ **Final answer:** SSE = 1; half SSE = 0.5; MSE ≈ 0.500000000. ### Three squared-error conventions: [2, 5, 1] **Question. Predictions are [2, 5, 1] and targets are [1, 3, 2]. Find SSE, half SSE and MSE.** ### Given Predictions = [2, 5, 1]. Targets = [1, 3, 2]. Number of scalar predictions n = 3. ### To find SSE, half SSE and MSE separately. ### Formula — which one and why All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention. ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Square sab mein; half mein 2, mean mein n se divide. ### Solution — step by step **Step 1: calculate error 1.** $$ e_1=2-1=1 $$ **Step 2: square error 1.** $$ e_1^2=(1)^2=1 $$ **Step 3: calculate error 2.** $$ e_2=5-3=2 $$ **Step 4: square error 2.** $$ e_2^2=(2)^2=4 $$ **Step 5: calculate error 3.** $$ e_3=1-2=-1 $$ **Step 6: square error 3.** $$ e_3^2=(-1)^2=1 $$ **Step 7: add squared errors.** $$ \mathrm{SSE}=1+4+1=6 $$ **Step 8: divide by 2 for half SSE.** $$ L_{\mathrm{half}}=\frac{6}{2}=3 $$ **Step 9: divide by the number of predictions for MSE.** $$ \mathrm{MSE}=\frac{6}{3}\approx2.000000000 $$ **Final answer:** SSE = 6; half SSE = 3; MSE ≈ 2.000000000. ### Three squared-error conventions: [4] **Question. Predictions are [4] and targets are [1]. Find SSE, half SSE and MSE.** ### Given Predictions = [4]. Targets = [1]. Number of scalar predictions n = 1. ### To find SSE, half SSE and MSE separately. ### Formula — which one and why All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention. ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Square sab mein; half mein 2, mean mein n se divide. ### Solution — step by step **Step 1: calculate error 1.** $$ e_1=4-1=3 $$ **Step 2: square error 1.** $$ e_1^2=(3)^2=9 $$ **Step 3: add squared errors.** $$ \mathrm{SSE}=9=9 $$ **Step 4: divide by 2 for half SSE.** $$ L_{\mathrm{half}}=\frac{9}{2}=4.5 $$ **Step 5: divide by the number of predictions for MSE.** $$ \mathrm{MSE}=\frac{9}{1}\approx9.000000000 $$ **Final answer:** SSE = 9; half SSE = 4.5; MSE ≈ 9.000000000. ### Binary cross-entropy: target 1, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.8 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.223143551)=0.223143551 $$ **Final answer:** Loss ≈ 0.223143551 nats. ### Binary cross-entropy: target 0, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 0; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ 1-p=1-0.8=0.2 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.2)\approx-1.609437912 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-1.609437912)=1.609437912 $$ **Final answer:** Loss ≈ 1.609437912 nats. ### Binary cross-entropy: target 1, predicted positive-class probability 0.5 **Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.5; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.5 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.693147181)=0.693147181 $$ **Final answer:** Loss ≈ 0.693147181 nats. ### Sigmoid+BCE logit gradient: p=0.8, y=1 **Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 1; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(-1.25)(0.16)=-0.2 $$ **Final answer:** dL/dz = -0.2. No further sigmoid derivative is needed. ### Sigmoid+BCE logit gradient: p=0.8, y=0 **Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 0; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-0}{0.16}=5 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(5)(0.16)=0.8 $$ **Final answer:** dL/dz = 0.8. No further sigmoid derivative is needed. --- ## MT2 · Forward propagation through multiple neurons Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s10/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s10 **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations. ### Input, hidden and output layers The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task. In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections. ![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg) Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights. A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge. ### General layer equations and dimensions Let a⁽⁰⁾ = x. For layer l: $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f^{(l)}(z^{(l)}) $$ If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j. $$ W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1} $$ For one hidden layer, these computations compose to give: $$ h=f^{(1)}(W^{(1)}x+b^{(1)}) $$ $$ \hat y=f^{(2)}(W^{(2)}h+b^{(2)}) $$ The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer. ### Forward propagation versus training Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture. ### Complete two-layer numerical **Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.** ### Given A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. ### To find Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg) Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge. **Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output. **Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias. $$ z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3 $$ **Step 3: calculate the second hidden preactivation.** $$ z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5 $$ **Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them. $$ a^{(1)}_1=\max(0,0.3)=0.3 $$ $$ a^{(1)}_2=\max(0,0.5)=0.5 $$ $$ a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix} $$ **Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs. $$ z^{(2)}=0.5(0.3)+0.6(0.5)+0.1 $$ $$ z^{(2)}=0.15+0.30+0.10=0.55 $$ **Step 6: apply the output activation.** $$ \hat y=\max(0,0.55)=0.55 $$ **Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass. --- ## MT2 · Backpropagation with one hidden neuron Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s11/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s11 **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. --- ## MT2 · More outputs and vectorized backpropagation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s12/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s12 **Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path. ### Why the hidden derivative is a sum If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives: $$ \frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2} $$ For any number of connected outputs, sum their contributions and then differentiate the hidden activation: $$ \delta_{h_j}=\left(\sum_k W_{2,kj}\delta_{o_k}\right)f'(z_{h_j}) $$ Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss. ![The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_branch_gradient.svg) Diagram: The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.. Labels: Two output paths both affect the hidden neuron | Two output paths both affect the hidden neuron | Example: hidden activation h = 0.6, using sigmoid in the hidden neuron. | h = 0.6 | Output 1 delta = 0.2 | Connecting weight = 0.4 | Output 2 delta = −0.1 | Connecting weight = 0.5 | Downstream sum = 0.2 × 0.4 + (−0.1) × 0.5 = 0.03 | Hidden delta = 0.03 × 0.6 × (1 − 0.6) = 0.0072 | Add the two downstream paths first. Then multiply by the hidden activation | derivative. The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative. Tap the diagram to enlarge. ### Matrix equations for one training sample Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂: $$ L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2 $$ $$ \delta_2=(\hat y-y)\odot f_2'(z_2) $$ $$ \frac{\partial L}{\partial W_2}=\delta_2a_1^T,\qquad\frac{\partial L}{\partial b_2}=\delta_2 $$ $$ \delta_1=(W_2^T\delta_2)\odot f_1'(z_1) $$ $$ \frac{\partial L}{\partial W_1}=\delta_1x^T,\qquad\frac{\partial L}{\partial b_1}=\delta_1 $$ Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation. | Quantity | Shape | Reason | | --- | --- | --- | | W₁ and its gradient | m × d | m receiving hidden neurons, d inputs | | W₂ and its gradient | k × m | k receiving output neurons, m hidden activations | | δ₂ | k × 1 | One derivative per output preactivation | | W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit | | δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ | ### How the loss changes the output delta For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target: $$ \delta_{\mathrm{output}}=\hat y-y $$ That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required. For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question. --- ## MT2 · Model answer: forward propagation, backpropagation and updating Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s13/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s13 ## Question **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations. ## Answer — step by step Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs. ![Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge. | Operation | Direction or action | Result | | --- | --- | --- | | Forward propagation | Inputs move through successive layers | Predictions and intermediate activations | | Loss calculation | Compare predictions with targets | A scalar objective for the example or batch | | Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters | | Optimizer update | Use those gradients to change parameters | A new set of weights and biases | ### 1. Forward propagation At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ ### 2. Loss Use the loss specified by the task. For one regression output, half squared loss is one possible choice: $$ L=\frac12(a-y)^2 $$ Here a is the prediction and y is the target. Other stated losses produce different derivatives. ### 3. Backpropagation Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it. ### 4. Update and repeat $$ w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w} $$ Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations. --- ## MT2 · Counting parameters in a dense network Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s14/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s14 **Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases. ### Deriving the parameter count For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters: $$ P_{\mathrm{dense}}=mn+n=(m+1)n $$ Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram. ![Count all input-to-neuron connections and then one bias per receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/v_dense_count.svg) Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters. Count all input-to-neuron connections and then one bias per receiving neuron. Tap the diagram to enlarge. ### Complete numerical **Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters. ### Given Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs. Every hidden/output neuron has one bias. ### To find Count weights and biases at each receiving layer, then the total. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step **Step 1: identify the consecutive layer widths.** $$ 10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4 $$ There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters. **Step 2: count weights and biases for each transformation.** | Transformation | Weights | Biases | Layer total | | --- | --- | --- | --- | | Input → hidden 1 | 10 × 5 = 50 | 5 | 55 | | Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 | | Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 | | Hidden 3 → output | 5 × 4 = 20 | 4 | 24 | **Step 3: add the layer totals.** $$ P=55+30+30+24=139 $$ Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139. **Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network. --- ## MT2 · What you should understand before optimization Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s15/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s15 You should be able to explain why a target and prediction differ, compute a weighted sum, choose an activation, measure a loss, find a one-neuron gradient and update a weight. If these steps are weak, repeat the first training example and the ReLU backprop example. They contain the entire calculation pattern that the next chapters extend. ![These four operations form the training loop used by the optimizers in the next chapter.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: These four operations form the training loop used by the optimizers in the next chapter.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. These four operations form the training loop used by the optimizers in the next chapter. Tap the diagram to enlarge. --- ## MT2 · Complete sigmoid training example Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s16/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s16 **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. --- ## MT2 · Cross-entropy for probability distributions Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c4-s17/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/4/mt2-c4-s17 **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities. $$ H(p,q)=-\sum_{i=1}^{K}p_i\log q_i $$ Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms. ### Logarithm base and units Natural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2: $$ \log_2 q=\frac{\ln q}{\ln 2} $$ For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities. ### A target distribution with two nonzero entries **Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.** ### Given A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms. ### To find Find cross-entropy in the requested units. ### Formula — which one and why **Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute. pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ H_2(p,q)=\frac{H(p,q)}{\ln2} $$ **Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai. ### Solution — step by step **Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded. $$ H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)] $$ **Step 2: calculate the logarithms.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: weight each negative logarithm by its target probability.** $$ -0.75\ln(0.5)\approx0.519860385 $$ $$ -0.25\ln(0.5)\approx0.173286795 $$ **Step 4: add the contributions, retaining calculator precision.** $$ H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats} $$ **Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1. $$ H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit} $$ **Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale. ### Cross-entropy, entropy and relative entropy These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p) $$ For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus. **Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.** ### Given For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats. ### To find Find entropy and KL divergence in nats. ### Formula — which one and why **Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p. p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p) $$ **Yaad rakhne ka cue:** KL = cross-entropy minus target ki own entropy. ### Solution — step by step **Step 1: calculate the target entropy.** $$ H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)] $$ $$ \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361 $$ $$ H(p)\approx0.215761554+0.346573590=0.562335145 $$ **Step 2: subtract target entropy from cross-entropy.** $$ D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036 $$ **Final answer:** H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ. --- ## MT2 · Reading an optimizer formula without getting lost Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s0 An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated. ![Old value, current gradient and new value belong to different moments in the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. Old value, current gradient and new value belong to different moments in the update. Tap the diagram to enlarge. ### Old, current and new At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position. $$ \theta_t=\theta_{t-1}-\eta g_t $$ η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition. ### What a stored state means An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares. ### A weighted average from ordinary arithmetic If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square. $$ 0.9\times4=3.6,\qquad(1-0.9)\times9=0.9 $$ $$ s_{\mathrm{new}}=3.6+0.9=4.5 $$ The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum. ### Squared memory and signed direction For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g². ### Epsilon and hats ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second. --- ## MT2 · Optimizer notation: old value, memory, new value Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s1 ### 1. A parameter and a stored state are different θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries. ![The time label tells you which value to use. Carry the previous memory into the current update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: The time label tells you which value to use. Carry the previous memory into the current update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. The time label tells you which value to use. Carry the previous memory into the current update. Tap the diagram to enlarge. At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values. ### 2. A supplied gradient versus a supplied objective If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead. For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64. ### 3. A squared gradient keeps size and loses sign Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square. ### 4. Square root and epsilon belong to the denominator If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero. ### 5. A table prevents mixing iterations Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1. The worked examples below use this exact order, with every state shown. --- ## MT2 · What an optimizer is trying to do Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s2 **Definition.** Optimization is the process of selecting model parameters to minimize a scalar objective. In neural-network training, a gradient-based optimizer updates weights and biases using derivatives of the training objective. ### Objective and gradient For empirical risk with an optional regularizer: $$ J(\theta)=\frac1N\sum_{i=1}^{N}L_i(\theta)+\lambda\Omega(\theta) $$ The loss value indicates the current objective. Its gradient indicates local sensitivity to parameters. Backpropagation computes this gradient efficiently; the optimizer determines how the parameters are updated from it. A gradient and a loss value cannot be interchanged in the update equation. ### Why gradient descent subtracts the gradient For a small displacement Δθ, first-order Taylor expansion approximates the objective by: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ Choose Δθ = −η∇J(θ), with η > 0. Substitution gives: $$ J(\theta-\eta\nabla J)\approx J(\theta)-\eta\|\nabla J\|_2^2 $$ The squared norm is nonnegative. Therefore the negative-gradient direction is locally a descent direction when the gradient is nonzero. The approximation explains the sign; it does not guarantee that an arbitrarily large finite step decreases loss. ![A gradient specifies a local direction; the learning rate determines the finite parameter step.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: A gradient specifies a local direction; the learning rate determines the finite parameter step.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. A gradient specifies a local direction; the learning rate determines the finite parameter step. Tap the diagram to enlarge. ### Local minima, global minima and saddle points A local minimum has objective value no larger than nearby points. A global minimum is minimal over the whole feasible domain. At a saddle point, the objective increases in some directions and decreases in others. A small gradient alone does not establish a minimum: a stationary point can also be a maximum or saddle. Neural-network objectives are generally nonconvex. Training can encounter flat regions, ill-conditioned valleys, noisy gradient estimates and poor step sizes. Momentum modifies the update direction using previous steps; adaptive methods rescale parameter-specific updates. These methods improve particular optimization behavior, without guaranteeing the global optimum or best test performance. --- ## MT2 · The ordinary gradient descent update Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s3 **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate. ### Explanation and working $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent. Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction. ![Calculate the slope first, multiply it by the learning rate, then subtract that correction.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: Calculate the slope first, multiply it by the learning rate, then subtract that correction.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. Calculate the slope first, multiply it by the learning rate, then subtract that correction. Tap the diagram to enlarge. **Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations. ### Given J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1. ### To find Differentiate J and perform two gradient-descent iterations. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ². **Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79. **Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061. **Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**. The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum. **Common error:** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients. --- ## MT2 · Batch stochastic and mini batch gradient descent Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s4 **Definition.** Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ. ### Explanation and working All three use the same update idea. They differ in how much data contributes to one gradient. ![Count outlined groups, not individual cells, to count gradient updates in one epoch.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: Count outlined groups, not individual cells, to count gradient updates in one epoch.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. Count outlined groups, not individual cells, to count gradient updates in one epoch. Tap the diagram to enlarge. | Method | Data for one update | Updates in one epoch with N samples | Typical behaviour | | --- | --- | --- | --- | | Batch GD | All N samples | 1 | Stable full-data gradient; costly updates | | Stochastic GD | One sample | N | Frequent noisy updates | | Mini-batch GD | B samples | Ceiling of N/B if final partial batch is retained | Efficient parallel calculation with moderate noise | **Question.** N = 1,000 and B = 100. Find updates per epoch. ### Given N = 1,000 examples; mini-batch size B = 100; one epoch. ### To find Count full-batch, stochastic and mini-batch updates per epoch. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step **Step 1: count full-batch updates.** All 1,000 samples form one group, so one epoch has 1 update. **Step 2: count stochastic updates.** One sample per update gives 1,000 updates per epoch. **Step 3: count mini-batches.** $$ \frac{N}{B}=\frac{1000}{100}=10 $$ **Final answer:** batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch. ### Mini-batch calculation with a remainder **Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.** ### Given N = 105; B = 20; retain the final incomplete batch. ### To find The number of updates per epoch. ### Formula — which one and why Every processed group causes one update. A remaining nonempty group also counts. The ceiling symbol rounds upward to the next integer. $$ U=\lceil N/B\rceil $$ **Yaad rakhne ka cue:** Bacha hua batch process kiya toh ek aur update. ### Solution — step by step **Step 1: count complete batches.** $$ 5(20)=100 $$ **Step 2: count remaining examples.** $$ 105-100=5 $$ **Step 3: include the nonempty last batch.** $$ U=5+1=6 $$ **Final answer:** 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5. --- ## MT2 · Model answer: batch, stochastic and mini-batch training Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s5 ## Question **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration. ## Answer — step by step All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update. ![All methods visit the same six examples. Their grouping determines how often they update.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: All methods visit the same six examples. Their grouping determines how often they update.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge. A **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations. | Basis | Batch GD | Stochastic GD | Mini-batch GD | | --- | --- | --- | --- | | Examples in one gradient | Entire training dataset | One example | A smaller group of B examples | | Updates per epoch | One | N | Ceiling of N/B if the remainder is retained | | Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging | | Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation | ### Example to distinguish the three counts With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch. The basic parameter update is unchanged: $$ \theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g $$ For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples. --- ## MT2 · A tiny dataset shows why the methods differ Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s6 **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order. ### Given Model ŷ = wx; initial w = 1; η = 0.1. Samples in order: (1,2), (2,4). Mean half squared loss for the batch. ### To find Compare one batch update with a sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### One batch update **Step 1: predict sample 1 with the original weight.** $$ \hat y_1=1(1)=1 $$ **Step 2: calculate its error.** $$ e_1=1-2=-1 $$ **Step 3: multiply its error by its input.** $$ g_1=(-1)(1)=-1 $$ **Step 4: predict sample 2 with the same original weight.** $$ \hat y_2=1(2)=2 $$ **Step 5: calculate its error.** $$ e_2=2-4=-2 $$ **Step 6: multiply its error by its input.** $$ g_2=(-2)(2)=-4 $$ **Step 7: average the two gradients.** $$ g=\frac{-1+(-4)}2=-2.5 $$ **Step 8: update once.** $$ w_{\mathrm{new}}=1-0.1(-2.5)=1.25 $$ ![This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge. ### One sequential SGD epoch **Step 1: use the first sample gradient.** $$ g_1=-1 $$ **Step 2: update after the first sample.** $$ w_1=1-0.1(-1)=1.1 $$ **Step 3: predict sample 2 using the changed weight.** $$ \hat y_2=1.1(2)=2.2 $$ **Step 4: find sample 2 error.** $$ e_2=2.2-4=-1.8 $$ **Step 5: find the new sample 2 gradient.** $$ g_2=(-1.8)(2)=-3.6 $$ **Step 6: update after sample 2.** $$ w_2=1.1-0.1(-3.6)=1.46 $$ **Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples. Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim. --- ## MT2 · Learning rate and common training problems Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s7 **Definition.** The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss. ### Explanation and working A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate. ![Illustrative loss traces show why smoothness and progress are different](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative loss traces show why smoothness and progress are different. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative loss traces show why smoothness and progress are different Tap the diagram to enlarge. | Observation | Possible reason | Sensible first action | | --- | --- | --- | | Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation | | Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate | | Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum | | Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping | | All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights | Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum. --- ## MT2 · A moving average remembers the recent past Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s8 **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients. $$ m_t=\beta m_{t-1}+(1-\beta)g_t,\qquad0\le\beta<1 $$ For m₀ = 0, expanding the recurrence gives: $$ m_t=(1-\beta)\sum_{k=1}^{t}\beta^{t-k}g_k $$ The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history. ### Signed gradients versus squared gradients An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form. **Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂. ### Given β = 0.9; initial m₀ = 0. Gradients in order: g₁ = 2, g₂ = −1. ### To find Find m₁ and m₂, preserving the sign of each gradient. ### Formula — which one and why **Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state. β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state. $$ m_t=\beta m_{t-1}+(1-\beta)g_t $$ **Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai. ### Solution — step by step **Step 1: find the recent-gradient coefficient.** $$ 1-0.9=0.1 $$ **Step 2: retain the old memory for step 1.** $$ 0.9(0)=0 $$ **Step 3: weight the first gradient.** $$ 0.1(2)=0.2 $$ **Step 4: add to obtain the first memory.** $$ m_1=0+0.2=0.2 $$ **Step 5: retain the first memory for step 2.** $$ 0.9(0.2)=0.18 $$ **Step 6: weight the second gradient.** $$ 0.1(-1)=-0.1 $$ **Step 7: add to obtain the second memory.** $$ m_2=0.18+(-0.1)=0.08 $$ **Final answer:** m₁ = 0.2; m₂ = 0.08. The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable. --- ## MT2 · Momentum remembers movement Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s9 **Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions. ### Update equations and convention This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter: $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters. ![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg) Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients. Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration. **Step 1: Iteration 1: gradient, velocity, parameter.** $$ g_1=J'(1)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Iteration 2: use θ₁ and v₁.** $$ g_2=J'(0.9)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ | Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter | | --- | --- | --- | --- | --- | | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.9 | −0.18 | 0.72 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient. ### Properties and alternative notation Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective. Some texts store a positive gradient accumulator u instead: $$ u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t $$ With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions. --- ## MT2 · Nesterov looks before correcting Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s10/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s10 **Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update. ### What changes relative to momentum Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured: $$ \theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look},t}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁. ![Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective. **Iteration 1 — step 1: locate the lookahead point.** $$ \theta_{\mathrm{look},1}=1+0.9(0)=1 $$ **Step 2: evaluate the gradient there.** $$ g_1=J'(1)=1 $$ **Step 3: calculate the new velocity and actual parameter.** $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Iteration 2 — step 1: use the previous actual parameter and velocity.** $$ \theta_{\mathrm{look},2}=0.9+0.9(-0.1)=0.81 $$ **Step 2: evaluate the gradient at 0.81, not at 0.9.** $$ g_2=J'(0.81)=0.81 $$ **Step 3: calculate the new velocity.** $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ **Step 4: update the old actual parameter.** $$ \theta_2=0.9+(-0.171)=0.729 $$ | Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ | | --- | --- | --- | --- | --- | --- | | 1 | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice. --- ## MT2 · Model answer: momentum and Nesterov Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s11/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s11 ## Question **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ? ## Answer — step by step Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions. ![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge. ### Momentum formulas This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained. $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ### Nesterov's change Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look}}) $$ The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point. | Basis | Momentum | Nesterov | | --- | --- | --- | | Gradient evaluated at | Current parameter | Lookahead parameter | | Information retained | Previous velocity | Previous velocity | | Intuition | Continue useful movement | Inspect the slope where old momentum points | | Initial zero velocity | No movement contribution yet | First lookahead equals current parameter | Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently. --- ## MT2 · AdaGrad adds every squared gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s12/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s12 **Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator. ### State, effective learning rate and update For one parameter θ, let gₜ be its current gradient and s₀ = 0: $$ s_t=s_{t-1}+g_t^2 $$ $$ \eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon} $$ $$ \theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t $$ s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign. ![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg) Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update. The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge. ### Complete numerical: supplied gradients **Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter. **Iteration 1 — step 1: update the accumulator.** $$ s_1=0+2^2=4 $$ **Step 2: calculate the effective learning rate.** $$ \eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05 $$ **Step 3: apply the signed gradient.** $$ \theta_1=1-0.05(2)=0.9 $$ **Iteration 2 — step 1: retain s₁ and add the new squared gradient.** $$ s_2=4+1^2=5 $$ **Step 2: calculate the new coefficient.** $$ \eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360 $$ **Step 3: update from θ₁.** $$ \theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640 $$ **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5. ### Advantage, limitation and epsilon convention Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes. ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical. --- ## MT2 · RMSProp remembers recent squared gradients Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s13/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s13 **Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator. ### Recurrence and interpretation Using decay β and zero initial state: $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update. If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever. ![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg) Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient. Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β = 0.9. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step **Iteration 1 — step 1: calculate the squared-gradient average.** $$ s_1=0.9(0)+0.1(2^2)=0.4 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_1}=\sqrt{0.4}\approx0.632455532 $$ $$ \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766 $$ **Step 3: update the parameter.** $$ \theta_1=1-0.316227766\approx0.683772234 $$ **Iteration 2 — step 1: use s₁, not the original zero state.** $$ s_2=0.9(0.4)+0.1(1^2)=0.46 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_2}=\sqrt{0.46}\approx0.678232998 $$ $$ \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956 $$ **Step 3: update from θ₁, retaining calculator precision.** $$ \theta_2\approx0.683772234-0.147441956\approx0.536330278 $$ **Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46. ### Comparison with AdaGrad and Adam AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum. --- ## MT2 · Adam remembers both direction and squared size Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s14/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s14 **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. --- ## MT2 · Adam bias correction: why the denominator is one minus beta to a power Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s15/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s15 **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate. ### Explanation and working ### 1. What the moving average actually contains Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization. ![The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam_correction.svg) Diagram: The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.. Labels: Correct the early zero-initialized memory | Correct the early zero-initialized memory | Example: β₁ = 0.9, first gradient = 2, initial m = 0. | Raw memory | m₁ = 0.9 × 0 + 0.1 × | 2 = 0.2 | Correction factor | 1 − 0.9¹ = 0.1 | Corrected memory | m̂₁ = 0.2 ÷ 0.1 = 2 | The small raw value is partly caused by the starting zero. Bias correction | removes that effect. The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor. Tap the diagram to enlarge. $$ m_1=(1-\beta)g_1 $$ At the second step, substitute the first expression into the recurrence: $$ m_2=\beta(1-\beta)g_1+(1-\beta)g_2 $$ ### 2. Add the coefficients After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term: $$ \beta(1-\beta)+(1-\beta)=(1-\beta)(1+\beta)=1-\beta^2 $$ After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ The squared-gradient memory uses its own decay β₂ and the same reasoning: $$ \widehat s_t=\frac{s_t}{1-\beta_2^t} $$ ### 3. A constant-gradient example makes the purpose visible Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38. $$ \widehat m_1=\frac{0.2}{1-0.9}=2 $$ $$ \widehat m_2=\frac{0.38}{1-0.9^2}=\frac{0.38}{0.19}=2 $$ Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient. ### 4. Two betas mean two separate powers At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators. ### 5. Correct first, then combine The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows. This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude. --- ## MT2 · Model answer: AdaGrad, RMSProp and Adam Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s16/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s16 ## Question **Question.** Compare AdaGrad, RMSProp and Adam. Explain what each method stores and how that memory changes a parameter update. ## Answer — step by step AdaGrad, RMSProp and Adam are adaptive gradient-based optimizers. Each stores separate state for each trainable parameter and uses that state to scale its update. Let gₜ be the current gradient for parameter θ, η the base learning rate and ε a small positive stabilizer. The equations below put ε outside the square root. ### AdaGrad: cumulative squared-gradient history $$ s_t=s_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Starting at s₀ = 0, every squared gradient contributes permanently. Parameters with frequent large gradients acquire smaller effective learning rates. This can be useful for sparse features, but the growing denominator can make later updates too small. The base learning rate still has to be chosen. ### RMSProp: a decaying squared-gradient average $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Old contributions decay by repeated multiplication by β. The state can decrease when recent squared gradients are small, avoiding AdaGrad's permanent accumulation. Basic RMSProp has one squared-gradient state and no separate signed-gradient average. ### Adam: first and second moments with bias correction Initialize m₀ = s₀ = 0 and count the first update as t = 1: $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad\hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The first moment stores signed directional information; the second stores squared-gradient scale. Correction compensates for their zero initialization. The corrected first moment is already the numerator's gradient estimate, so the update is not multiplied by gₜ again. Raw moments are carried into the next iteration. | Aspect | AdaGrad | RMSProp | Adam | | --- | --- | --- | --- | | Squared-gradient history | Cumulative sum | Decaying average | Decaying average | | Separate signed-gradient average | No | No in the basic method | Yes | | Standard initialization correction | No | No in the basic method | Corrects both moments | | State values per parameter | One | One | Two | | Main concern | Effective rate can become very small | Learning rate and decay need tuning | More state; hyperparameters still matter | All three adapt updates coordinate by coordinate. None removes the need for validation or guarantees a global minimum for a general neural-network objective. --- ## MT2 · One comparison to rebuild all optimizer formulas Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s17/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s17 | Method | What is remembered | What changes the step | | --- | --- | --- | | GD or SGD | No optimizer memory in the basic rule | Current gradient × η | | Momentum | Previous movement | Old velocity plus current correction | | Nesterov | Previous movement | Gradient evaluated at lookahead | | AdaGrad | Sum of all squared gradients | Divide current gradient by accumulated root | | RMSProp | Moving average of squared gradients | Divide current gradient by recent root | | Adam | Moving averages of gradients and squares | Divide corrected mean by corrected root | **Working order.** Start each method on a separate line with its states initialized. The solved examples use the same supplied gradients to show how their stored states differ. They do not have to give the same parameter value. The prescribed gradient sequence is practice data; if a question instead gives J(θ), recalculate its gradient at the appropriate current or lookahead parameter. ![Compare the memory rules first. Then use the matching memory in the denominator of the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad_rms.svg) Diagram: Compare the memory rules first. Then use the matching memory in the denominator of the update.. Labels: What happens to an old squared gradient? | What happens to an old squared gradient? | Start with old memory 4. The current gradient is −3, so its square is 9. | AdaGrad: keep adding | Old sum G + new g² | 4 + 9 = 13 | An old contribution stays. | RMSProp: decay and add | ρs + (1 − ρ)g² | 0.9 × 4 + 0.1 × 9 = 4.5 | Old contributions fade. | Both divide the signed current gradient by a square-root scale plus epsilon. Compare the memory rules first. Then use the matching memory in the denominator of the update. Tap the diagram to enlarge. --- ## MT2 · AdaGrad and RMSProp with a negative gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c5-s18/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/5/mt2-c5-s18 The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction. ![The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad_rms.svg) Diagram: The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory.. Labels: What happens to an old squared gradient? | What happens to an old squared gradient? | Start with old memory 4. The current gradient is −3, so its square is 9. | AdaGrad: keep adding | Old sum G + new g² | 4 + 9 = 13 | An old contribution stays. | RMSProp: decay and add | ρs + (1 − ρ)g² | 0.9 × 4 + 0.1 × 9 = 4.5 | Old contributions fade. | Both divide the signed current gradient by a square-root scale plus epsilon. The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory. Tap the diagram to enlarge. For an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9. **AdaGrad:** add the new square to the entire old sum: 4 + 9 = 13. **RMSProp with decay 0.9:** keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5. AdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training. --- ## MT2 · Recognizing overfitting and underfitting Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s0 **Definitions.** Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details. ### Interpreting learning curves Underfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves. ![Compare training and validation behavior when selecting model capacity or a stopping point.](https://deep-learning-study-room.vercel.app/assets/figures/overfitting.svg) Diagram: Compare training and validation behavior when selecting model capacity or a stopping point.. Labels: Loss | Epochs | Validation | Training | Best validation region | Save the best validation checkpoint; do not choose by test loss. Compare training and validation behavior when selecting model capacity or a stopping point. Tap the diagram to enlarge. | Observation | Plausible explanation | Appropriate response | | --- | --- | --- | | Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization | | Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping | | Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude | ### Bias–variance perspective Insufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data. The validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation. --- ## MT2 · L1 and L2 add a cost for large weights Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s1 **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. --- ## MT2 · Model answer: regularization and the L1–L2 difference Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s2 ## Question **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength. ## Answer — step by step Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost. ![Follow the stated penalty convention, including lambda and any half factor, when you form the objective.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: Follow the stated penalty convention, including lambda and any half factor, when you form the objective.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge. ### Objectives used in this course $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question. | Basis | L1 | L2 | | --- | --- | --- | | Quantity penalized | Absolute weight magnitude | Squared weight magnitude | | Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention | | Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes | | Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights | The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total. Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective. --- ## MT2 · Dropout removes activations during training Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s3 **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training. ### Explanation and working Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count. ![Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout.svg) Diagram: Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.. Labels: Dropout temporarily removes selected activations | Dropout temporarily removes selected activations | Example: keep probability q = 0.5. Inverted dropout during training. | Original | 0.8 | -0.4 | 0.6 | 0.2 | Mask | 1 | 0 | 1 | 0 | After mask ÷ q | 1.6 | 0 | 1.2 | 0 | The kept values double: 0.8 ÷ 0.5 = 1.6 and 0.6 ÷ 0.5 = 1.2. At inference, no | mask is used. Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge. Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions. $$ a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout} $$ $$ a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout} $$ **Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs. ### Given Activations = [0.8,−0.4,0.6,0.2]. Dropout rate = 50%, so keep probability q = 0.5. Chosen mask = [1,0,1,0]. ### To find Find training outputs under inverted dropout. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step **Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key. **Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer. Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition. --- ## MT2 · Early stopping uses the best validation point Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s4 **Definition.** Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint. ### Explanation and working Monitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set. ![Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum. Tap the diagram to enlarge. **Question.** Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore? ### Given Validation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55]. Patience = 2; no required minimum improvement. ### To find Find the stop epoch and the best checkpoint to restore. ### Formula — which one and why **Early stopping rule.** Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint. Patience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement. **Yaad rakhne ka cue:** Best validation save; patience khatam toh stop aur best restore. ### Solution — step by step **Step 1: process epoch 1.** Loss 0.8 establishes the first best checkpoint; wait = 0. **Step 2: process epoch 2.** 0.6 < 0.8: save epoch 2; wait = 0. **Step 3: process epoch 3.** 0.5 < 0.6: save epoch 3; wait = 0. **Step 4: process epoch 4.** 0.53 > 0.5: no improvement; wait = 1. **Step 5: process epoch 5.** 0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop. **Step 6: restore the best checkpoint.** The smallest observed validation loss is 0.5 at epoch 3. Restore its weights. **Final answer:** stop after epoch 5; restore epoch 3. These are different epochs. ### Early stopping with [0.8, 0.7, 0.72, 0.68, 0.69, 0.7] **Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.** ### Given Validation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. Patience = 2; smaller loss is better; ties are not improvements. ### To find The stopping epoch and best checkpoint. ### Formula — which one and why Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs. Patience counts consecutive non-improvements. A new best resets the count. **Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore. ### Solution — step by step **Step 1: process epoch 1.** Validation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8. **Step 2: process epoch 2.** Validation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7. **Step 3: process epoch 3.** Validation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7. **Step 4: process epoch 4.** Validation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 4, loss 0.68. **Step 5: process epoch 5.** Validation loss = 0.69; no strict improvement: wait count becomes 1. Best checkpoint is epoch 4, loss 0.68. **Step 6: process epoch 6.** Validation loss = 0.7; no strict improvement: wait count becomes 2. Best checkpoint is epoch 4, loss 0.68. **Final answer:** Stop after epoch 6; restore epoch 4, with validation loss 0.68. ### Early stopping with [0.6, 0.6, 0.61, 0.62] **Question. Validation losses are [0.6, 0.6, 0.61, 0.62]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.** ### Given Validation losses in epoch order = [0.6, 0.6, 0.61, 0.62]. Patience = 2; smaller loss is better; ties are not improvements. ### To find The stopping epoch and best checkpoint. ### Formula — which one and why Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs. Patience counts consecutive non-improvements. A new best resets the count. **Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore. ### Solution — step by step **Step 1: process epoch 1.** Validation loss = 0.6; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.6. **Step 2: process epoch 2.** Validation loss = 0.6; no strict improvement: wait count becomes 1. Best checkpoint is epoch 1, loss 0.6. **Step 3: process epoch 3.** Validation loss = 0.61; no strict improvement: wait count becomes 2. Best checkpoint is epoch 1, loss 0.6. **Final answer:** Stop after epoch 3; restore epoch 1, with validation loss 0.6. --- ## MT2 · Model answer: generalization and early stopping Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s5 ## Question **Question.** Distinguish underfitting and overfitting. Explain how validation data and early stopping help control overfitting. ## Answer — step by step Underfitting occurs when a model has not captured enough useful structure in the training data. Both training and validation errors are usually high relative to a sensible task baseline. Causes can include insufficient capacity, inadequate features, excessive regularization or incomplete optimization. ![The best training score is not necessarily the best score on unseen validation examples.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: The best training score is not necessarily the best score on unseen validation examples.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. The best training score is not necessarily the best score on unseen validation examples. Tap the diagram to enlarge. Overfitting occurs when a model fits training-specific details or noise that do not generalize. Training error is low while validation error is substantially higher. A mismatch between training and validation data can produce a similar gap, so the split must also be examined. ### How early stopping works 1. Train using the training set; do not learn parameters from the validation targets as training samples. 2. After an epoch or another chosen interval, evaluate validation loss. 3. Save the parameters whenever validation loss reaches a new best value. 4. Count consecutive intervals with no qualifying improvement. 5. When the count reaches the chosen patience, stop and restore the saved best parameters. For validation losses 0.8, 0.6, 0.5, 0.53 and 0.55, epoch 3 is best. With patience 2, epochs 4 and 5 supply two non-improvements: stop after epoch 5 and restore epoch 3. The stopping epoch and the retained model epoch are different. Other remedies for overfitting include appropriate L1/L2 penalties, dropout, augmentation and more representative training data. Underfitting may instead require a better model, adequate training or less regularization. Use the test set for final evaluation after those choices, not for repeatedly choosing the stopping point. --- ## MT2 · Choosing a remedy Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s6 For overfitting: try L1/L2, dropout, suitable data augmentation, a smaller model, more representative training data and early stopping. For underfitting: improve features or architecture, train adequately, check optimization, or reduce excessive regularization. Evaluate changes on validation data. ![Read the training and validation pattern before selecting a remedy.](https://deep-learning-study-room.vercel.app/assets/figures/v_remedies.svg) Diagram: Read the training and validation pattern before selecting a remedy.. Labels: Match the remedy to the observed problem | Match the remedy to the observed problem | Compare training and validation behavior before changing the model. | Both losses high: underfitting | Model may be too simple. | Training may be insufficient. | Check optimization and features. | Train low, validation high | Possible overfitting. | Use regularization or more data. | Consider early stopping. | A remedy is a choice based on the pattern; no single setting fixes every problem. Read the training and validation pattern before selecting a remedy. Tap the diagram to enlarge. Data augmentation makes label-preserving variations of training examples, such as suitable crops or brightness changes in images. It must respect the task: flipping a digit or a medical image may change meaning. Augmentation happens within the training split; do not leak augmented copies into validation. | Change | Usual bias tendency | Usual variance tendency | | --- | --- | --- | | Stronger weight regularization | Increase | Decrease | | More hidden units with other factors fixed | Decrease | Increase | | Dropout | Increase | Decrease | | More representative data from same distribution | Roughly unchanged model bias | Decrease | These are classical tendencies for exam discussion, not exact laws for every deep-network training run. Distinguish model capacity from optimization failure. --- ## MT2 · Inverted dropout during training and inference Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s7 **Definition.** Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required. ### Explanation and working Inverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero. ![Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation. Tap the diagram to enlarge. For activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation. At inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference. ### Expectation calculation For a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q: $$ \mathbb E\left[\frac{ma}{q}\right]=\frac aq\mathbb E[m]=\frac aq\,q=a $$ The equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction. --- ## MT2 · Model answer: dropout with training and inference explained Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c6-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/6/mt2-c6-s8 ## Question **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling. ## Answer — step by step Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights. ![The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge. Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop. ### Inverted dropout during training $$ a'=\frac{ma}{q} $$ A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks. For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is: $$ 0.5(6)+0.5(0)=3 $$ ### Inference Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions. Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data. --- ## MT2 · Reading image shapes before doing CNN arithmetic Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s0 **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension. ### Explanation and working ### Height, width and channels The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors. ![Count height, width and channels separately. Three channels are stacked at the same spatial positions.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_shape.svg) Diagram: Count height, width and channels separately. Three channels are stacked at the same spatial positions.. Labels: Read height, width and channels separately | Read height, width and channels separately | The text uses 4 × 5 × 3: four rows, five columns and three channels. | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 3 stacked channels | H × W × C | H = 4 rows | W = 5 columns | C = 3 channels | Total entries = 60 | A filter spans the incoming channels. Its spatial width is a different quantity. Count height, width and channels separately. Three channels are stacked at the same spatial positions. Tap the diagram to enlarge. ### One position versus the whole map A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels. ### Why filter depth matches input channels A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels. ### Reading a complete shape calculation The spatial-size formula computes height and width. It does not compute the channel count. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position. ### Shape, parameters and computation are three different counts **Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights. In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count. --- ## MT2 · A computer sees an image as numbers Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s1 **Definition.** A digital image is a spatially arranged numerical tensor. In a CNN, convolutions transform input channels into learned feature maps using local connectivity and spatially shared kernel parameters. ### Explanation and working A pixel is one position in an image. A grayscale image stores one intensity per position. A colour RGB image usually stores three numbers per position: red, green and blue. Therefore 32 × 32 × 3 means height 32, width 32 and 3 input channels. It contains 32 × 32 × 3 = 3,072 numbers, not 32 neurons. ![Each visible pixel corresponds to a matrix entry. Brightness is a number the network can calculate with.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_grid.svg) Diagram: Each visible pixel corresponds to a matrix entry. Brightness is a number the network can calculate with.. Labels: Pixels are numerical entries | Pixels are numerical entries | A small grayscale image can be stored as a matrix of brightness values. | 0 | 0 | 0 | 0 | 0 | 180 | 180 | 0 | 0 | 180 | 180 | 0 | 0 | 0 | 0 | 0 | Image | 4 rows × 4 columns | An RGB image has three such channels. A pixel location has a red, green and blue | value. Each visible pixel corresponds to a matrix entry. Brightness is a number the network can calculate with. Tap the diagram to enlarge. A **feature** is a useful pattern, such as an edge or texture. A **feature map** is a grid of responses that says where a learned filter matches the input. Early CNN layers often detect local simple patterns; later layers combine them into parts and more complex patterns. A filter, also called a kernel in many exam questions, is a small set of learned weights. Slide it across the image, multiply corresponding numbers, and add. The same weights are reused at every spatial position. This is **weight sharing**. --- ## MT2 · The convolution calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s2 **Definition.** A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention. ### Explanation and working In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it. **Question.** Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias. ### Given Input X = $\begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}$ . Kernel K = $\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$ . Stride = 1; padding = 0; bias = 0; no kernel flip. ### To find Find the 2 × 2 output map. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_cd39cabe48.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 0 | 0 | −1 | Matrix 2 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 Given matrices for this question Tap the diagram to enlarge. $$ X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix} $$ **Step 1. Top-left window.** 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4. **Step 2. Move right by one.** 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4. **Step 3. Move down and return left.** 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4. **Step 4. Bottom-right window.** 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is $\begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}$ . If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested. ![Four windows produce four output cells](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: Four windows produce four output cells. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. Four windows produce four output cells Tap the diagram to enlarge. --- ## MT2 · Stride padding and floor Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s3 **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output. ### Explanation and working Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension. ![The border adds to both sides of the input. The stride changes the next starting position of the filter window.](https://deep-learning-study-room.vercel.app/assets/figures/v_stride_padding.svg) Diagram: The border adds to both sides of the input. The stride changes the next starting position of the filter window.. Labels: Padding adds a border. Stride moves the window. | Padding adds a border. Stride moves the window. | Example: a 3 × 3 input becomes 5 × 5 after one zero border. | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 3 | 0 | 0 | 4 | 5 | 6 | 0 | 0 | 7 | 8 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | Padding P = 1 | 0 | 1 | 2 | 3 | 4 | Stride S = 2 | Starting positions: 0, 2, … | For each spatial dimension: add 2P, subtract K, divide by S, take floor, then add | 1. The border adds to both sides of the input. The stride changes the next starting position of the filter window. Tap the diagram to enlarge. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: $\left\lfloor 2.5\right\rfloor$ = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels. **Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1. ### Output dimension with a fractional intermediate result **Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.** ### Given N = 6; K = 3; S = 2; P = 0; dilation = 1. ### To find The integer output side length. ### Formula — which one and why The formula counts complete kernel placements. Flooring removes an incomplete final placement. N is input side; P is padding on each side; K is kernel side; S is stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1. ### Solution — step by step **Step 1: include padding and subtract kernel size.** $$ 6+2(0)-3=3 $$ **Step 2: divide by the stride.** $$ \frac32=1.5 $$ **Step 3: take the floor.** $$ \lfloor1.5\rfloor=1 $$ **Step 4: count the first placement.** $$ N_{\mathrm{out}}=1+1=2 $$ **Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter. ### Padding on both sides **Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.** ### Given N = 5; K = 3; S = 1; P = 1 on each side. ### To find Padded input and output dimensions. ### Formula — which one and why Padding adds P entries on each of two opposite sides; the output counts complete kernel placements. N = unpadded side; K = kernel side; S = stride; P = per-side padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\lfloor(N+2P-K)/S\rfloor+1 $$ **Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain. ### Solution — step by step **Step 1: count both padding edges.** $$ 2P=2(1)=2 $$ **Step 2: calculate the padded input side.** $$ N+2P=5+2=7 $$ **Step 3: subtract kernel width.** $$ 7-3=4 $$ **Step 4: divide by stride and take the floor.** $$ \lfloor4/1\rfloor=4 $$ **Step 5: include the first placement.** $$ N_{out}=4+1=5 $$ **Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter. “Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case. --- ## MT2 · Convolution over several channels Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s4 **Definition.** A standard convolutional filter spans every input channel. At one spatial position, channel-wise weighted sums are accumulated and one filter bias is added, producing one scalar output in that filter's feature map. ### Explanation and working A 3 × 3 filter receiving RGB input has depth 3, so it contains 3 × 3 × 3 weights. At one position, compute a weighted sum in each of the three channels, add those sums, and then add the filter’s one bias. One filter produces one output channel. Sixteen filters produce sixteen output channels. ![Add the channel contributions into one output cell. The filter bias is added once after those contributions.](https://deep-learning-study-room.vercel.app/assets/figures/v_channels.svg) Diagram: Add the channel contributions into one output cell. The filter bias is added once after those contributions.. Labels: One filter combines all incoming channels | One filter combines all incoming channels | The three channel contributions join into one cell, not three separate maps. | Red channel | Weighted sum = 2 | Green channel | Weighted sum = −1 | Blue channel | Weighted sum = 4 | Add contributions and one bias | 2 + (−1) + 4 + 0.5 = 5.5 | The filter spans all three channels. Add its bias once after summing the channel | contributions. Add the channel contributions into one output cell. The filter bias is added once after those contributions. Tap the diagram to enlarge. **Tiny example.** At one position the red, green and blue channel sums are 2, −1 and 4. With bias 0.5, the output is 2 − 1 + 4 + 0.5 = 5.5. Do not keep three separate outputs for this one filter; they have been added. --- ## MT2 · One filter over multiple channels: all contributions go into one cell Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s5 **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels. ### Explanation and working ### 1. Separate spatial size from depth Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3. ![Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.](https://deep-learning-study-room.vercel.app/assets/figures/v_channel_products.svg) Diagram: Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.. Labels: A 1 × 1 filter still mixes all three channels | A 1 × 1 filter still mixes all three channels | Use input [2, 5, 1], weights [0.1, −0.2, 0.5] and one bias 0.3. | Channel 1 | 2 × 0.1 = 0.2 | Channel 2 | 5 × (−0.2) = −1 | Channel 3 | 1 × 0.5 = 0.5 | One filter → one output value | 0.2 + (−1) + 0.5 + 0.3 = 0 | One spatial position can contain several channel values. The one-by-one filter | mixes them. Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example. Tap the diagram to enlarge. $$ z=0.1(2)+(-0.2)(5)+0.5(1)+0.3 $$ $$ z=0.2-1+0.5+0.3=0 $$ The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels. ### 2. A larger filter has one spatial grid per input channel A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias. The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified. ### 3. What changes when there are several filters? Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth. For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters. ### 4. One-by-one does not mean one weight A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed. This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately. --- ## MT2 · Choosing more or fewer filters Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s6 **Definition.** The number of convolutional filters determines the number of output channels. Kernel height, kernel width, stride and padding determine spatial processing; input-channel count determines each filter's depth. ### Explanation and working Each filter produces one output channel and can learn a different pattern. More filters increase representational capacity, output depth, parameter count and activation memory. They may improve performance but can overfit small datasets or exceed a compute budget. Choose counts using validation results and resource constraints. More filters in one layer increase its width, not the number of network layers. ![Each filter has the full incoming depth. Each produces one map, so the number of filters sets the output depth.](https://deep-learning-study-room.vercel.app/assets/figures/v_filters.svg) Diagram: Each filter has the full incoming depth. Each produces one map, so the number of filters sets the output depth.. Labels: Each filter produces one output feature map | Each filter produces one output feature map | Three incoming channels; two filters; two output maps. | Input | H × W × 3 | Filter A | K × K × 3 | Filter B | K × K × 3 | Map A | Hout × Wout | Map B | Hout × Wout | Two filters mean output depth 2. Filter depth matches the incoming depth 3. Each filter has the full incoming depth. Each produces one map, so the number of filters sets the output depth. Tap the diagram to enlarge. --- ## MT2 · Pooling needs comparisons or averages Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s7 **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases. ### Explanation and working Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged. **Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding: ### Given The 4 × 4 input map is printed below. Window = 2 × 2; stride = 2; padding = 0. ### To find Find the pooled output by the operation(s) requested. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step $$ A=\begin{bmatrix}21&11&15&10\\12&9&10&11\\11&13&18&12\\7&4&15&12\end{bmatrix} $$ ### Top-left window **Step 1: select the four entries.** [21, 11, 12, 9] **Step 2: choose the largest entry.** $$ y_{\max}=21 $$ **Step 3: add all entries.** $$ 21+11+12+9=53 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{53}4=13.25 $$ ### Top-right window **Step 1: select the four entries.** [15, 10, 10, 11] **Step 2: choose the largest entry.** $$ y_{\max}=15 $$ **Step 3: add all entries.** $$ 15+10+10+11=46 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{46}4=11.5 $$ ### Bottom-left window **Step 1: select the four entries.** [11, 13, 7, 4] **Step 2: choose the largest entry.** $$ y_{\max}=13 $$ **Step 3: add all entries.** $$ 11+13+7+4=35 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{35}4=8.75 $$ ### Bottom-right window **Step 1: select the four entries.** [18, 12, 15, 12] **Step 2: choose the largest entry.** $$ y_{\max}=18 $$ **Step 3: add all entries.** $$ 18+12+15+12=57 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{57}4=14.25 $$ Outputs are max $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ and average $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . ![Each pooling window reduces several values to one](https://deep-learning-study-room.vercel.app/assets/figures/pooling.svg) Diagram: Each pooling window reduces several values to one. Labels: INPUT 4 × 4 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 | MAX POOL | 21 | 15 | 13 | 18 | AVERAGE POOL | 13.25 | 11.5 | 8.75 | 14.25 | 2 × 2 windows | stride 2 Each pooling window reduces several values to one Tap the diagram to enlarge. Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance. --- ## MT2 · Parameters are different from activation values Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s8 **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters. ### Explanation and working An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights. ![The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations. Tap the diagram to enlarge. $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation. ### One filter, then all filters **Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.** ### Given Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter. ### To find Weights, biases, total trainable parameters and output channels. ### Formula — which one and why Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights. Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters. $$ P=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply. ### Solution — step by step **Step 1: count weights in one filter.** $$ 3(3)(3)=27 $$ **Step 2: include its one bias.** $$ 27+1=28 $$ **Step 3: count all weights.** $$ 27(16)=432 $$ **Step 4: count all biases.** $$ 1(16)=16 $$ **Step 5: add and state channel count.** $$ P=432+16=448 $$ Each filter produces one channel, so output channels = 16. **Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels. $$ P_{\mathrm{pool}}=0,\quad P_{\mathrm{flatten}}=0,\quad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values. --- ## MT2 · One complete CNN pipeline Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s9 **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron. ### Given Input = 128 × 128 × 3. Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2. Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2. Flatten → dense 240 → dense 120 → dense 5. One bias per learned filter/dense neuron. ### To find Find every output shape and layer parameter count, then the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step **Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448. ![Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.](https://deep-learning-study-room.vercel.app/assets/figures/v_cnn_pipeline.svg) Diagram: Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.. Labels: Follow the complete CNN question, one layer at a time | Follow the complete CNN question, one layer at a time | Start with 128 × 128 × 3. Shape and learned-parameter count are separate. | LAYER | OUTPUT SHAPE / LENGTH | PARAMETERS | 1. Convolution 1 | 126 × 126 × 16 | 448 | 2. Pooling 1 | 63 × 63 × 16 | 0 | 3. Convolution 2 | 61 × 61 × 32 | 4,640 | 4. Pooling 2 | 30 × 30 × 32 | 0 | 5. Flatten | 28,800 | 0 | 6. Dense 240 | 240 | 6,912,240 | 7. Dense 120 | 120 | 28,920 | 8. Classifier | 5 | 605 | Total learned parameters = 6,946,853 | Each output is the next layer’s input. Pooling and flattening add no learned | parameters. Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count. Tap the diagram to enlarge. **Step 2. Pool 1 size.** $\left\lfloor\frac{126 - 2}{2}\right\rfloor+1$ = 63; output 63 × 63 × 16. Parameters = 0. **Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640. **Step 4. Pool 2 size.** $\left\lfloor\frac{61 - 2}{2}\right\rfloor+1$ = $\left\lfloor 29.5\right\rfloor$ + 1 = 30; output 30 × 30 × 32. Parameters = 0. **Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0. **Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240. **Step 7. Dense 120.** (240 + 1) × 120 = 28,920. **Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer. **Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer. --- ## MT2 · CNN versus a fully connected image network Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s10/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s10 **Definition.** A convolutional neural network uses local receptive fields and spatial parameter sharing, unlike a dense layer's independent connection weights. These architectural constraints exploit spatial structure and can greatly reduce parameter counts. ### Explanation and working A fully connected layer can learn image features, but flattening pixels and connecting every pixel to every neuron ignores the useful local structure built into a CNN and can require many parameters. CNNs exploit local connections and shared filters. They learn hierarchical features and reuse a pattern detector at many locations. ![Compare where connections go and which weights are reused.](https://deep-learning-study-room.vercel.app/assets/figures/v_fc_cnn.svg) Diagram: Compare where connections go and which weights are reused.. Labels: Fully connected and convolutional layers connect differently | Fully connected and convolutional layers connect differently | Both learn weights; the connection pattern changes. | Fully connected image layer | Every output sees every input. | Each connection has a weight. | Spatial layout is not built in. | Convolutional layer | A filter sees a local patch. | Reuse it at many positions. | Preserve a spatial feature map. | Parameter sharing makes a convolution economical when the same feature can occur | in many places. Compare where connections go and which weights are reused. Tap the diagram to enlarge. For a 32 × 32 × 3 input, a dense layer with 100 neurons has (3,072 + 1) × 100 = 307,300 parameters. A convolution with 100 filters of size 3 × 3 has (27 + 1) × 100 = 2,800 parameters. The outputs are different shapes, so this is a parameter-efficiency illustration, not two identical computational tasks. --- ## MT2 · Model answer: CNN structure and parameter sharing Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s11/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s11 ## Question **Question.** Explain the main layers of a convolutional neural network and why CNNs are suitable for image data. Compare them with a fully connected image network. ## Answer — step by step A CNN processes spatially arranged data using filters that examine local regions. The same filter weights are reused at different positions. This combination of local connectivity and weight sharing builds an image-specific structure into the model and can greatly reduce the number of learned weights. ![The highlighted patch uses the same filter that will be reused at the next position.](https://deep-learning-study-room.vercel.app/assets/figures/v_weight_sharing.svg) Diagram: The highlighted patch uses the same filter that will be reused at the next position.. Labels: Slide the same filter; reuse its weights | Slide the same filter; reuse its weights | A CNN does not learn a fresh filter for every image location. | 1 | 2 | 0 | 1 | 0 | 1 | 3 | 2 | 2 | 0 | 1 | 1 | 1 | 2 | 0 | 3 | 1 | 0 | 0 | -1 | Same four weights | 0 | -1 | -2 | 0 | 0 | 2 | 0 | 0 | -2 | Output map | Move the window to calculate the next output cell. | Local connections restrict the visible patch. Weight sharing reuses the learned | pattern. The highlighted patch uses the same filter that will be reused at the next position. Tap the diagram to enlarge. ### Typical layer sequence **Input → convolution → nonlinear activation → optional pooling → further feature extraction → classifier.** | Layer | Operation | Learned parameters? | | --- | --- | --- | | Convolution | Multiply a local window by a filter, sum and add its bias | Filter weights and usually one bias per filter | | ReLU | Replace negative values by zero | No ordinary ReLU parameters | | Pooling | Summarize each window by maximum or average | None for ordinary max/average pooling | | Flatten | Rearrange H × W × C values into a vector | None | | Dense layer | Form weighted combinations of all incoming values | Weights and receiving-neuron biases | | Softmax | Convert class scores into probabilities | No additional weights of its own | Early filters can learn local patterns such as edges. Later layers combine features over larger receptive fields. A filter responding at several image positions uses one shared set of weights, not a new set at every location. Flattening an image and using a dense layer does not enforce this local, shared structure. A dense layer has a separate connection weight for each input–output pair. CNNs can therefore be more parameter-efficient for visual patterns, although they still need suitable data and training. Pooling can provide some robustness to small shifts; it does not guarantee complete invariance to every translation. --- ## MT2 · Output size and parameter count are different Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s12/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s12 **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters. ### Explanation and working **Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias. ### Given Input = 7 × 7 × 3; two 3 × 3 filters. Stride = 2; padding = 1; include biases. ### To find Find the output shape and trainable parameter count. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step **Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2. ![Output size counts produced values; parameter count counts adjustable weights and biases.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Output size counts produced values; parameter count counts adjustable weights and biases.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Output size counts produced values; parameter count counts adjustable weights and biases. Tap the diagram to enlarge. **Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters. --- ## MT2 · Model answer: shape, activations and parameters Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s13/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s13 ## Question **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them. ## Answer — step by step Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions. ![Keep output shape, activation count and parameter count as three separate quantities.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Keep output shape, activation count and parameter count as three separate quantities.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Keep output shape, activation count and parameter count as three separate quantities. Tap the diagram to enlarge. ### Spatial output shape For an ordinary convolution with dilation 1: $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters. ### Output activations $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ This is the number of output values for one example, excluding any batch dimension. ### Trainable parameters $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions. ### Small example A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels. --- ## MT2 · Faculty example: a complete binary-image convolution Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s14/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s14 **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)** ### Given Image X = $\begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}$ . Filter K = $\begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}$ . These are the values in the Unit 4 lecture example. ### To find The output shape and all nine feature-map values. ### Formula — which one and why The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window. ![Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.](https://deep-learning-study-room.vercel.app/assets/figures/u4_binary_windows.svg) Diagram: Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.. Labels: u4 binary windows | One filter, nine window positions | Input: 5 × 5 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | Shared filter: 3 × 3 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | → | Output: 3 × 3 | 4 | 3 | 4 | 2 | 4 | 3 | 2 | 3 | 4 | Orange window → orange output cell = 4. Move one cell at a time. Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions. Tap the diagram to enlarge. ### Solution — step by step **Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3. **Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**. **Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**. **Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**. **Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**. **Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**. **Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**. **Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**. **Final answer:** Y = $\begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}$ . All nine outputs use the same nine filter weights. They do not introduce nine different filters. **Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain. --- ## MT2 · Faculty example: combine three RGB channel sums Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s15/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s15 **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)** ### Given Red patch = $\begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}$ ; red kernel = $\begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}$ . Green patch = $\begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}$ ; green kernel = $\begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}$ . Blue patch = $\begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}$ ; blue kernel = $\begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}$ . Bias = 1. These are three slices of **one** full-depth filter, not three independent output filters. ### To find The contribution of each input channel and the final scalar output at this position. ### Formula — which one and why Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions. $$ z=s_R+s_G+s_B+b $$ The quantities s\_R, s\_G and s\_B are intermediate weighted sums, not separate final output channels. ### Solution — step by step **Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1. **Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**. **Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1). **Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**. **Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1. **Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**. **Step 7: add the channel contributions.** 308 −498 +164 = −26. **Step 8: add one filter bias.** −26 +1 = **−25**. **Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages. **Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo. --- ## MT2 · Max, min and average pooling on the same lecture grid Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s16/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s16 **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases. **Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)** ### Given X = $\begin{bmatrix}12 & 20 & 30 & 0\\8 & 12 & 2 & 0\\34 & 70 & 37 & 4\\112 & 100 & 25 & 12\end{bmatrix}$ . Window = 2 × 2; stride = 2; padding = 0. ### To find The shared output shape and each of the four cells for each pooling method. ### Formula — which one and why Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap. $$ y_{\max}=\max(x_1,x_2,x_3,x_4) $$ $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{\mathrm{avg}}=\frac{x_1+x_2+x_3+x_4}{4} $$ ![The four colours identify four separate pooling windows. Each window creates one cell in each output map.](https://deep-learning-study-room.vercel.app/assets/figures/u4_pooling_windows.svg) Diagram: The four colours identify four separate pooling windows. Each window creates one cell in each output map.. Labels: u4 pooling windows | Same four windows, three different summaries | Input: 4 × 4 | 12 | 20 | 30 | 0 | 8 | 12 | 2 | 0 | 34 | 70 | 37 | 4 | 112 | 100 | 25 | 12 | Max | 20 | 30 | 112 | 37 | Min | 8 | 0 | 34 | 4 | Average | 13 | 8 | 79 | 19.5 | Bottom-right average: (37 + 4 + 25 + 12) ÷ 4 = 19.5 | Pooling changes values and shape; ordinary pooling learns no parameters. The four colours identify four separate pooling windows. Each window creates one cell in each output map. Tap the diagram to enlarge. ### Solution — step by step **Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2. **Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13. **Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8. **Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79. **Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20. **Step 6: put each result back in its window's position.** Max = $\begin{bmatrix}20 & 30\\112 & 37\end{bmatrix}$ . Min = $\begin{bmatrix}8 & 0\\34 & 4\end{bmatrix}$ . Average = $\begin{bmatrix}13 & 8\\79 & 19.5\end{bmatrix}$ . **Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation. ### Why the choice matters Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour. Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter. **Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide. --- ## MT2 · Faculty CNN table: shapes, activations and 62958 parameters Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s17/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s17 **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)** ### Given Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question. ### To find Output shape and number of output values for each layer, its trainable parameters, and the total parameters. ### Formula — which one and why Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ A counts produced activations. P\_conv and P\_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here. ### Solution — step by step **Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0. **Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272. **Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**. **Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight. **Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600. **Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**. **Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters. **Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations. **Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations. **Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation. **Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**. | Layer | Output shape | Activations | Parameters | | --- | --- | --- | --- | | Input | 32 × 32 × 3 | 3,072 | 0 | | Conv1 | 28 × 28 × 8 | 6,272 | 608 | | Pool1 | 14 × 14 × 8 | 1,568 | 0 | | Conv2 | 10 × 10 × 16 | 1,600 | 3,216 | | Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 | | Dense 120 | 120 | 120 | 48,120 | | Dense 84 | 84 | 84 | 10,164 | | Dense 10 → softmax | 10 | 10 | 850 + 0 | **Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network. **Yaad rakho:** Shape aage carry karo; weights aur activations ko alag columns mein rakho. --- ## MT2 · Even-sized kernels: SAME padding is not always symmetric Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c7-s18/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/7/mt2-c7-s18 **Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides. For a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention. **Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho. ### Example 1: one entry on every side **Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)** ### Given Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter. ### To find The padded input dimensions and the convolution output dimensions. ### Formula — which one and why The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel. ### Solution — step by step **Step 1: count the two padding borders.** $$ 2P=2(1)=2 $$ **Step 2: find the padded side length.** $$ N_{\mathrm{padded}}=5+2=7 $$ The padded input is 7 × 7 × 1. **Step 3: subtract the kernel size.** $$ 7-2=5 $$ **Step 4: divide by stride, take the floor and count the first placement.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5}{1}\right\rfloor+1=5+1=6 $$ **Step 5: write height, width and channels.** Both spatial dimensions equal 6. There is one filter, so there is one output channel. **Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel. ### Example 2: preserve the size with SAME padding **Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)** ### Given Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found. ### To find Padding above, below, left and right; padded-input shape; final output shape. ### Formula — which one and why For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+P_{\mathrm{before}}+P_{\mathrm{after}}-K}{S}\right\rfloor+1 $$ At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1. $$ P_{\mathrm{total}}=P_{\mathrm{before}}+P_{\mathrm{after}}=K-1 $$ ### Solution — step by step **Step 1: calculate total padding along the height.** $$ P_{\mathrm{total}}=2-1=1 $$ Only one extra row in total is needed. **Step 2: place the row using the stated convention.** Padding above = 0; padding below = 1. **Step 3: apply the same calculation to width.** Only one extra column in total is needed. Padding left = 0; padding right = 1. **Step 4: find the padded-input shape.** $$ H_{\mathrm{padded}}=5+0+1=6 $$ $$ W_{\mathrm{padded}}=5+0+1=6 $$ The padded input is 6 × 6 × 1. **Step 5: count the output positions along each axis.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5+0+1-2}{1}\right\rfloor+1=4+1=5 $$ **Step 6: include the output channel.** One filter creates one output channel. **Final answer:** add **0 above, 1 below, 0 left and 1 right**; padded input = **6 × 6 × 1**; output = **5 × 5 × 1**. The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula. --- ## MT2 · AlexNet and the reason for its components Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s0 **Definition.** AlexNet is a deep convolutional architecture introduced for ImageNet classification, conventionally described as five convolutional layers followed by three fully connected layers. Its historical significance is the effective combination of learned convolutional features, nonlinear activations, GPU training and regularization at large scale. ### Architecture and information flow Input image → convolution and ReLU stages → spatial pooling → fully connected representations → class scores and softmax. The original model used local response normalization in selected early stages and overlapping max pooling. Grouped computation across two GPUs was part of the original implementation. Exact sizes must follow the architecture specified in an examination question. ![Convolution extracts representations before the fully connected classification stages.](https://deep-learning-study-room.vercel.app/assets/figures/alexnet.svg) Diagram: Convolution extracts representations before the fully connected classification stages.. Labels: Input | 227×227×3 | Conv1 | 11×11, 96 | Pool | 3×3, s2 | Conv2 | 5×5, 256 | Pool | 3×3, s2 | Conv3 | 3×3, 384 | Conv4 | 3×3, 384 | Conv5 | 3×3, 256 | Pool + flat | 9216 | Dense | 4096 | Dense | 4096 | Output | 1000 | Read rows in order. Apply nonlinearities between learned layers. Convolution extracts representations before the fully connected classification stages. Tap the diagram to enlarge. ### Role of the components **ReLU** supplies a nonlinear transformation and avoids sigmoid-style saturation for positive preactivations. **Convolution** uses local connectivity and spatial weight sharing. **Pooling** reduces spatial dimensions and increases tolerance to small local changes. **Dropout** regularizes the fully connected layers. **Data augmentation** supplies transformed training examples to reduce dependence on a fixed view. These components perform different jobs and should not be described as interchangeable methods of reducing overfitting. ### Limitations and exam distinction The dense layers contribute a large number of parameters; the model has substantial memory and computation requirements. AlexNet is much shallower than later residual architectures. Its original use of local response normalization is different from batch normalization. For a comparison answer, identify its sequential convolutional design, then contrast it with Inception's parallel branches and ResNet's residual addition. --- ## MT2 · Inception examines several spatial scales Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s1 **Definition.** An Inception module is a convolutional building block that processes the same input through parallel branches using different receptive-field operations and concatenates their output feature maps along the channel dimension. ### Structure and working A conventional early Inception module includes 1 × 1, 3 × 3 and 5 × 5 convolution branches and a pooling branch. Suitable stride and padding preserve compatible spatial dimensions. The branch outputs are then concatenated. If branch channel counts are c₁, c₂, c₃ and c₄, the output depth is c₁ + c₂ + c₃ + c₄. ![Branches operate on the same input. Matching spatial outputs are joined along the channel axis.](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Branches operate on the same input. Matching spatial outputs are joined along the channel axis.. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Branches operate on the same input. Matching spatial outputs are joined along the channel axis. Tap the diagram to enlarge. ### Why use several branches? Different branch receptive fields can capture patterns at different spatial scales. A 1 × 1 convolution mixes channel information at each position. Before an expensive larger kernel, it can reduce the number of input channels and therefore the parameter count and multiplication cost. The bottleneck calculation is developed in the following section. ### Concatenation is not addition Concatenation increases channel depth and retains each branch's feature maps separately. Elementwise addition requires equal shapes and combines corresponding values. Inception commonly uses concatenation; a standard residual block uses addition. A question giving branch sizes tests this distinction as well as the convolution formula. Early Inception designs reduce reliance on a large fully connected classifier through global average pooling. The parallel design still needs compatible branch shapes and carefully chosen channel widths; increasing the number of branches or filters increases computation. --- ## MT2 · Why one by one convolution saves work Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s2 **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts. ![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg) Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications. Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge. **Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.** ### Given Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size. ### To find Direct MACs, total two-stage bottleneck MACs and percentage saved. ### Formula — which one and why Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs. ### Solution — step by step **Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400 $$ **Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels. $$ \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448 $$ **Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels. $$ \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200 $$ **Step 4: add the two bottleneck costs.** $$ \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648 $$ **Step 5: calculate the saving.** $$ \mathrm{saving}=120422400-12443648=107978752 $$ $$ \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\% $$ **Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption. ### A second bottleneck with different dimensions **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1. ### To find Direct MACs, bottleneck MACs and saving. ### Formula — which one and why Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers. ### Solution — step by step **Step 1: direct spatial convolution.** $$ 8(8)(8)(3)(3)(16)=73728 $$ **Step 2: 1 × 1 reduction.** $$ 8(8)(4)(1)(1)(16)=4096 $$ **Step 3: reduced-depth 3 × 3 convolution.** $$ 8(8)(8)(3)(3)(4)=18432 $$ **Step 4: add the stages and subtract from the direct cost.** $$ 4096+18432=22528 $$ $$ 73728-22528=51200 $$ **Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values. --- ## MT2 · ResNet adds a shortcut Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s3 **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x. $$ y=F(x)+x $$ If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch. ![The two paths must have equal output shape before elementwise addition.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge. ### Effect on gradient propagation For the identity-shortcut formulation, the derivative includes a direct identity contribution. $$ \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I $$ This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients. ### Projection when dimensions differ If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride. $$ y=F(x)+W_sx $$ An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition. **Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum. ### Given x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3]. ### To find Find the three components of x+F(x). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. ### Solution — step by step **Step 1: add the first matching pair.** $$ 1+0.1=1.1 $$ **Step 2: add the second matching pair.** $$ 2+(-0.2)=1.8 $$ **Step 3: add the third matching pair.** $$ 3+0.3=3.3 $$ **Step 4: assemble in the original order.** $$ y=[1.1,1.8,3.3] $$ The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum. --- ## MT2 · Transfer learning from a small dataset Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s4 **Definition.** Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers. ### Explanation and working Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort. ![Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change. Tap the diagram to enlarge. **Full scenario.** Build a face-attendance model with few images per student. Explain a training plan. **Step 1. Define and split the data.** Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization. **Step 2. Choose a suitable pretrained backbone.** For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available. **Step 3. Replace the classifier.** Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy. **Step 4. Freeze and train.** Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring. **Step 5. Fine-tune if justified.** Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint. **Step 6. Evaluate.** Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition. Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario. --- ## MT2 · Model answer: feature extraction and fine-tuning Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s5 ## Question **Question.** Explain transfer learning for image classification with limited labeled data. Distinguish feature extraction from fine-tuning and describe the training procedure. ## Answer — step by step Transfer learning reuses knowledge represented by a model trained on a source task to support a related target task. A pretrained CNN can provide useful visual representations, reducing the target data and training effort needed compared with learning all features from random initialization. ![Freezing and fine-tuning differ in whether selected pretrained weights are allowed to update.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Freezing and fine-tuning differ in whether selected pretrained weights are allowed to update.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Freezing and fine-tuning differ in whether selected pretrained weights are allowed to update. Tap the diagram to enlarge. ### Procedure 1. Define the target classes and create separate training, validation and test splits. Prevent duplicate or closely related captures from leaking across splits. 2. Choose a suitable pretrained backbone and apply its required input size and preprocessing. 3. Replace the original classifier with a new head whose output count matches the target classes. 4. Freeze the backbone and train the new head. Only the unfrozen parameters receive optimizer updates. 5. If validation results justify it, unfreeze selected later backbone layers and continue training with a suitably smaller learning rate. 6. Select hyperparameters and checkpoints using validation data. Evaluate the selected model on the held-out test set. | Feature extraction | Fine-tuning | | --- | --- | | Pretrained backbone weights remain fixed | Some or all selected pretrained weights are updated | | New classifier learns from fixed representations | Representations can adapt to the target task | | Usually fewer trainable parameters | Usually more trainable parameters and greater risk of overfitting on a small dataset | The benefit depends on how useful the source features are for the target data. Transfer is not guaranteed to help every unrelated task. For a numerical, count all stored parameters separately from the trainable subset; freezing a layer does not delete its weights. --- ## MT2 · How to draw the architecture diagrams Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s6 For AlexNet, draw a sequence of convolution/pooling stages followed by dense layers and classifier. For Inception, draw four parallel branches and a channel-concatenation join. For ResNet, draw one transformation branch and one shortcut meeting at a plus sign. For transfer learning, draw frozen pretrained base → new trainable head, then mark selected base layers optionally unfrozen for fine-tuning. ![Draw the sequential path, parallel branches or additive shortcut that explains the architecture.](https://deep-learning-study-room.vercel.app/assets/figures/v_architectures.svg) Diagram: Draw the sequential path, parallel branches or additive shortcut that explains the architecture.. Labels: Three architecture ideas to remember | Three architecture ideas to remember | These are different designs, not three consecutive layers of one network. | AlexNet | Sequential | convolution and | pooling stages | Inception | Parallel branches; | concatenate their | channels | ResNet | Learn a branch; add a | shortcut | A diagram should show the mechanism, not only the architecture name. Draw the sequential path, parallel branches or additive shortcut that explains the architecture. Tap the diagram to enlarge. --- ## MT2 · Inception concatenation and residual addition Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s7 **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths. ### Explanation and working Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations. ![Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.](https://deep-learning-study-room.vercel.app/assets/figures/v_join_add.svg) Diagram: Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.. Labels: Concatenation and addition produce different shapes | Concatenation and addition produce different shapes | Use height and width 14 × 14 throughout. | CONCATENATE four branches | Channels: 16 + 32 + 8 + 8 | Result: 14 × 14 × 64 | Stack all 64 channels. | ADD two matching tensors | 14 × 14 × 64 + 14 × 14 × 64 | Result: 14 × 14 × 64 | Add each matching pair. | A 14 × 14 × 32 shortcut needs a projection before addition to a 14 × 14 × 64 | branch. Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape. Tap the diagram to enlarge. Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64. A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition. --- ## MT2 · Model answer: AlexNet, Inception and ResNet Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s8 ## Question **Question.** Compare the main ideas of AlexNet, Inception and ResNet. Explain how to recognize their architecture diagrams. ## Answer — step by step All three are CNN architectures, but their characteristic structures differ. ![Each block illustrates the defining design idea; the full architecture diagrams give its detailed layout.](https://deep-learning-study-room.vercel.app/assets/figures/v_architectures.svg) Diagram: Each block illustrates the defining design idea; the full architecture diagrams give its detailed layout.. Labels: Three architecture ideas to remember | Three architecture ideas to remember | These are different designs, not three consecutive layers of one network. | AlexNet | Sequential | convolution and | pooling stages | Inception | Parallel branches; | concatenate their | channels | ResNet | Learn a branch; add a | shortcut | A diagram should show the mechanism, not only the architecture name. Each block illustrates the defining design idea; the full architecture diagrams give its detailed layout. Tap the diagram to enlarge. | Architecture | Main idea | What to show in a diagram | | --- | --- | --- | | AlexNet | Successive convolutional feature extraction followed by dense classification | Five convolutional layers and three fully connected layers, with the stated pooling/normalization stages | | Inception v1 | Parallel branches analyze features at different spatial scales | 1 × 1, reduced 3 × 3, reduced 5 × 5 and pooling branches joined by channel concatenation | | ResNet | Learn a residual transformation and add a shortcut | A transformation branch and input shortcut meeting at a plus sign | AlexNet's exam-relevant features include ReLU, GPU-based training, augmentation and dropout in the fully connected part. A parameter calculation must follow the given channel connectivity; a grouped implementation and a simplified fully connected-channel diagram need not have the same count. Inception uses 1 × 1 convolutions to mix channels and, when a smaller number of filters is chosen, reduce the channel depth before expensive spatial filters. Branch outputs must have compatible height and width for concatenation. Concatenation adds channel counts by joining their lists. ResNet computes a learned transformation F(x) and combines it with the shortcut: $$ H(x)=F(x)+x $$ This is entrywise addition. The two shapes must match; a projection can adapt a shortcut when they do not. The shortcut supplies an additional route for information and gradients and makes identity-like mappings easy to represent. **Key distinction:** an Inception join concatenates feature maps; a residual join adds corresponding values. Their diagram symbols and output-channel calculations must therefore differ. --- ## MT2 · Applications of convolutional neural networks Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s9 **Definition.** A CNN application combines a convolutional feature extractor with an output head designed for a particular prediction task. Shared local filters learn spatial patterns; the output head determines what is predicted and the loss determines how it is trained. ### Classification, detection and segmentation | Task | Required output | What distinguishes it | | --- | --- | --- | | Image classification | One label or class distribution for the image | Answers what category the whole image belongs to | | Object detection | Object classes and bounding-box coordinates | Also identifies where each detected object is located | | Semantic segmentation | A class label or distribution for each pixel | Produces a spatial label map rather than one image-wide class | | Face recognition | Identity class or a learned face representation | Compares or classifies learned visual features | A softmax image classifier alone does not produce boxes or pixel labels. Detection and segmentation require appropriate task heads, targets and objectives. In an applications question, explain the prediction required instead of listing model names without their purpose. ### Examples across domains **Medical imaging:** classify an image or use a segmentation model to identify regions of interest. The target may be an image-level label or a pixel mask, so the architecture and loss must match that choice. **Autonomous perception:** use camera images for tasks such as detecting pedestrians and vehicles or segmenting road regions. These outputs are inputs to a larger perception and decision system. **Manufacturing:** classify product images as defective or non-defective, or localize defects when position information is needed. **Document analysis:** classify cropped handwritten characters; a complete document-reading system must also locate and order text regions. **Remote sensing:** classify land-use regions or segment different surface categories in satellite images. Spatial context and texture can help distinguish classes. ### Complete exam-style application answer **Question. Describe how a CNN can classify product photographs into three classes: acceptable, scratched and cracked. Explain the input, network stages, output activation, loss, training and evaluation.** **Step 1: define data and labels.** Each example contains an image and one of the three class labels. Use separate training, validation and test sets, keeping near-duplicate photographs of the same item out of different splits. Apply the chosen model's input size and scaling consistently. **Step 2: extract spatial features.** Convolutional layers calculate local responses using shared kernels. Nonlinear activations allow more complex combinations of those responses. Further convolution stages combine local patterns; pooling may reduce spatial dimensions. **Step 3: produce three class scores.** Convert the final representation into a feature vector, using flattening or an appropriate global pooling operation, and apply a dense layer with three outputs. These raw outputs are logits. **Step 4: calculate class probabilities and loss.** For mutually exclusive classes, use softmax and categorical cross-entropy. A one-hot label selects the probability of the correct class. $$ q_k=\frac{e^{z_k}}{\sum_{j=1}^{3}e^{z_j}},\qquad L=-\sum_{k=1}^{3}y_k\ln q_k $$ **Step 5: train the parameters.** Run forward propagation, evaluate the loss, backpropagate derivatives and update the trainable convolutional and dense parameters using the selected optimizer. Use validation results for hyperparameters and stopping decisions. **Step 6: evaluate the selected model.** Assess the final model on held-out photographs. Report overall classification performance and inspect which defects are confused. With limited data, a pretrained CNN can provide features and selected layers can be fine-tuned. **Numerical extension. If the last pooled feature vector has 64 components, how many parameters does the three-class dense head have, including bias?** **Step 1:** each output receives 64 weights, so three outputs require 64 × 3 = 192 weights. **Step 2:** each of the three outputs has one bias, giving 3 biases. $$ P_{\mathrm{head}}=64(3)+3=(64+1)3=195 $$ **Final answer:** 195 trainable parameters in this head. Softmax adds no trainable parameters. The backbone's parameters are counted separately. --- ## MT2 · Reading LeNet and AlexNet layer diagrams Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s10/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s10 **A layer diagram describes two different things:** the size of the data moving through the network and the trainable operations that transform that data. A number such as 9,216 beside a flatten operation describes data length; it does not automatically mean another dense layer with 9,216 neurons. ### LeNet: follow the image as it becomes a class prediction The classroom LeNet layout begins with a 32 × 32 grayscale image. A valid 5 × 5 convolution with six filters produces 28 × 28 × 6. A 2 × 2 pooling operation with stride 2 produces 14 × 14 × 6. The next valid 5 × 5 convolution produces 10 × 10 × 16, and pooling reduces this to 5 × 5 × 16. A 5 × 5 convolution across those 16 channels produces 1 × 1 × 120. The final representation passes through 84 units and then ten class outputs. | Stage | What happens | Output shape | | --- | --- | --- | | Input | One grayscale image | 32 × 32 × 1 | | C1 | Six 5 × 5 filters, stride 1, no padding | 28 × 28 × 6 | | S2 | 2 × 2 average pooling, stride 2 | 14 × 14 × 6 | | C3 | Sixteen 5 × 5 filters, stride 1, no padding | 10 × 10 × 16 | | S4 | 2 × 2 average pooling, stride 2 | 5 × 5 × 16 | | C5 | 120 filters covering the complete 5 × 5 spatial map | 1 × 1 × 120 | | F6 | Dense transformation | 84 values | | Classifier | Modern classroom softmax classifier | Ten probabilities | The original LeNet-5 used details such as partial channel connectivity, trainable subsampling and an RBF output. A modern full-channel implementation with ordinary pooling and softmax is a different parameter-count convention. For a numerical, follow the question's diagram and state the convention. The complete LeNet parameter numerical in the solved-paper section uses the modern classroom convention. ### AlexNet: identify the flatten step correctly ![Five convolutional layers extract image features; three dense layers produce the classifier output. Flattening only rearranges the final feature maps.](https://deep-learning-study-room.vercel.app/assets/figures/alexnet.svg) Diagram: Five convolutional layers extract image features; three dense layers produce the classifier output. Flattening only rearranges the final feature maps.. Labels: Input | 227×227×3 | Conv1 | 11×11, 96 | Pool | 3×3, s2 | Conv2 | 5×5, 256 | Pool | 3×3, s2 | Conv3 | 3×3, 384 | Conv4 | 3×3, 384 | Conv5 | 3×3, 256 | Pool + flat | 9216 | Dense | 4096 | Dense | 4096 | Output | 1000 | Read rows in order. Apply nonlinearities between learned layers. Five convolutional layers extract image features; three dense layers produce the classifier output. Flattening only rearranges the final feature maps. Tap the diagram to enlarge. For the faculty's 227 × 227 × 3 classroom input, the main shape sequence is 55 × 55 × 96 → 27 × 27 × 96 → 27 × 27 × 256 → 13 × 13 × 256 → 13 × 13 × 384 → 13 × 13 × 384 → 13 × 13 × 256 → 6 × 6 × 256. Flattening gives 9,216 values; the three dense outputs are 4,096, 4,096 and 1,000. There is no extra learned 9,216-unit layer between flattening and the first 4,096-unit layer. **Question (practice, 3 marks). An AlexNet-style first layer receives a 227 × 227 RGB image and uses 96 filters of size 11 × 11, stride 4 and no padding. Find its output shape, trainable parameters including bias, and the number of output activation values.** ### Given Input height and width = 227; input channels = 3; kernel = 11; stride = 4; padding = 0; output filters = 96. One bias is used per filter. ### To find The three-dimensional output shape, parameter count and activation count. ### Formula — which one and why The output-size formula counts valid window positions. Parameter count counts stored filter coefficients, reused at every position. Activation count counts the values produced for one image. $$ H_o=W_o=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ A=H_oW_oC_{\mathrm{out}} $$ Here N is input side length, P is padding per side, K is kernel side length, S is stride, and C denotes channel count. The symbol P in the output formula means padding; the labelled quantity P\_conv means parameters. ### Solution — step by step **Step 1: calculate one spatial side.** $$ \frac{227-11}{4}+1=\frac{216}{4}+1=54+1=55 $$ **Step 2: attach the output depth.** There is one output map per filter, so the output is **55 × 55 × 96**. **Step 3: count the weights in one filter.** $$ 11(11)(3)=363 $$ **Step 4: add one bias and multiply by 96 filters.** $$ P_{\mathrm{conv}}=(363+1)(96)=34944 $$ **Step 5: count activation values for one image.** $$ A=55(55)(96)=290400 $$ **Final answer:** output = 55 × 55 × 96; parameters = 34,944; activations = 290,400. The weights are reused, so parameters and activations are different counts. The full faculty-style ungrouped AlexNet table totals 62,378,344 parameters; the original grouped model uses different channel connectivity. **Yaad rakho:** shape = kitne answers nikle; parameters = kitne numbers seekhe. --- ## MT2 · Top-1 and top-5 error: what an ImageNet result means Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s11/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s11 **Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models. For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures. **Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.** ### Given Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes. ### To find The percentage of mistakes under each rule. ### Formula — which one and why Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage. $$ E_k=\frac{N-C_k}{N}\times100\% $$ N means total evaluated images; C\_k means images whose correct class appears among the top k predictions. ### Solution — step by step **Step 1: top-1 mistakes.** 100 − 78 = 22. **Step 2: top-1 error percentage.** $$ E_1=\frac{22}{100}\times100\%=22\% $$ **Step 3: top-5 mistakes.** 100 − 92 = 8. **Step 4: top-5 error percentage.** $$ E_5=\frac{8}{100}\times100\%=8\% $$ **Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions. **Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success. --- ## MT2 · VGG16: small filters in a regular sequence Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s12/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s12 ![Follow the five blocks and their output sizes. The 16 counts learned layers, not pooling or flattening.](https://deep-learning-study-room.vercel.app/assets/figures/u4_vgg16.svg) Diagram: Follow the five blocks and their output sizes. The 16 counts learned layers, not pooling or flattening.. Labels: u4 vgg16 | VGG16: 13 convolution layers + 3 dense layers | Block 1 | 2 convolutions | Block 2 | 2 convolutions | Block 3 | 3 convolutions | Block 4 | 3 convolutions | Block 5 | 3 convolutions | 112 × 112 | 56 × 56 | 28 × 28 | 14 × 14 | 7 × 7 | Sizes shown after each 2 × 2 pool, starting from a 224 × 224 image. | 7 × 7 × 512 → flatten 25,088 → 4,096 → 4,096 → 1,000 Follow the five blocks and their output sizes. The 16 counts learned layers, not pooling or flattening. Tap the diagram to enlarge. **Definition.** VGG16 is a convolutional architecture with 13 convolutional layers and three dense layers. Its standard convolution blocks contain 2, 2, 3, 3 and 3 convolutions. Pooling, activations and flattening are not counted in the name's 16 weight layers. Each convolution uses a 3 × 3 kernel, stride 1 and padding 1, preserving height and width. Each block ends with 2 × 2 max pooling of stride 2. The final classifier is 4,096 → 4,096 → 1,000 for the original 1,000-class task. | Block | Convolutions | Channels after its convolutions | Size after its pooling | | --- | --- | --- | --- | | 1 | Two | 64 | 112 × 112 × 64 | | 2 | Two | 128 | 56 × 56 × 128 | | 3 | Three | 256 | 28 × 28 × 256 | | 4 | Three | 512 | 14 × 14 × 512 | | 5 | Three | 512 | 7 × 7 × 512 | This table starts with 224 × 224 × 3. The flattened length is 7 × 7 × 512 = 25,088. Flattening has zero learned parameters. Stacking small filters provides repeated nonlinear transformations while progressively combining spatial information. A limitation is the large dense classifier and its memory cost. **Question (practice, 3 marks). VGG16's first block contains two same-padded 3 × 3 convolutions with 64 output filters each, followed by 2 × 2 max pooling with stride 2. Input is 224 × 224 × 3. Find both convolution parameter counts including bias and the pooled output shape.** ### Given Conv1: 3 input channels, 64 output channels. Conv2: 64 input channels, 64 output channels. Both have kernel 3, stride 1 and padding 1. Pooling has kernel 2, stride 2 and no padding. ### To find Trainable parameters in each convolution and the output after pooling. ### Formula — which one and why $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Use each layer's own input depth; the second layer receives the first layer's 64 maps. Use the spatial-output formula for pooling. Max pooling introduces no trainable parameters. ### Solution — step by step **Step 1: first convolution.** Each filter has 3 × 3 × 3 = 27 weights and one bias. $$ P_1=(27+1)(64)=1792 $$ **Step 2: second convolution.** Each filter now spans 64 channels, giving 3 × 3 × 64 = 576 weights. $$ P_2=(576+1)(64)=36928 $$ **Step 3: add parameters.** $$ P_{\mathrm{block}}=1792+36928=38720 $$ **Step 4: calculate pooled side length.** The same-padded convolutions retained side length 224. $$ \frac{224-2}{2}+1=111+1=112 $$ **Final answer:** 1,792 and 36,928 parameters; total = 38,720; pooled shape = **112 × 112 × 64**. **Exam convention:** use the architecture explicitly supplied in a calculation. For the standard VGG16 architecture, remember **2–2–3–3–3 convolutions and 2 × 2 pooling**; 25,088 is the flattened feature length. --- ## MT2 · Global average pooling: one average for each channel Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s13/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s13 ![Average each channel separately: these are the exact values used in the worked example.](https://deep-learning-study-room.vercel.app/assets/figures/u4_gap.svg) Diagram: Average each channel separately: these are the exact values used in the worked example.. Labels: u4 gap | Global average pooling keeps channels separate | Channel A | 1 | 3 | 5 | 7 | → | 4 | 16 ÷ 4 | Channel B | 2 | 4 | 6 | 8 | → | 5 | 20 ÷ 4 | 2 × 2 × 2 input → two channel means → [4, 5] Average each channel separately: these are the exact values used in the worked example. Tap the diagram to enlarge. **Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier. **Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = $\begin{bmatrix}1 & 3\\5 & 7\end{bmatrix}$ and channel B = $\begin{bmatrix}2 & 4\\6 & 8\end{bmatrix}$ . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.** ### Given H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs. ### To find The GAP result and dense-head parameter count. ### Formula — which one and why Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias. $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Solution — step by step **Step 1: add channel A.** 1 + 3 + 5 + 7 = 16. **Step 2: average channel A.** 16 ÷ 4 = 4. **Step 3: add channel B.** 2 + 4 + 6 + 8 = 20. **Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**. **Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights. **Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero. **Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters. **Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao. --- ## MT2 · The faculty's 480-channel bottleneck numerical Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s14/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s14 ![Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.](https://deep-learning-study-room.vercel.app/assets/figures/u4_bottleneck_480.svg) Diagram: Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.. Labels: u4 bottleneck 480 | Same output shape, different computation | Input | 14 × 14 × 480 | → | Direct 5 × 5 | 48 filters | → | Output | 14 × 14 × 48 | 112,896,000 MACs | 1 × 1 reduction | 480 → 16 channels | → | Reduced 5 × 5 | 16 → 48 channels | → | Output | 14 × 14 × 48 | 1,505,280 + 3,763,200 = 5,268,480 MACs | The lower path also starts from 14 × 14 × 480. Count both layers. Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers. Tap the diagram to enlarge. **Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.** ### Given Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters. ### To find MACs for the direct path, both bottleneck stages, their total and percentage saved. ### Formula — which one and why $$ M=H_oW_oC_oK^2C_i $$ M is the MAC count. H\_oW\_oC\_o counts output values; K²C\_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck. $$ \mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}} $$ ### Solution — step by step **Step 1: direct output count.** 14 × 14 × 48 = 9,408. **Step 2: products per direct output.** 5 × 5 × 480 = 12,000. **Step 3: direct MACs.** $$ 9408(12000)=112896000 $$ **Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values. $$ 14(14)(16)(480)=1505280 $$ **Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values. $$ 9408(400)=3763200 $$ **Step 6: add both bottleneck costs.** $$ 1505280+3763200=5268480 $$ **Step 7: calculate the saving.** $$ 112896000-5268480=107627520 $$ $$ 100\frac{107627520}{112896000}\approx95.333333\% $$ **Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone. --- ## MT2 · Batch normalization inside a residual network Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s15/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s15 **Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training. For one feature with m values in a training batch: $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ The batch mean μ\_B is the centre; batch variance σ²\_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch. **Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.** ### Given x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5. ### To find Batch mean, variance, normalized values and scale-and-shift outputs. ### Formula — which one and why Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale. ### Solution — step by step **Step 1: mean.** $$ \mu_B=\frac{2+4}{2}=3 $$ **Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1. **Step 3: squared deviations and variance.** $$ \sigma_B^2=\frac{(-1)^2+1^2}{2}=\frac{1+1}{2}=1 $$ **Step 4: stable denominator.** $$ d=\sqrt{1+0.00001}\approx1.00000499999 $$ **Step 5: normalize separately.** $$ \hat{x}_1=\frac{-1}{d}\approx-0.99999500004 $$ $$ \hat{x}_2=\frac{1}{d}\approx0.99999500004 $$ **Step 6: scale the first normalized value, then shift it.** $$ y_1=2(-0.99999500004)+0.5\approx-1.49999000008 $$ **Step 7: scale the second normalized value, then shift it.** $$ y_2=2(0.99999500004)+0.5\approx2.49999000008 $$ **Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here. **Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo. --- ## MT2 · Residual-block order and architecture qualifications Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s16/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s16 The original post-activation basic block has this sequence: **convolution → BN → ReLU → convolution → BN → add shortcut → ReLU**. Thus the second branch convolution is not followed by a separate ReLU before the addition in this particular block. A pre-activation ResNet is another design; use the stated diagram. ![Add the shortcut to the residual branch, then apply the post-addition ReLU when the specified block requires it.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: Add the shortcut to the residual branch, then apply the post-addition ReLU when the specified block requires it.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. Add the shortcut to the residual branch, then apply the post-addition ReLU when the specified block requires it. Tap the diagram to enlarge. An identity shortcut has no learned parameters. A projection shortcut changes the shape using learned weights. Original ImageNet ResNets include an initial max-pooling operation; a basic residual block does not need a max-pooling layer inside it. ResNet-18/34 use two-convolution basic blocks, whereas deeper standard variants use bottleneck blocks. Avoid treating every ResNet as the same two-layer diagram. The main advantage is easier optimization of deep networks: shortcuts provide an additional information and gradient path. This reduces the degradation problem but does not prove immunity to every vanishing/exploding-gradient case. Large networks still need compute, memory, suitable data and regularization. Reusing early features can help; it does not guarantee that every deeper model generalizes better. --- ## MT2 · Choosing frozen layers from dataset size and similarity Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s17/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s17 Early image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom. | Target dataset | Sensible starting choice | Reason | | --- | --- | --- | | Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely | | Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters | | Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky | | Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested | These are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently. **Example.** With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate. **Yaad rakho:** data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho. --- ## MT2 · Four ways to reuse a pretrained model Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c8-s18/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/8/mt2-c8-s18 **Definition.** A pretrained model has already learned parameters from an earlier training task. We can reuse its complete classifier, reuse its feature extractor, or use its learned weights as the starting point for further training. The choice determines which output labels are available and which parameters change. **First identify two parts.** The **backbone** converts an input image into useful features. The **head** converts those features into the required prediction, such as class scores. A new task can reuse the backbone while replacing the head. **Yaad rakhne ka cue:** Purane model ka poora answer use karna, sirf features lena, ya uske weights ko aage train karna—yeh teen decisions alag hain. The faculty reference divides these choices into the following four uses. The two feature-extractor uses differ in whether features are saved separately or computed inside the new model. | Use | What happens to the pretrained part? | What learns for the target task? | Concrete example | | --- | --- | --- | --- | | Direct classifier | Keep the complete model and its existing output head. Run it on new inputs. | Nothing is updated during ordinary prediction. | A model already trained to classify cats and dogs labels new cat/dog photographs. It can do this because the required labels match its existing outputs. | | Standalone feature extractor | Run images through the fixed backbone and save the resulting feature vectors. A feature vector is an ordered list of numbers describing the image. | Train a separate classifier using these saved vectors. | Extract one vector per product photograph, then train a classifier to distinguish acceptable, scratched and cracked products. The extractor stays unchanged. | | Integrated feature extractor | Place the pretrained backbone inside the new network and freeze it. Add a new head. The backbone calculates features whenever the new model runs. | Train the new head while the backbone's weights remain fixed. | Use a frozen image backbone followed by a new three-output head for acceptable, scratched and cracked products. The complete image-to-class pipeline is one model. | | Weight initialization followed by fine-tuning | Start from the pretrained weights, then allow selected pretrained layers to update. Replace or adapt the head as required. | Train the new head and the unfrozen part of the backbone. | After first training a product-defect head, unfreeze the last backbone block and train it with a smaller learning rate to adapt to the factory's photographs. | ### Why the existing class labels matter A cat/dog head has two outputs with learned meanings: cat and dog. Giving it a photograph of a scratched product does not create a new output called “scratched.” For the product task, use an appropriate new head and train it on labeled product examples. Direct reuse of the original classifier is suitable only when its output meanings and preprocessing match the required task. When the task and data distribution are unchanged, this is ordinary inference rather than adaptation to a new task. ### Standalone versus integrated features Both methods can keep the backbone frozen. In the standalone method, feature vectors are calculated first and saved as a separate dataset; a later classifier reads that dataset. In the integrated method, each image passes through the backbone and head within one pipeline. Freezing means that the backbone parameters receive no training updates; it does not mean that the backbone stops performing calculations or that its stored weights disappear. ### When transfer learning is useful It is useful when labeled target data are limited and a pretrained model has learned representations relevant to the target task. It can reduce training time and the amount of target data needed. Improvement is not guaranteed: unrelated source features, unsuitable preprocessing or excessive fine-tuning on a small dataset can hurt performance. Choose the frozen or fine-tuned configuration using validation data, then evaluate the chosen model on the held-out test set. **Exam answer structure:** define transfer learning → identify the backbone and head → explain the selected reuse mode → state which weights remain fixed and which learn → give a matching example and a limitation. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c9-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/9/mt2-c9-intro This recurrent-network material is retained as an additional reference. The faculty update received on 12 October focuses MT2 on Units 3 and 4; use the TEE route for RNN study. ![At each time step, combine the new input with a summary carried from the previous step.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: At each time step, combine the new input with a summary carried from the previous step.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. At each time step, combine the new input with a summary carried from the previous step. Tap the diagram to enlarge. --- ## MT2 · What makes a sequence different Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c9-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/9/mt2-c9-s0 **Definition.** A recurrent neural network (RNN) is a neural architecture that models sequential data by updating a hidden state from the current input and the previous state, using shared parameters across time steps. $$ h_t=\phi(W_{xh}x_t+W_{hh}h_{t-1}+b_h) $$ The state hₜ is a learned representation of information available up to time t. It is not a guaranteed lossless record of the entire sequence. The initial state is supplied, initialized to zero or learned, according to the model or question. ### Why order matters In language, audio and time series, the meaning of an observation can depend on earlier observations and their order. A basic feedforward network treating each input independently has no recurrent state. An RNN allows the output at a time step to depend on previous processing through hₜ₋₁. ![Unrolling shows successive evaluations of one recurrent cell with shared parameters.](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Unrolling shows successive evaluations of one recurrent cell with shared parameters.. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Unrolling shows successive evaluations of one recurrent cell with shared parameters. Tap the diagram to enlarge. ### Parameter sharing The unrolled cells use the same Wₓₕ, Wₕₕ and bₕ at every time step. A longer input sequence increases the number of evaluations, not the number of independently learned copies of these parameters. Each new hidden state must be calculated before it is used by the next step. ### MT2 interpretation The TEE route contains the state equation, time-step computations, shared parameters, sequence mappings, BPTT, vanishing/exploding gradients, LSTM and GRU. RNN is outside the current focused MT2 path based on the 12 October faculty update. --- ## MT2 · Model answer: why recurrent networks use a hidden state Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c9-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/9/mt2-c9-s1 ## Question **Question.** What is a recurrent neural network? Explain how a hidden state allows it to process a sequence and distinguish it from a feedforward network. ## Answer — step by step An RNN processes ordered inputs using a hidden state that is carried from one time step to the next. The current state combines information from the new input with information retained from earlier inputs. The same recurrent weights are reused across time steps. ![The previous hidden state brings earlier information into the current calculation alongside the new input.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: The previous hidden state brings earlier information into the current calculation alongside the new input.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. The previous hidden state brings earlier information into the current calculation alongside the new input. Tap the diagram to enlarge. For a simple tanh RNN: $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h) $$ | Symbol | Meaning | | --- | --- | | xₜ | Input at the current time step | | hₜ₋₁ | Hidden state passed from the previous step | | Wxh | Weights connecting the current input to the hidden state | | Whh | Weights connecting the previous state to the current state | | bₕ | Hidden-state bias | | hₜ | New state after the activation | For words in a sentence, earlier words affect the state used when the next word arrives. Changing the order can therefore change the final representation. The state is a learned summary; it is not a guaranteed exact copy of every earlier input. A feedforward network maps inputs through its layers without this recurrent state transition. An RNN can accept a sequence, reuse its parameters at each step and produce either an output at each step or an output from a final state, depending on the task. Examples include sentiment classification, next-word prediction and time-series modeling. Unrolling an RNN draws its repeated use at t = 1, 2, 3 and so on. Those boxes share parameters; drawing three time steps does not create three independently learned copies of Whh. --- ## MT2 · The vanilla RNN equations from scratch Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c9-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/9/mt2-c9-s2 **Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction. ### Hidden-state and output equations For a tanh recurrent layer: $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h $$ $$ h_t=\tanh(a_t) $$ aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit. For classification at time t, an output layer can produce logits and probabilities: $$ z_t=W_{hy}h_t+b_y,\qquad p_t=\operatorname{softmax}(z_t) $$ A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation. ![The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state. Tap the diagram to enlarge. ### Complete numerical: two recurrent time steps **Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation. ### Given h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0. x₁ = 1; x₂ = 2; activation = tanh. ### To find Find h₁ and h₂ in order. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step **Step 1: Time step 1 — calculate the preactivation.** $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ **Apply the recurrent activation.** $$ h_1=\tanh(0.5)\approx0.462117157 $$ This activated value, not a₁ = 0.5, is passed to the next time step. **Step 2: Time step 2 — combine the new input and previous state.** $$ a_2=0.5(2)+0.2h_1+0 $$ $$ a_2\approx1+0.2(0.462117157)\approx1.092423431 $$ **Apply tanh to the complete new preactivation.** $$ h_2=\tanh(a_2)\approx0.797760954 $$ | Time step | Current input | Previous state | Preactivation | New state | | --- | --- | --- | --- | --- | | 1 | 1 | 0 | 0.5 | 0.462117157 | | 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 | **Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update. --- ## MT2 · A complete vector RNN calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c9-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/9/mt2-c9-s3 **Question from R26 Q5a.** xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons. ### Given Input x = [0,1]ᵀ; previous h = [0,1]ᵀ. Wxh = $\begin{bmatrix}0.6 & 0.4\\0.3 & 0.7\end{bmatrix}$ . Whh = $\begin{bmatrix}0.5 & 0.3\\0.2 & 0.4\end{bmatrix}$ ; bias = 0. Apply tanh separately to the two sums. ### To find Find the new two-component hidden state; output-layer parameters are not supplied. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_deac39f14c.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Wxh | 0.6 | 0.4 | 0.3 | 0.7 | Whh | 0.5 | 0.3 | 0.2 | 0.4 Given matrices for this question Tap the diagram to enlarge. **Step 1 new input:** Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7]. **Step 2 old memory:** Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4]. **Step 3 add:** a = [0.4+0.3, 0.7+0.4] = [0.7,1.1]. **Step 4 activate each element:** h = [tanh(0.7), tanh(1.1)] = **[0.604368,0.800499]**. An output-layer prediction cannot be numerically calculated without its weights and bias. --- ## MT2 · Sequence mappings and applications Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c9-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/9/mt2-c9-s4 **Definition.** A sequence-model configuration specifies how many inputs and outputs are processed and how their time indices relate. One-to-many, many-to-one and many-to-many configurations correspond to different prediction tasks. ### Explanation and working ![The number and timing of inputs and outputs determine the sequence mapping](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: The number and timing of inputs and outputs determine the sequence mapping. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. The number and timing of inputs and outputs determine the sequence mapping Tap the diagram to enlarge. | Mapping | Meaning | Example and output | | --- | --- | --- | | One to many | One input conditions several outputs | Image captioning: image to a word sequence | | Many to one | Sequence summarized as one output | Review sentiment: words to positive neutral or negative | | Many to many aligned | Output associated with each input time | Video frame labeling or per-word tagging | | Many to many unaligned | Output sequence length may differ | Encoder-decoder translation | Music generation can use earlier notes to predict the next note repeatedly. Next-word prediction uses the prefix to produce a probability distribution over a vocabulary. A word with the largest predicted probability is a simple choice, but the precise prediction cannot be calculated without the learned model's weights. **Sentence explanation:** In “The dogs that bark near my house are loud,” the model should retain the plural subject across the intervening words when choosing “are”. A useful hidden representation should retain subject information while reading later words. This is an interpretation of what a trained model may learn, not a claim that an individual gate is guaranteed to represent grammar. **Sentiment model answer:** tokenize the review, map tokens to numerical embeddings, process them with a vanilla RNN, use a final state or pooled sequence representation, then a dense layer and softmax over three sentiment classes. Train with categorical cross-entropy on labeled reviews; evaluate on held-out data. A bidirectional model is not needed for this answer and is not explicitly listed in the current syllabus. --- ## MT2 · Shared weights and RNN applications Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c9-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/9/mt2-c9-s5 **Definition.** Temporal parameter sharing means that one recurrent transition function is reused at each sequence step. It permits variable sequence lengths without learning a separate transition matrix for each position. ### Explanation and working The RNN applies the same input and recurrent weights at every time step. The hidden state changes as new inputs arrive; the learned rule itself is reused. This is temporal weight sharing. ![The same learned rule combines previous state and new input at each time step.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: The same learned rule combines previous state and new input at each time step.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. The same learned rule combines previous state and new input at each time step. Tap the diagram to enlarge. For sentiment classification, a sequence of words leads to one sentiment label: many-to-one. For image captioning, an image representation conditions a sequence of words: one-to-many. These patterns describe how inputs and outputs are arranged, not separate update rules. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-intro Sources S03, S01 and the three Teams photographs supplied later. Use the short neuron and gradient questions first. The longer teacher examples are backup practice after completing the Unit 3 and Unit 4 priorities; their presence here is not a confirmed MT2 announcement. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## MT2 · MT1A Q2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s0 ## Question **Question (3 marks).** A sigmoid neuron has inputs [2,1,3], weights [0.4,−0.2,0.5] and bias 0.1. Compute its output. ## Answer — step by step ### Given Inputs x₁ = 2,x₂ = 1,x₃ = 3. Matching weights w₁ = 0.4,w₂ = −0.2,w₃ = 0.5. Bias b = 0.1; activation = sigmoid. ### To find Calculate net input z and activated output a. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. --- ## MT2 · MT1A Q3 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s1 ## Question **Question (3 marks).** For z = [−1,0,1], calculate sigmoid, softmax and Leaky ReLU with α = 0.01 (an explicit assumption because the paper does not give the slope). ## Answer — step by step ### Given Scores z = [−1,0,1]. Calculate sigmoid, softmax and Leaky ReLU. The paper omits the Leaky slope: α = 0.01 is the stated practice assumption. ### To find Calculate every component under each of the three activations. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step Sigmoid is applied separately: σ(−1) = $\frac{1}{1+e^{1}}$ = 0.268941421; σ(0) = 1/2 = 0.5; σ(1) = $\frac{1}{1+e^{-1}}$ = 0.731058579. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-2, -1, 0]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.135335283, 0.367879441, 1]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.090030573, 0.244728471, 0.665240956]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Leaky ReLU multiplies only negative values by α: [0.01 × (−1),0,1] = **[−0.01,0,1]**. Softmax normalizes the entire vector; independent sigmoid values need not sum to 1. --- ## MT2 · MT1A Q4 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s2 ## Question **Question (3 marks).** Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_bbb5f79ed7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.5 | W₂ | 0.8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [2]. W₁ is printed with the question. b₁ = [0.1]. W₂ is printed with the question. b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. ### To find Find all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.5 \times 2) + (0.1) = 1.1 $$ $$ h_{1} = \max(0,1.1) = 1.1 $$ **Output neuron 1:** $$ z_{o_1} = (0.8 \times 1.1) + (0.2) = 1.08 $$ $$ o_{1} = \max(0,1.08) = 1.08 $$ **Step 2 loss convention.** The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates. Prediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (1.08-1) \times 1 = 0.08 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**1.1**h1 × Receiving delta**0.08**δ at o1 = Connection gradient**0.088**h1 → o1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(0.08)\times(1.1)\approx 0.088 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.08** × Old outgoing weight**0.8** = Path contribution**0.064** ↓ Add every path contribution**0.064**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.8)\times(0.08)\approx 0.064 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx 0.064 $$ $$ \delta_{h_1} = 0.064 \times 1 = 0.064 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**2**x1 × Receiving delta**0.064**δ at h1 = Connection gradient**0.128**x1 → h1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.064)\times(2)\approx 0.128 $$ **Requested answers:** dL/dw₂ = **0.088** and dL/db₂ = **0.08**. Hidden gradients above explain the dependency and are additional practice. --- ## MT2 · MT1B Q3 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s3 ## Question **Question (3 marks).** Calculate sigmoid, ReLU and tanh for z = [−2,0,2]. ## Answer — step by step ### Given Input scores = [−2,0,2]. Required activations: sigmoid, ReLU and tanh. ### To find Calculate all three activation outputs for each score. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a². z = value entering tanh; a = its output. A prime means derivative, or local rate of change. $$ a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}} $$ $$ \frac{da}{dz}=1-a^2 $$ **Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square. ### Solution — step by step ### Preactivation z = -2 **Step 1: negate z for sigmoid.** $$ -z=2 $$ **Step 2: evaluate the exponential.** $$ e^{-z}\approx7.389056099 $$ **Step 3: add one to form the denominator.** $$ 1+7.389056099=8.389056099 $$ **Step 4: take the reciprocal for sigmoid.** $$ \sigma(-2)=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: compare z with zero for ReLU.** $$ \mathrm{ReLU}(-2)=\max(0,-2)=0 $$ **Step 6: calculate the tanh numerator.** $$ e^{-2}-e^{2}\approx0.135335283-7.389056099=-7.253720816 $$ **Step 7: calculate the tanh denominator.** $$ e^{-2}+e^{2}\approx0.135335283+7.389056099=7.524391382 $$ **Step 8: divide for tanh.** $$ \tanh(-2)\approx\frac{-7.253720816}{7.524391382}\approx-0.964027580 $$ ### Preactivation z = 0 **Step 1: negate z for sigmoid.** $$ -z=0 $$ **Step 2: evaluate the exponential.** $$ e^{-z}\approx1.000000000 $$ **Step 3: add one to form the denominator.** $$ 1+1.000000000=2.000000000 $$ **Step 4: take the reciprocal for sigmoid.** $$ \sigma(0)=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: compare z with zero for ReLU.** $$ \mathrm{ReLU}(0)=\max(0,0)=0 $$ **Step 6: calculate the tanh numerator.** $$ e^{0}-e^{0}\approx1.000000000-1.000000000=0.000000000 $$ **Step 7: calculate the tanh denominator.** $$ e^{0}+e^{0}\approx1.000000000+1.000000000=2.000000000 $$ **Step 8: divide for tanh.** $$ \tanh(0)\approx\frac{0.000000000}{2.000000000}\approx0.000000000 $$ ### Preactivation z = 2 **Step 1: negate z for sigmoid.** $$ -z=-2 $$ **Step 2: evaluate the exponential.** $$ e^{-z}\approx0.135335283 $$ **Step 3: add one to form the denominator.** $$ 1+0.135335283=1.135335283 $$ **Step 4: take the reciprocal for sigmoid.** $$ \sigma(2)=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: compare z with zero for ReLU.** $$ \mathrm{ReLU}(2)=\max(0,2)=2 $$ **Step 6: calculate the tanh numerator.** $$ e^{2}-e^{-2}\approx7.389056099-0.135335283=7.253720816 $$ **Step 7: calculate the tanh denominator.** $$ e^{2}+e^{-2}\approx7.389056099+0.135335283=7.524391382 $$ **Step 8: divide for tanh.** $$ \tanh(2)\approx\frac{7.253720816}{7.524391382}\approx0.964027580 $$ **Final answer:** sigmoid = [0.119202922, 0.5, 0.880797078]; ReLU = [0, 0, 2]; tanh = [−0.964027580, 0, 0.964027580]. --- ## MT2 · MT1B Q4 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s4 ## Question **Question (3 marks).** Input x = [3]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ec03263237.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.4 | W₂ | 0.6 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [3]. W₁ is printed with the question. b₁ = [0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. ### To find Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.4 \times 3) + (0.2) = 1.4 $$ $$ h_{1}=\frac{1}{1+e^{-(1.4)}}=0.802183889 $$ **Output neuron 1:** $$ z_{o_1} = (0.6 \times 0.802183889) + (0.1) = 0.581310333 $$ $$ o_{1}=\frac{1}{1+e^{-(0.581310333)}}=0.641368858 $$ **Step 2 loss convention.** Use the half squared loss stated in this question. No learning rate is supplied. Prediction minus target = [-0.358631142]. Squared errors = [0.128616296]. Half-sum squared loss = 0.064308148; mean squared error = 0.128616296. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.641368858-1) \times 0.230014846 = -0.082490487 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.802183889**h1 × Receiving delta**−0.082490487**δ at o1 = Connection gradient**−0.066172539**h1 → o1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.082490487)\times(0.802183889)\approx -0.066172539 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.082490487** × Old outgoing weight**0.6** = Path contribution**−0.049494292** ↓ Add every path contribution**−0.049494292**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.6)\times(-0.082490487)\approx -0.049494292 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.049494292 $$ $$ \delta_{h_1} = -0.049494292 \times 0.158684897 = -0.007853997 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**3**x1 × Receiving delta**−0.007853997**δ at h1 = Connection gradient**−0.02356199**x1 → h1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.007853997)\times(3)\approx -0.02356199 $$ **Requested answers:** dL/dw₂ = **−0.066172539**; dL/db₂ = **−0.082490487**. Negative gradients mean ordinary gradient descent would increase those parameters. --- ## MT2 · Teacher vectorized ReLU example Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s5 ![The teacher vectorized example with its exact connections](https://deep-learning-study-room.vercel.app/assets/figures/net_handwritten.svg) Diagram: The teacher vectorized example with its exact connections. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=2 | x2=3 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.11 + x2 × 0.21; bias 0 | h2 incoming: x1 × 0.12 + x2 × 0.08; bias 0 | o1 incoming: h1 × 0.14 + h2 × 0.15; bias 0 | Use these OLD connections for all gradients in the same pass. The teacher vectorized example with its exact connections Tap the diagram to enlarge. **Question.** Input x = [2, 3]. Each row below lists the incoming weights of one receiving neuron. W₁ = $\begin{bmatrix}0.11 & 0.21\\0.12 & 0.08\end{bmatrix}$ ; b₁ = [0, 0]; W₂ = $\begin{bmatrix}0.14 & 0.15\end{bmatrix}$ ; b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes. ### Given Input x = [2, 3]. W₁ = $\begin{bmatrix}0.11 & 0.21\\0.12 & 0.08\end{bmatrix}$ . b₁ = [0, 0]. W₂ = $\begin{bmatrix}0.14 & 0.15\end{bmatrix}$ . b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.11 \times 2) + (0.21 \times 3) + (0) = 0.85 $$ $$ h_{1} = \max(0,0.85) = 0.85 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.12 \times 2) + (0.08 \times 3) + (0) = 0.48 $$ $$ h_{2} = \max(0,0.48) = 0.48 $$ **Output neuron 1:** $$ z_{o_1} = (0.14 \times 0.85) + (0.15 \times 0.48) + (0) = 0.191 $$ $$ o_{1} = \max(0,0.191) = 0.191 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.809]. Squared errors = [0.654481]. Half-sum squared loss = 0.3272405; mean squared error = 0.654481. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.191-1) \times 1 = -0.809 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.85**h1 × Receiving delta**−0.809**δ at o1 = Connection gradient**−0.68765**h1 → o1 Now update this weight Old weight**0.14** → New weight**0.1743825** Subtract η × gradient = **−0.0343825**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.809)\times(0.85)\approx -0.68765 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.14-0.05(-0.68765)\approx 0.1743825 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.48**h2 × Receiving delta**−0.809**δ at o1 = Connection gradient**−0.38832**h2 → o1 Now update this weight Old weight**0.15** → New weight**0.169416** Subtract η × gradient = **−0.019416**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.809)\times(0.48)\approx -0.38832 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.15-0.05(-0.38832)\approx 0.169416 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.809** × Old outgoing weight**0.14** = Path contribution**−0.11326** ↓ Add every path contribution**−0.11326**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.14)\times(-0.809)\approx -0.11326 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.11326 $$ $$ \delta_{h_1} = -0.11326 \times 1 = -0.11326 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.809** × Old outgoing weight**0.15** = Path contribution**−0.12135** ↓ Add every path contribution**−0.12135**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.15)\times(-0.809)\approx -0.12135 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.12135 $$ $$ \delta_{h_2} = -0.12135 \times 1 = -0.12135 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**2**x1 × Receiving delta**−0.11326**δ at h1 = Connection gradient**−0.22652**x1 → h1 Now update this weight Old weight**0.11** → New weight**0.121326** Subtract η × gradient = **−0.011326**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.11326)\times(2)\approx -0.22652 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.11-0.05(-0.22652)\approx 0.121326 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**3**x2 × Receiving delta**−0.11326**δ at h1 = Connection gradient**−0.33978**x2 → h1 Now update this weight Old weight**0.21** → New weight**0.226989** Subtract η × gradient = **−0.016989**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.11326)\times(3)\approx -0.33978 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.21-0.05(-0.33978)\approx 0.226989 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**2**x1 × Receiving delta**−0.12135**δ at h2 = Connection gradient**−0.2427**x1 → h2 Now update this weight Old weight**0.12** → New weight**0.132135** Subtract η × gradient = **−0.012135**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.12135)\times(2)\approx -0.2427 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.12-0.05(-0.2427)\approx 0.132135 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**3**x2 × Receiving delta**−0.12135**δ at h2 = Connection gradient**−0.36405**x2 → h2 Now update this weight Old weight**0.08** → New weight**0.0982025** Subtract η × gradient = **−0.0182025**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.12135)\times(3)\approx -0.36405 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.08-0.05(-0.36405)\approx 0.0982025 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0-0.05(-0.113260000)=0.005663000 $$ $$ b_{h_2,\mathrm{new}}=0-0.05(-0.121350000)=0.006067500 $$ $$ b_{o_1,\mathrm{new}}=0-0.05(-0.809000000)=0.040450000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Correction to the handwritten working:** the hidden deltas are negative (−0.11326 and −0.12135). A positive value at that step is a sign error. Keep output error as prediction minus target throughout. --- ## MT2 · Teams photos full sigmoid example Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c10-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/10/mt2-c10-s6 The three photographs show this 2 → 2 → 2 network and its chain rule. The photo label for w₃ is read as 0.25, consistent with the visible output/error values and the matching worked example in S05. Learning rate is not legible/specified in these photos; η = 0.5 below is an explicit practice assumption. **Question.** Input x = [0.05, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ = $\begin{bmatrix}0.15 & 0.2\\0.25 & 0.3\end{bmatrix}$ ; b₁ = [0.35, 0.35]; W₂ = $\begin{bmatrix}0.4 & 0.45\\0.5 & 0.55\end{bmatrix}$ ; b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. Compute all outputs, total half squared loss, all eight weight gradients and one simultaneous weight update. Map weights in order: w₁ = x₁ → h₁, w₂ = x₂ → h₁, w₃ = x₁ → h₂, w₄ = x₂ → h₂; w₅ = h₁ → o₁, w₆ = h₂ → o₁, w₇ = h₁ → o₂, w₈ = h₂ → o₂. ### Given Input x = [0.05, 0.1]. W₁ = $\begin{bmatrix}0.15 & 0.2\\0.25 & 0.3\end{bmatrix}$ . b₁ = [0.35, 0.35]. W₂ = $\begin{bmatrix}0.4 & 0.45\\0.5 & 0.55\end{bmatrix}$ . b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![Every arrow in the teacher example is labeled so the two backward paths can be traced](https://deep-learning-study-room.vercel.app/assets/figures/teams.svg) Diagram: Every arrow in the teacher example is labeled so the two backward paths can be traced. Labels: INPUTS | HIDDEN | OUTPUTS | x1 | x2 | h1 | h2 | o1 | o2 | 0.05 | 0.10 | w1 = .15 | w2 = .20 | w3 = .25 | w4 = .30 | w5 = .40 | w6 = .45 | w7 = .50 | w8 = .55 | target .01 | target .99 | Hidden biases = .35 Output biases = .60 | Sigmoid at both learned layers. Half summed squared loss. Every arrow in the teacher example is labeled so the two backward paths can be traced Tap the diagram to enlarge. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.15 \times 0.05) + (0.2 \times 0.1) + (0.35) = 0.3775 $$ $$ h_{1}=\frac{1}{1+e^{-(0.3775)}}=0.593269992 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.25 \times 0.05) + (0.3 \times 0.1) + (0.35) = 0.3925 $$ $$ h_{2}=\frac{1}{1+e^{-(0.3925)}}=0.596884378 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.593269992) + (0.45 \times 0.596884378) + (0.6) = 1.105905967 $$ $$ o_{1}=\frac{1}{1+e^{-(1.105905967)}}=0.75136507 $$ **Output neuron 2:** $$ z_{o_2} = (0.5 \times 0.593269992) + (0.55 \times 0.596884378) + (0.6) = 1.224921404 $$ $$ o_{2}=\frac{1}{1+e^{-(1.224921404)}}=0.772928465 $$ **Step 2 loss convention.** The photos explicitly use ½(t₁−o₁)² + ½(t₂−o₂)². With exactly two output components, this equals their mean squared error, 0.298371109. The unhalved sum is 0.596742218. Prediction minus target = [0.74136507, -0.217071535]. Squared errors = [0.549622166, 0.047120051]. Half-sum squared loss = 0.298371109; mean squared error = 0.298371109. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.75136507-0.01) \times 0.186815602 = 0.138498562 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.772928465-0.99) \times 0.175510053 = -0.038098237 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.593269992**h1 × Receiving delta**0.138498562**δ at o1 = Connection gradient**0.082167041**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.35891648** Subtract η × gradient = **0.04108352**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(0.138498562)\times(0.593269992)\approx 0.082167041 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.5(0.082167041)\approx 0.35891648 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.596884378**h2 × Receiving delta**0.138498562**δ at o1 = Connection gradient**0.082667628**h2 → o1 Now update this weight Old weight**0.45** → New weight**0.408666186** Subtract η × gradient = **0.041333814**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(0.138498562)\times(0.596884378)\approx 0.082667628 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.45-0.5(0.082667628)\approx 0.408666186 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.593269992**h1 × Receiving delta**−0.038098237**δ at o2 = Connection gradient**−0.02260254**h1 → o2 Now update this weight Old weight**0.5** → New weight**0.51130127** Subtract η × gradient = **−0.01130127**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(-0.038098237)\times(0.593269992)\approx -0.02260254 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.5-0.5(-0.02260254)\approx 0.51130127 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.596884378**h2 × Receiving delta**−0.038098237**δ at o2 = Connection gradient**−0.022740242**h2 → o2 Now update this weight Old weight**0.55** → New weight**0.561370121** Subtract η × gradient = **−0.011370121**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(-0.038098237)\times(0.596884378)\approx -0.022740242 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.55-0.5(-0.022740242)\approx 0.561370121 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.138498562** × Old outgoing weight**0.4** = Path contribution**0.055399425** Output 2 Output delta**−0.038098237** × Old outgoing weight**0.5** = Path contribution**−0.019049118** ↓ Add every path contribution**0.036350306**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(0.138498562)\approx 0.055399425 $$ **Contribution from output 2.** $$ C_2=(0.5)\times(-0.038098237)\approx -0.019049118 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.036350306 $$ $$ \delta_{h_1} = 0.036350306 \times 0.241300709 = 0.008771355 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.138498562** × Old outgoing weight**0.45** = Path contribution**0.062324353** Output 2 Output delta**−0.038098237** × Old outgoing weight**0.55** = Path contribution**−0.02095403** ↓ Add every path contribution**0.041370323**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.45)\times(0.138498562)\approx 0.062324353 $$ **Contribution from output 2.** $$ C_2=(0.55)\times(-0.038098237)\approx -0.02095403 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.041370323 $$ $$ \delta_{h_2} = 0.041370323 \times 0.240613417 = 0.009954255 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.05**x1 × Receiving delta**0.008771355**δ at h1 = Connection gradient**0.000438568**x1 → h1 Now update this weight Old weight**0.15** → New weight**0.149780716** Subtract η × gradient = **0.000219284**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.008771355)\times(0.05)\approx 0.000438568 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.15-0.5(0.000438568)\approx 0.149780716 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.1**x2 × Receiving delta**0.008771355**δ at h1 = Connection gradient**0.000877135**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199561432** Subtract η × gradient = **0.000438568**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.008771355)\times(0.1)\approx 0.000877135 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.5(0.000877135)\approx 0.199561432 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.05**x1 × Receiving delta**0.009954255**δ at h2 = Connection gradient**0.000497713**x1 → h2 Now update this weight Old weight**0.25** → New weight**0.249751144** Subtract η × gradient = **0.000248856**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.009954255)\times(0.05)\approx 0.000497713 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.25-0.5(0.000497713)\approx 0.249751144 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.1**x2 × Receiving delta**0.009954255**δ at h2 = Connection gradient**0.000995425**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.299502287** Subtract η × gradient = **0.000497712**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.009954255)\times(0.1)\approx 0.000995425 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.5(0.000995425)\approx 0.299502287 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.35-0.5(0.008771355)=0.345614322 $$ $$ b_{h_2,\mathrm{new}}=0.35-0.5(0.009954255)=0.345022872 $$ $$ b_{o_1,\mathrm{new}}=0.6-0.5(0.138498562)=0.530750719 $$ $$ b_{o_2,\mathrm{new}}=0.6-0.5(-0.038098237)=0.619049118 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Why the photos add two terms for h₁:** dL/dh₁ = δo₁w₅ + δo₂w₇ = 0.138498562 × 0.4 + (−0.038098237) × 0.5 = 0.055399425−0.019049118 = 0.036350306. Now multiply by h₁(1−h₁) = 0.241300709 to obtain δh₁ = 0.008771355; multiply by x₁ = 0.05 to obtain dL/dw₁ = 0.000438568. Adding the two output paths is essential. **Memory sequence:** error → output slope → outgoing weight → add paths → hidden slope → input. If you can explain why each factor appears, changed numbers or extra output neurons are much less confusing. **Bias convention:** the photo draws one bias value per layer. If these are literally tied scalars rather than equal initial per-neuron biases, sum the relevant deltas: hidden shared b becomes 0.35−0.5(0.008771355+0.009954255) = 0.340637195; output shared b becomes 0.6−0.5(0.138498562−0.038098237) = 0.549799837. If only weights are requested, do not invent a bias update requirement. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-intro The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper. --- ## MT2 · F25 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s0 ## Question **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. --- ## MT2 · F25 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s1 ## Question **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. --- ## MT2 · F25 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s2 ## Question **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs. --- ## MT2 · F25 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s3 ## Question **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. --- ## MT2 · F25 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s4 ## Question **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters. ## Answer — step by step ### Given N = 1,000; mini-batch size = 100. J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9. Use signed velocity. ### To find Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Step 1: count full-batch updates.** All 1,000 samples contribute to one update: 1 per epoch. **Step 2: count stochastic updates.** One update per sample: 1,000 per epoch. **Step 3: count mini-batches.** $$ 1000/100=10 $$ Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ. ### Momentum iteration 1 **Step 1: evaluate the gradient at the current parameter.** $$ g_1=1 $$ **Step 2: retain the previous displacement.** $$ 0.9(0)=0 $$ **Step 3: scale the current gradient.** $$ 0.1(1)=0.1 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_1=0-0.1=-0.1 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_1=1+(-0.1)=0.9 $$ ### Momentum iteration 2 **Step 1: evaluate the gradient at the current parameter.** $$ g_2=0.9 $$ **Step 2: retain the previous displacement.** $$ 0.9(-0.1)=-0.09 $$ **Step 3: scale the current gradient.** $$ 0.1(0.9)=0.09 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_2=-0.09-0.09=-0.18 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_2=0.9+(-0.18)=0.72 $$ **Hyperparameters:** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update. --- ## MT2 · F25 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s5 ## Question **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD. ## Answer — step by step AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise: $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change. Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864. Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average. --- ## MT2 · F25 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s6 ## Question **Question (10 marks).** Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. Given matrices Input A Input A | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Stride 2; P = 0 (no padding). ## Answer — step by step ### Given Input 128 × 128 × 3. Conv K = 3, filters = 16, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 3, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Dense 240 units. Dense 120 units. Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. ### To find Find every layer shape, its parameter count and the network total. Find every convolution output cell. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**128 × 128 × 3**height × width × channels → Convolution**16 filters · 3 × 3 · S = 1 · P = 0** → Output shape**126 × 126 × 16**height × width × channels Weights**432** + Biases**16** = Trainable parameters**448** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **128 × 128 × 3**. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{128+2(0)-3}{1}\right\rfloor+1=126 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **126 × 126 × 16**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times3\times16=432 $$ $$ N_{\mathrm{biases}}=16 $$ $$ P_1=432+16=448 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**126 × 126 × 16**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**63 × 63 × 16**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **126 × 126 × 16**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{126-2}{2}\right\rfloor+1=63 $$ Pooling acts independently on each channel. The output is **63 × 63 × 16**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**63 × 63 × 16**height × width × channels → Convolution**32 filters · 3 × 3 · S = 1 · P = 0** → Output shape**61 × 61 × 32**height × width × channels Weights**4,608** + Biases**32** = Trainable parameters**4,640** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **63 × 63 × 16**. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{63+2(0)-3}{1}\right\rfloor+1=61 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **61 × 61 × 32**. **3. Count trainable values.** Each filter receives all 16 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times16\times32=4{,}608 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_3=4{,}608+32=4{,}640 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**61 × 61 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**30 × 30 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **61 × 61 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{61-2}{2}\right\rfloor+1=30 $$ Pooling acts independently on each channel. The output is **30 × 30 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 5 — flatten Layer 5: what changes and what is learned Input shape**30 × 30 × 32**height × width × channels → Flatten**Rearrange activations** → Output shape**28,800**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **30 × 30 × 32**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=30\times30\times32=28{,}800 $$ Output: a vector with **28,800 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 6 — fully connected layer Layer 6: what changes and what is learned Input shape**28,800**input features → Dense layer**240 neurons** → Output shape**240**output features Weights**6,912,000** + Biases**240** = Trainable parameters**6,912,240** Every output neuron receives every input feature, with one bias per output neuron. Input: **28,800 features**. Output: **240 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=28{,}800\times240=6{,}912{,}000 $$ $$ N_{\mathrm{biases}}=240 $$ $$ P_6=6{,}912{,}000+240=6{,}912{,}240 $$ ### Layer 7 — fully connected layer Layer 7: what changes and what is learned Input shape**240**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**28,800** + Biases**120** = Trainable parameters**28,920** Every output neuron receives every input feature, with one bias per output neuron. Input: **240 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=240\times120=28{,}800 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_7=28{,}800+120=28{,}920 $$ ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**120**input features → Dense layer**5 neurons** → Output shape**5**output features Weights**600** + Biases**5** = Trainable parameters**605** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **5 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times5=600 $$ $$ N_{\mathrm{biases}}=5 $$ $$ P_8=600+5=605 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{128+2 \times 0-3}{1}\right\rfloor+1$ | 126 x 126 x 16 | 432 | 16 | 448 | | 2 pool | $\left\lfloor\frac{126-2}{2}\right\rfloor+1$ | 63 x 63 x 16 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{63+2 \times 0-3}{1}\right\rfloor+1$ | 61 x 61 x 32 | 4,608 | 32 | 4,640 | | 4 pool | $\left\lfloor\frac{61-2}{2}\right\rfloor+1$ | 30 x 30 x 32 | 0 | 0 | 0 | | 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 | | 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 | | 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 | | 8 dense | 120 → 5 | 5 | 600 | 5 | 605 | **Parameter substitutions, in layer order:** Layer 1: (3 x 3 x 3 + 1) x 16 = 448. Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640. Layer 6: (28800 + 1) x 240 = 6,912,240. Layer 7: (240 + 1) x 120 = 28,920. Layer 8: (120 + 1) x 5 = 605. **Total:** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. **Separate convolution calculation:** **Step 1 shape:** $\left\lfloor\frac{5+2 \times 0-3}{2}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −2 | | 4 | 0 | −4 | | 3 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−2) = 1** Row 2**4 + 0 + (−4) = 0** Row 3**3 + 0 + (−1) = 2** **1 + 0 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(1)\times(0)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(2)\times(-2)=0 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(0)+(2)=3 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 2 | 0 | −1 | | 4 | 0 | 0 | | 1 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 + (−1) = 1** Row 2**4 + 0 + 0 = 4** Row 3**1 + 0 + (−1) = 0** **1 + 4 + 0 = 5** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(2)\times(0)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(0)\times(-2)=4 $$ **Window row 3.** $$ R_3=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(1)+(4)+(0)=5 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −1 | | 0 | 0 | 0 | | 1 | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−1) = 2** Row 2**0 + 0 + 0 = 0** Row 3**1 + 0 + 0 = 1** **2 + 0 + 1 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Window row 2.** $$ R_2=(0)\times(2)+(3)\times(0)+(0)\times(-2)=0 $$ **Window row 3.** $$ R_3=(1)\times(1)+(1)\times(0)+(0)\times(-1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(2)+(0)+(1)=3 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 1 | 0 | −1 | | 0 | 0 | −4 | | 0 | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 + (−1) = 0** Row 2**0 + 0 + (−4) = −4** Row 3**0 + 0 + (−3) = −3** **0 + (−4) + (−3) = −7** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0)\times(2)+(1)\times(0)+(2)\times(-2)=-4 $$ **Window row 3.** $$ R_3=(0)\times(1)+(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(0)+(-4)+(-3)=-7 $$ **Final output:** $\begin{bmatrix}3 & 5\\3 & -7\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. --- ## MT2 · F25 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c11-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/11/mt2-c11-s7 ## Question **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages. ## Answer — step by step A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification. ![One filter is reused at successive spatial positions](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: One filter is reused at successive spatial positions. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. One filter is reused at successive spatial positions Tap the diagram to enlarge. Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities. Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation. For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-intro The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper. --- ## MT2 · R26 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s0 ## Question **Question (5 marks).** Explain L1 and L2 regularization. Apply 50% dropout to outputs [0.8,−0.4,0.6,0.2] using an explicitly assumed mask. ## Answer — step by step ### Given Activations = [0.8,−0.4,0.6,0.2]; dropout rate = 50%. The worked example assumes mask [1,0,1,0] and uses inverted dropout. No numerical weights or λ are supplied for the regularization explanation. ### To find Explain L1/L2 and calculate post-dropout activations under the stated mask. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** This answer uses λ/2 times the sum of squared weights. Differentiating a square supplies a factor of 2, which cancels the one-half, so the added gradient is λw. Use this same convention in the explanation below. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\frac{\lambda}{2}\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+\lambda w_i $$ **Yaad rakhne ka cue:** L2 mein half diya hai: square ka 2 cancel, gradient mein lambda × weight. **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. ### Numerical with one explicitly chosen mask **Step 1: convert dropout probability to keep probability.** $$ q=1-0.5=0.5 $$ **Step 2: state a possible mask.** Choose [1,0,1,0]; 1 means keep and 0 means drop. The question allows an assumed mask; others give different realizations. **Step 3: multiply each activation by its mask entry.** $$ [0.8(1),(-0.4)(0),0.6(1),0.2(0)]=[0.8,0,0.6,0] $$ **Step 4: divide the kept values by q for inverted dropout.** $$ 0.8/0.5=1.6 $$ $$ 0.6/0.5=1.2 $$ **Step 5: assemble the training output.** $$ a_{\mathrm{train}}=[1.6,0,1.2,0] $$ **Final answer:** simple masking gives [0.8,0,0.6,0]; inverted training dropout gives [1.6,0,1.2,0]. At inference inverted dropout is disabled. --- ## MT2 · R26 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s1 ## Question **Question (5 marks).** Explain an RNN for positive, negative or neutral e-commerce review sentiment. Why are recurrence and word order relevant? ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. For this task select a many-to-one output: tokenize and embed the full review, update the recurrent state across its words, and classify the final sequence representation with a three-unit softmax. Train using one-hot sentiment labels and categorical cross-entropy. “Not worth buying” and “worth buying” illustrate why negation and order matter. --- ## MT2 · R26 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s2 ## Question **Question (10 marks).** Explain AlexNet regularization and training improvements; explain Inception multiscale features and 1 × 1 convolutions; recommend an architecture for large-scale image classification with justification. ## Answer — step by step AlexNet uses a sequence of convolutional layers followed by fully connected classification layers. Its influential design used ReLU activations, max pooling, dropout in the fully connected part, data augmentation and GPU training. ReLU's positive derivative supports learning compared with heavily saturated sigmoid hidden units. Augmentation supplies varied training views; dropout reduces co-adaptation; GPU computation makes a large network practical to train. ![The common classroom AlexNet layout contains five convolutions and three dense layers](https://deep-learning-study-room.vercel.app/assets/figures/alexnet.svg) Diagram: The common classroom AlexNet layout contains five convolutions and three dense layers. Labels: Input | 227×227×3 | Conv1 | 11×11, 96 | Pool | 3×3, s2 | Conv2 | 5×5, 256 | Pool | 3×3, s2 | Conv3 | 3×3, 384 | Conv4 | 3×3, 384 | Conv5 | 3×3, 256 | Pool + flat | 9216 | Dense | 4096 | Dense | 4096 | Output | 1000 | Read rows in order. Apply nonlinearities between learned layers. The common classroom AlexNet layout contains five convolutions and three dense layers Tap the diagram to enlarge. The common 227-pixel classroom layout is: Conv 11 stride 4 with 96 filters; pool 3 stride 2; Conv 5 padding 2 with 256 filters; pool 3 stride 2; three Conv 3 padding 1 layers with 384,384,256 filters; pool 3 stride 2; dense 4096; dense 4096; output 1000. Nonlinearities are applied between learned layers. Exact padding, grouping and input conventions must follow the question: the original model used channel groups in some layers, so parameter counts differ from a fully connected-channel simplification. AlexNet follows a mainly sequential feature-extraction path. An Inception module processes several spatial scales in parallel and concatenates them, often using 1 × 1 bottlenecks to reduce cost. Neither architecture is universally best; the task, resource budget and validation performance determine the choice. An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. **Recommendation:** an Inception-family model is a defensible choice when varied object scales and computation efficiency matter, because parallel scales with bottlenecks provide diverse features at controlled cost. Compare held-out accuracy, latency and memory with available alternatives. This is a justified choice, not a claim that Inception always beats AlexNet on every dataset. --- ## MT2 · R26 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s3 ## Question **Question (10 marks).** How should CNN filter counts be selected? Explain effects on feature extraction, dimensionality, computation/memory, overfitting and hierarchical learning. ## Answer — step by step Choose filter counts as architecture hyperparameters using task complexity, available data, validation performance and resource limits. There is no universal correct count. 1. More filters can represent a wider variety of learned features at a layer; redundant filters do not automatically improve accuracy. 2. Cout filters produce Cout output channels; spatial height/width still follow kernel, padding and stride. 3. A K × K layer uses K²CinCout weights plus Cout biases. More output channels increase current activations and the input-channel cost of the next convolution. 4. Extra capacity can overfit limited data, so use validation, augmentation, transfer learning or regularization. Too few channels may underfit. 5. Later layers often use more channels after spatial downsampling, allowing complex feature combinations at a smaller spatial size. This is a common design choice, not a compulsory rule. Example: a 3 × 3 layer with 16 input channels has (9 × 16+1) × 32 = 4,640 parameters for 32 filters, versus 9,280 for 64 filters. Output-channel count doubles while the same spatial-size formula applies. --- ## MT2 · R26 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s4 ## Question **Question (10 marks).** Compare batch GD and SGD for loss oscillations with a graph, memory and large datasets. Discuss learning rate extremes. For J(θ) = 0.7 θ³, θ₀ = 1, η = 0.1, perform two iterations. ## Answer — step by step ### Given J(θ) = 0.7θ³; θ₀ = 1; η = 0.1. ### To find Calculate the derivative and perform two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable. ![Illustrative full batch and stochastic traces distinguish smoothness from progress](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative full batch and stochastic traces distinguish smoothness from progress. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative full batch and stochastic traces distinguish smoothness from progress Tap the diagram to enlarge. Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory. Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories. Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD. **Step 1: differentiate the objective before substituting.** $$ J(\theta)=0.7\theta^3 $$ $$ g(\theta)=0.7(3)\theta^2=2.1\theta^2 $$ **Step 2: evaluate gradient at the original parameter.** $$ g_1=2.1(1)^2=2.1 $$ **Step 3: scale the first gradient.** $$ 0.1(2.1)=0.21 $$ **Step 4: subtract to obtain the first update.** $$ \theta_1=1-0.21=0.79 $$ **Step 5: square the changed parameter.** $$ 0.79^2=0.6241 $$ **Step 6: recalculate the gradient.** $$ g_2=2.1(0.6241)=1.31061 $$ **Step 7: scale the second gradient.** $$ 0.1(1.31061)=0.131061 $$ **Step 8: subtract from the first updated parameter.** $$ \theta_2=0.79-0.131061=0.658939 $$ **Final answer:** θ₁ = 0.79; θ₂ = 0.658939. No individual samples are supplied, so these are the stated scalar gradient updates. The cubic is unbounded below; two steps are not a convergence proof. --- ## MT2 · R26 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s5 ## Question **Question (10 marks).** Explain feedforward limitations for sequential context and how recurrence helps. Calculate h using tanh for Wxh is shown below, Whh is shown below, x = [0,1], hprevious = [0,1], zero bias. Explain next-word probabilities. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_deac39f14c.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Wxh | 0.6 | 0.4 | 0.3 | 0.7 | Whh | 0.5 | 0.3 | 0.2 | 0.4 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Wxh and Whh are printed in the question; use their receiving-neuron rows. Current input = [0,1]; previous hidden state = [0,1]; bias = 0. Activation = tanh. ### To find Calculate both entries of the new hidden state. Output-layer weights are not supplied. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. **Step 1: multiply the input vector into row 1.** $$ 0.6(0)+0.4(1)=0+0.4=0.4 $$ **Step 2: multiply the input vector into row 2.** $$ 0.3(0)+0.7(1)=0+0.7=0.7 $$ **Step 3: multiply previous state into recurrent row 1.** $$ 0.5(0)+0.3(1)=0+0.3=0.3 $$ **Step 4: multiply previous state into recurrent row 2.** $$ 0.2(0)+0.4(1)=0+0.4=0.4 $$ **Step 5: combine contributions for hidden unit 1.** $$ a_1=0.4+0.3+0=0.7 $$ **Step 6: combine contributions for hidden unit 2.** $$ a_2=0.7+0.4+0=1.1 $$ **Step 7: apply tanh separately to each complete sum.** $$ h_1=\tanh(0.7)\approx0.604367777 $$ $$ h_2=\tanh(1.1)\approx0.800499022 $$ **Final answer:** h ≈ [0.604367777,0.800499022]. The absent vocabulary output weights prevent calculating a numerical next-word probability list. --- ## MT2 · R26 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s6 ## Question **Question (10 marks).** Explain RMSProp, its update rule and accumulated squared gradients. Give advantages and disadvantages relative to SGD. ## Answer — step by step RMSProp keeps an exponentially weighted average of squared gradients, instead of AdaGrad's ever-growing sum. For each parameter, initialize s₀ = 0 and use: $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ ρ controls how strongly earlier squared gradients are retained. A large average magnitude produces a larger denominator and a smaller step. A small average magnitude produces a relatively larger step. Squaring happens before averaging; averaging signed gradients instead would allow cancellation and would not be RMSProp. Example: θ₀ = 1, η = 0.1, ρ = 0.9, s₀ = 0, g₁ = 2. Then s₁ = 0.9 × 0+0.1 × 4 = 0.4. θ₁ = 1−0.2/√0.4 = 0.683772234 when ε is negligible. If g₂ = 1, s₂ = 0.9 × 0.4+0.1 × 1 = 0.46 and θ₂ = 0.683772234−0.1/√0.46 = 0.536330278. Benefits include adapting to different coordinate scales and avoiding AdaGrad's permanent accumulation of all past squared gradients. Costs include extra state memory and tuning η, ρ and ε; noisy or badly chosen settings can still produce poor training. Unlike standard SGD it rescales each coordinate; unlike Adam it does not include a separate first-moment estimate in this basic form. --- ## MT2 · R26 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s7 ## Question **Question (10 marks).** For the binary CNN below, calculate output dimensions and trainable parameters after every layer. Input 256 × 256 × 3. → Conv K = 5, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 128, stride = 1, padding = 1. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 84 units. → Dense 32 units. → Dense 1 units. The printed output size is unspecified; assume one sigmoid output. Unspecified convolution padding is taken as valid. ## Answer — step by step ### Given Input 256 × 256 × 3. Conv K = 5, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 64, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 128, stride = 1, padding = 1. Pool K = 2, stride = 2, padding = 0. Dense 84 units. Dense 32 units. Dense 1 units. assume one sigmoid output. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**256 × 256 × 3**height × width × channels → Convolution**32 filters · 5 × 5 · S = 1 · P = 0** → Output shape**252 × 252 × 32**height × width × channels Weights**2,400** + Biases**32** = Trainable parameters**2,432** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **256 × 256 × 3**. Use 32 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{256+2(0)-5}{1}\right\rfloor+1=252 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **252 × 252 × 32**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times3\times32=2{,}400 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_1=2{,}400+32=2{,}432 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**252 × 252 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**126 × 126 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **252 × 252 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{252-2}{2}\right\rfloor+1=126 $$ Pooling acts independently on each channel. The output is **126 × 126 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**126 × 126 × 32**height × width × channels → Convolution**64 filters · 5 × 5 · S = 1 · P = 0** → Output shape**122 × 122 × 64**height × width × channels Weights**51,200** + Biases**64** = Trainable parameters**51,264** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **126 × 126 × 32**. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{126+2(0)-5}{1}\right\rfloor+1=122 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **122 × 122 × 64**. **3. Count trainable values.** Each filter receives all 32 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times32\times64=51{,}200 $$ $$ N_{\mathrm{biases}}=64 $$ $$ P_3=51{,}200+64=51{,}264 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**122 × 122 × 64**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**61 × 61 × 64**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **122 × 122 × 64**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{122-2}{2}\right\rfloor+1=61 $$ Pooling acts independently on each channel. The output is **61 × 61 × 64**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**61 × 61 × 64**height × width × channels → Convolution**128 filters · 5 × 5 · S = 1 · P = 1** → Output shape**59 × 59 × 128**height × width × channels Weights**204,800** + Biases**128** = Trainable parameters**204,928** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **61 × 61 × 64**. Use 128 filters of spatial size 5 × 5, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{61+2(1)-5}{1}\right\rfloor+1=59 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **59 × 59 × 128**. **3. Count trainable values.** Each filter receives all 64 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times64\times128=204{,}800 $$ $$ N_{\mathrm{biases}}=128 $$ $$ P_5=204{,}800+128=204{,}928 $$ ### Layer 6 — pooling Layer 6: what changes and what is learned Input shape**59 × 59 × 128**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**29 × 29 × 128**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **59 × 59 × 128**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{59-2}{2}\right\rfloor+1=29 $$ Pooling acts independently on each channel. The output is **29 × 29 × 128**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 28.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 7 — flatten Layer 7: what changes and what is learned Input shape**29 × 29 × 128**height × width × channels → Flatten**Rearrange activations** → Output shape**107,648**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **29 × 29 × 128**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=29\times29\times128=107{,}648 $$ Output: a vector with **107,648 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**107,648**input features → Dense layer**84 neurons** → Output shape**84**output features Weights**9,042,432** + Biases**84** = Trainable parameters**9,042,516** Every output neuron receives every input feature, with one bias per output neuron. Input: **107,648 features**. Output: **84 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=107{,}648\times84=9{,}042{,}432 $$ $$ N_{\mathrm{biases}}=84 $$ $$ P_8=9{,}042{,}432+84=9{,}042{,}516 $$ ### Layer 9 — fully connected layer Layer 9: what changes and what is learned Input shape**84**input features → Dense layer**32 neurons** → Output shape**32**output features Weights**2,688** + Biases**32** = Trainable parameters**2,720** Every output neuron receives every input feature, with one bias per output neuron. Input: **84 features**. Output: **32 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=84\times32=2{,}688 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_9=2{,}688+32=2{,}720 $$ ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**32**input features → Dense layer**1 neurons** → Output shape**1**output features Weights**32** + Biases**1** = Trainable parameters**33** Every output neuron receives every input feature, with one bias per output neuron. Input: **32 features**. Output: **1 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=32\times1=32 $$ $$ N_{\mathrm{biases}}=1 $$ $$ P_10=32+1=33 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{256+2 \times 0-5}{1}\right\rfloor+1$ | 252 x 252 x 32 | 2,400 | 32 | 2,432 | | 2 pool | $\left\lfloor\frac{252-2}{2}\right\rfloor+1$ | 126 x 126 x 32 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{126+2 \times 0-5}{1}\right\rfloor+1$ | 122 x 122 x 64 | 51,200 | 64 | 51,264 | | 4 pool | $\left\lfloor\frac{122-2}{2}\right\rfloor+1$ | 61 x 61 x 64 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{61+2 \times 1-5}{1}\right\rfloor+1$ | 59 x 59 x 128 | 204,800 | 128 | 204,928 | | 6 pool | $\left\lfloor\frac{59-2}{2}\right\rfloor+1$ | 29 x 29 x 128 | 0 | 0 | 0 | | 7 flat | 29 × 29 × 128 | 107648 | 0 | 0 | 0 | | 8 dense | 107648 → 84 | 84 | 9,042,432 | 84 | 9,042,516 | | 9 dense | 84 → 32 | 32 | 2,688 | 32 | 2,720 | | 10 dense | 32 → 1 | 1 | 32 | 1 | 33 | **Parameter substitutions, in layer order:** Layer 1: (5 x 5 x 3 + 1) x 32 = 2,432. Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264. Layer 5: (5 x 5 x 64 + 1) x 128 = 204,928. Layer 8: (107648 + 1) x 84 = 9,042,516. Layer 9: (84 + 1) x 32 = 2,720. Layer 10: (32 + 1) x 1 = 33. **Total:** 2,432 + 51,264 + 204,928 + 9,042,516 + 2,720 + 33 = **9,303,893 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. **Correction:** Conv 3 has (5 × 5 × 64+1) × 128 = **204,928** parameters, not 73,856 as printed in the key. Correct total is **9,303,893**. If binary classification is instead implemented with two softmax outputs, the last dense layer has(32+1) × 2 = 66 parameters and the total becomes 9,303,926; explicitly state the output convention. --- ## MT2 · R26 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c12-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/12/mt2-c12-s8 ## Question **Question (10 marks).** Explain transfer learning with CNNs and how to adapt a pretrained model to a new image classification task, including an example model. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. **Concrete example:** replace the 1000-class head of a pretrained ResNet with a 10-class vegetable head. Train the new head with the base frozen, then optionally fine-tune late residual blocks at a smaller rate. Input preprocessing must match the pretrained model. The 10 output probabilities sum to 1 for mutually exclusive classes. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-intro The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper. --- ## MT2 · F23 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s0 ## Question **Question (5 marks).** Identify the hyperparameters to tune when training an ANN and explain their significance. ## Answer — step by step Hyperparameters are chosen outside the ordinary weight-gradient updates. Learning rate controls step size; number of layers and units controls capacity; activation determines nonlinear behavior; batch size controls the data used per update and memory demand; epochs control training duration. Regularization strength, dropout rate and early-stopping patience affect generalization. Optimizer-specific choices include momentum β or Adam β₁, β₂,ε. Tune using validation data while keeping the final test set separate. Weights and biases are learned parameters; they are not hyperparameters merely because their initial values must be selected. --- ## MT2 · F23 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s1 ## Question **Question (5 marks).** Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Kernel K Kernel K | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1. ### To find Find every convolution output cell. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-3}{1}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 10 | 9 | 5 | | 5 | 14 | 6 | | 3 | 3 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**10 + 9 + 5 = 24** Row 2**5 + 14 + 6 = 25** Row 3**3 + 3 + 2 = 8** **24 + 25 + 8 = 57** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 1, column 1. **Window row 1.** $$ R_1=(10)\times(1)+(9)\times(1)+(5)\times(1)=24 $$ **Window row 2.** $$ R_2=(5)\times(1)+(7)\times(2)+(6)\times(1)=25 $$ **Window row 3.** $$ R_3=(3)\times(1)+(3)\times(1)+(2)\times(1)=8 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(24)+(25)+(8)=57 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **2–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 9 | 5 | 6 | | 7 | 12 | 7 | | 3 | 2 | 8 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**9 + 5 + 6 = 20** Row 2**7 + 12 + 7 = 26** Row 3**3 + 2 + 8 = 13** **20 + 26 + 13 = 59** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 1, column 2. **Window row 1.** $$ R_1=(9)\times(1)+(5)\times(1)+(6)\times(1)=20 $$ **Window row 2.** $$ R_2=(7)\times(1)+(6)\times(2)+(7)\times(1)=26 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(1)+(8)\times(1)=13 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(20)+(26)+(13)=59 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–4**, columns **1–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 5 | 7 | 6 | | 3 | 6 | 2 | | 3 | 2 | 6 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**5 + 7 + 6 = 18** Row 2**3 + 6 + 2 = 11** Row 3**3 + 2 + 6 = 11** **18 + 11 + 11 = 40** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 2, column 1. **Window row 1.** $$ R_1=(5)\times(1)+(7)\times(1)+(6)\times(1)=18 $$ **Window row 2.** $$ R_2=(3)\times(1)+(3)\times(2)+(2)\times(1)=11 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(1)+(6)\times(1)=11 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(18)+(11)+(11)=40 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–4**, columns **2–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 7 | 6 | 7 | | 3 | 4 | 8 | | 2 | 6 | 7 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**7 + 6 + 7 = 20** Row 2**3 + 4 + 8 = 15** Row 3**2 + 6 + 7 = 15** **20 + 15 + 15 = 50** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 2, column 2. **Window row 1.** $$ R_1=(7)\times(1)+(6)\times(1)+(7)\times(1)=20 $$ **Window row 2.** $$ R_2=(3)\times(1)+(2)\times(2)+(8)\times(1)=15 $$ **Window row 3.** $$ R_3=(2)\times(1)+(6)\times(1)+(7)\times(1)=15 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(20)+(15)+(15)=50 $$ **Final output:** $\begin{bmatrix}57 & 59\\40 & 50\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. --- ## MT2 · F23 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s2 ## Question **Question (10 marks).** Explain key advantages of Adam for ANN training and how it differs from SGD. ## Answer — step by step Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g. For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9. Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum. --- ## MT2 · F23 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s3 ## Question **Question (10 marks).** Determine feature-map dimensions and parameters at every layer of the pictured AlexNet-style CNN. The normalized architecture is Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units. ## Answer — step by step ### Given The normalized architecture is Input 227 × 227 × 3. Conv K = 11, filters = 96, stride = 4, padding = 0. Pool K = 3, stride = 2, padding = 0. Conv K = 5, filters = 256, stride = 1, padding = 2. Pool K = 3, stride = 2, padding = 0. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 256, stride = 1, padding = 1. Pool K = 3, stride = 2, padding = 0. Dense 4096 units. Dense 4096 units. Dense 1000 units. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**227 × 227 × 3**height × width × channels → Convolution**96 filters · 11 × 11 · S = 4 · P = 0** → Output shape**55 × 55 × 96**height × width × channels Weights**34,848** + Biases**96** = Trainable parameters**34,944** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **227 × 227 × 3**. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-11}{4}\right\rfloor+1=55 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **55 × 55 × 96**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=11\times11\times3\times96=34{,}848 $$ $$ N_{\mathrm{biases}}=96 $$ $$ P_1=34{,}848+96=34{,}944 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**55 × 55 × 96**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**27 × 27 × 96**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **55 × 55 × 96**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{55-3}{2}\right\rfloor+1=27 $$ Pooling acts independently on each channel. The output is **27 × 27 × 96**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**27 × 27 × 96**height × width × channels → Convolution**256 filters · 5 × 5 · S = 1 · P = 2** → Output shape**27 × 27 × 256**height × width × channels Weights**614,400** + Biases**256** = Trainable parameters**614,656** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **27 × 27 × 96**. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{27+2(2)-5}{1}\right\rfloor+1=27 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **27 × 27 × 256**. **3. Count trainable values.** Each filter receives all 96 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times96\times256=614{,}400 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_3=614{,}400+256=614{,}656 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**27 × 27 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**13 × 13 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **27 × 27 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{27-3}{2}\right\rfloor+1=13 $$ Pooling acts independently on each channel. The output is **13 × 13 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**884,736** + Biases**384** = Trainable parameters**885,120** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 256**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 256 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times256\times384=884{,}736 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_5=884{,}736+384=885{,}120 $$ ### Layer 6 — convolution Layer 6: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**1,327,104** + Biases**384** = Trainable parameters**1,327,488** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times384=1{,}327{,}104 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_6=1{,}327{,}104+384=1{,}327{,}488 $$ ### Layer 7 — convolution Layer 7: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**256 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 256**height × width × channels Weights**884,736** + Biases**256** = Trainable parameters**884,992** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 256**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times256=884{,}736 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_7=884{,}736+256=884{,}992 $$ ### Layer 8 — pooling Layer 8: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**6 × 6 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **13 × 13 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{13-3}{2}\right\rfloor+1=6 $$ Pooling acts independently on each channel. The output is **6 × 6 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 9 — flatten Layer 9: what changes and what is learned Input shape**6 × 6 × 256**height × width × channels → Flatten**Rearrange activations** → Output shape**9,216**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **6 × 6 × 256**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=6\times6\times256=9{,}216 $$ Output: a vector with **9,216 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**9,216**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**37,748,736** + Biases**4,096** = Trainable parameters**37,752,832** Every output neuron receives every input feature, with one bias per output neuron. Input: **9,216 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=9{,}216\times4{,}096=37{,}748{,}736 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_10=37{,}748{,}736+4{,}096=37{,}752{,}832 $$ ### Layer 11 — fully connected layer Layer 11: what changes and what is learned Input shape**4,096**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**16,777,216** + Biases**4,096** = Trainable parameters**16,781,312** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times4{,}096=16{,}777{,}216 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_11=16{,}777{,}216+4{,}096=16{,}781{,}312 $$ ### Layer 12 — fully connected layer Layer 12: what changes and what is learned Input shape**4,096**input features → Dense layer**1,000 neurons** → Output shape**1,000**output features Weights**4,096,000** + Biases**1,000** = Trainable parameters**4,097,000** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **1,000 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times1{,}000=4{,}096{,}000 $$ $$ N_{\mathrm{biases}}=1{,}000 $$ $$ P_12=4{,}096{,}000+1{,}000=4{,}097{,}000 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{227+2 \times 0-11}{4}\right\rfloor+1$ | 55 x 55 x 96 | 34,848 | 96 | 34,944 | | 2 pool | $\left\lfloor\frac{55-3}{2}\right\rfloor+1$ | 27 x 27 x 96 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{27+2 \times 2-5}{1}\right\rfloor+1$ | 27 x 27 x 256 | 614,400 | 256 | 614,656 | | 4 pool | $\left\lfloor\frac{27-3}{2}\right\rfloor+1$ | 13 x 13 x 256 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 884,736 | 384 | 885,120 | | 6 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 | | 7 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 256 | 884,736 | 256 | 884,992 | | 8 pool | $\left\lfloor\frac{13-3}{2}\right\rfloor+1$ | 6 x 6 x 256 | 0 | 0 | 0 | | 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 | | 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 | | 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 | | 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 | **Parameter substitutions, in layer order:** Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944. Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656. Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120. Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488. Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992. Layer 10: (9216 + 1) x 4096 = 37,752,832. Layer 11: (4096 + 1) x 4096 = 16,781,312. Layer 12: (4096 + 1) x 1000 = 4,097,000. **Total:** 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = **62,378,344 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. The count treats every listed convolution as connected to all input channels. Original grouped AlexNet implementations have different counts; do not insert grouping that is not shown in the question. --- ## MT2 · F23 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s4 ## Question **Question (10 marks).** For Inception V 1, justify different kernel sizes in one block, explain 1 × 1 convolutions and sketch one block. ## Answer — step by step An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. --- ## MT2 · F23 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s5 ## Question **Question (10 marks).** Explain how ResNet addresses vanishing gradients. Compare a block output with and without a residual connection. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. --- ## MT2 · F23 Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s6 ## Question **Question (10 marks).** Design and sketch a sentiment-analysis model whose input is a small paragraph. Explain how its output is generated. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. **Concrete many-to-one design:** tokenize paragraph → embeddings → vanilla RNN sequence processor → final hidden state → dense 3 → softmax for positive/neutral/negative. If the task specifies only two labels, use one sigmoid or two softmax outputs instead. Train with the matching cross-entropy. Keep token order, handle padding/masks consistently and evaluate held-out reviews. ![Sentiment uses the many-to-one pattern](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: Sentiment uses the many-to-one pattern. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. Sentiment uses the many-to-one pattern Tap the diagram to enlarge. --- ## MT2 · F23 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c13-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/13/mt2-c13-s7 ## Question **Question (10 marks).** Explain the bias-variance tradeoff. Fill effects on bias and variance for weight regularization, more hidden units, dropout and more training data from the same distribution. ## Answer — step by step Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data. ![Validation behavior distinguishes useful fitting from prolonged fitting to training details](https://deep-learning-study-room.vercel.app/assets/figures/overfitting.svg) Diagram: Validation behavior distinguishes useful fitting from prolonged fitting to training details. Labels: Loss | Epochs | Validation | Training | Best validation region | Save the best validation checkpoint; do not choose by test loss. Validation behavior distinguishes useful fitting from prolonged fitting to training details Tap the diagram to enlarge. Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws. Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate. Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting. | Change | Typical bias | Typical variance | | --- | --- | --- | | Stronger weight regularization | Increases | Decreases | | More hidden units | Decreases | Increases | | Dropout during training | Increases | Decreases | | More same-distribution training data | Approximately no change | Decreases | These are the conventional exam trends, assuming other conditions are comparable. They are not guarantees for every training run. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-intro The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper. --- ## MT2 · R24 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s0 ## Question **Question (5 marks).** A deep network strongly overfits its training set and performs poorly on unseen data. Explain remedies. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. Use a representative validation set to choose the remedy. Also inspect leakage, label quality and distribution mismatch. If the final test set was repeatedly used to tune changes, reserve a fresh untouched set for final reporting. --- ## MT2 · R24 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s1 ## Question **Question (5 marks).** For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_915e1db14b.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2. ### To find Find the requested pooled output(s). ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [21, 11, 12, 9] **Step 2: add all selected entries.** 21 + 11 + 12 + 9 = 53 **Step 3: divide by the 4 entries in this window.** 53/4 = **13.25**. Maximum = **21** ### Pooling window (1,2) **Step 1: select the entries.** [15, 10, 10, 11] **Step 2: add all selected entries.** 15 + 10 + 10 + 11 = 46 **Step 3: divide by the 4 entries in this window.** 46/4 = **11.5**. Maximum = **15** ### Pooling window (2,1) **Step 1: select the entries.** [11, 13, 7, 4] **Step 2: add all selected entries.** 11 + 13 + 7 + 4 = 35 **Step 3: divide by the 4 entries in this window.** 35/4 = **8.75**. Maximum = **13** ### Pooling window (2,2) **Step 1: select the entries.** [18, 12, 15, 12] **Step 2: add all selected entries.** 18 + 12 + 15 + 12 = 57 **Step 3: divide by the 4 entries in this window.** 57/4 = **14.25**. Maximum = **18** **Average output:** $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . **Max output:** $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. --- ## MT2 · R24 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s2 ## Question **Question.** Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999. **Syllabus-aligned extract:** source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated. ## Answer — step by step Use signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network. With β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time. --- ## MT2 · R24 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s3 ## Question **Question (10 marks).** Explain why regularization matters, list techniques and elaborate one. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. **Elaborating L2:** add(λ/2)Σw² to the data loss. For weight w = .5, data gradient.2, λ = .1, η = .1, total gradient = .2+.1 × .5 = .25, so wnew = .5−.1 × .25 = .475. Relative to the data-only update.48, the penalty pulls the weight further toward 0. Too large λ can remove useful signal, so tune on validation data. --- ## MT2 · R24 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s4 ## Question **Question (10 marks).** Calculate weights, biases and feature-map sizes of the CNN: Input 227 × 227 × 3. → Conv K = 9, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 120 units. → Dense 84 units. → Dense 10 units. ## Answer — step by step ### Given Input 227 × 227 × 3. Conv K = 9, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 64, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 64, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Dense 120 units. Dense 84 units. Dense 10 units. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**227 × 227 × 3**height × width × channels → Convolution**32 filters · 9 × 9 · S = 1 · P = 0** → Output shape**219 × 219 × 32**height × width × channels Weights**7,776** + Biases**32** = Trainable parameters**7,808** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **227 × 227 × 3**. Use 32 filters of spatial size 9 × 9, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-9}{1}\right\rfloor+1=219 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **219 × 219 × 32**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=9\times9\times3\times32=7{,}776 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_1=7{,}776+32=7{,}808 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**219 × 219 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**109 × 109 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **219 × 219 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{219-2}{2}\right\rfloor+1=109 $$ Pooling acts independently on each channel. The output is **109 × 109 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 108.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**109 × 109 × 32**height × width × channels → Convolution**64 filters · 5 × 5 · S = 1 · P = 0** → Output shape**105 × 105 × 64**height × width × channels Weights**51,200** + Biases**64** = Trainable parameters**51,264** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **109 × 109 × 32**. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{109+2(0)-5}{1}\right\rfloor+1=105 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **105 × 105 × 64**. **3. Count trainable values.** Each filter receives all 32 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times32\times64=51{,}200 $$ $$ N_{\mathrm{biases}}=64 $$ $$ P_3=51{,}200+64=51{,}264 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**105 × 105 × 64**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**52 × 52 × 64**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **105 × 105 × 64**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{105-2}{2}\right\rfloor+1=52 $$ Pooling acts independently on each channel. The output is **52 × 52 × 64**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 51.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**52 × 52 × 64**height × width × channels → Convolution**64 filters · 5 × 5 · S = 1 · P = 0** → Output shape**48 × 48 × 64**height × width × channels Weights**102,400** + Biases**64** = Trainable parameters**102,464** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **52 × 52 × 64**. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{52+2(0)-5}{1}\right\rfloor+1=48 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **48 × 48 × 64**. **3. Count trainable values.** Each filter receives all 64 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times64\times64=102{,}400 $$ $$ N_{\mathrm{biases}}=64 $$ $$ P_5=102{,}400+64=102{,}464 $$ ### Layer 6 — pooling Layer 6: what changes and what is learned Input shape**48 × 48 × 64**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**24 × 24 × 64**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **48 × 48 × 64**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{48-2}{2}\right\rfloor+1=24 $$ Pooling acts independently on each channel. The output is **24 × 24 × 64**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 7 — flatten Layer 7: what changes and what is learned Input shape**24 × 24 × 64**height × width × channels → Flatten**Rearrange activations** → Output shape**36,864**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **24 × 24 × 64**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=24\times24\times64=36{,}864 $$ Output: a vector with **36,864 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**36,864**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**4,423,680** + Biases**120** = Trainable parameters**4,423,800** Every output neuron receives every input feature, with one bias per output neuron. Input: **36,864 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=36{,}864\times120=4{,}423{,}680 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_8=4{,}423{,}680+120=4{,}423{,}800 $$ ### Layer 9 — fully connected layer Layer 9: what changes and what is learned Input shape**120**input features → Dense layer**84 neurons** → Output shape**84**output features Weights**10,080** + Biases**84** = Trainable parameters**10,164** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **84 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times84=10{,}080 $$ $$ N_{\mathrm{biases}}=84 $$ $$ P_9=10{,}080+84=10{,}164 $$ ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**84**input features → Dense layer**10 neurons** → Output shape**10**output features Weights**840** + Biases**10** = Trainable parameters**850** Every output neuron receives every input feature, with one bias per output neuron. Input: **84 features**. Output: **10 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=84\times10=840 $$ $$ N_{\mathrm{biases}}=10 $$ $$ P_10=840+10=850 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{227+2 \times 0-9}{1}\right\rfloor+1$ | 219 x 219 x 32 | 7,776 | 32 | 7,808 | | 2 pool | $\left\lfloor\frac{219-2}{2}\right\rfloor+1$ | 109 x 109 x 32 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{109+2 \times 0-5}{1}\right\rfloor+1$ | 105 x 105 x 64 | 51,200 | 64 | 51,264 | | 4 pool | $\left\lfloor\frac{105-2}{2}\right\rfloor+1$ | 52 x 52 x 64 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{52+2 \times 0-5}{1}\right\rfloor+1$ | 48 x 48 x 64 | 102,400 | 64 | 102,464 | | 6 pool | $\left\lfloor\frac{48-2}{2}\right\rfloor+1$ | 24 x 24 x 64 | 0 | 0 | 0 | | 7 flat | 24 × 24 × 64 | 36864 | 0 | 0 | 0 | | 8 dense | 36864 → 120 | 120 | 4,423,680 | 120 | 4,423,800 | | 9 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 | | 10 dense | 84 → 10 | 10 | 840 | 10 | 850 | **Parameter substitutions, in layer order:** Layer 1: (9 x 9 x 3 + 1) x 32 = 7,808. Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264. Layer 5: (5 x 5 x 64 + 1) x 64 = 102,464. Layer 8: (36864 + 1) x 120 = 4,423,800. Layer 9: (120 + 1) x 84 = 10,164. Layer 10: (84 + 1) x 10 = 850. **Total:** 7,808 + 51,264 + 102,464 + 4,423,800 + 10,164 + 850 = **4,596,350 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. --- ## MT2 · R24 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s5 ## Question **Question (10 marks).** A GoogLeNet classifier trains slowly and vanishing gradients are suspected. Suggest an architecture change and explain its important features. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. A ResNet is a suitable architecture to investigate because skip additions provide shorter gradient paths. First verify the diagnosis with gradients and training curves; slow training alone does not establish vanishing gradients. Match projection shortcuts when spatial size/channels change and compare validation performance after the redesign. --- ## MT2 · R24 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s6 ## Question **Question (10 marks).** A supermarket vegetable classifier has 10 classes but few examples per class and limited training resources. Explain different ways to use a pretrained model. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. For vegetables, replace the source head with 10 softmax outputs. Strategy 1: freeze the base and train only this head. Strategy 2: initialize from the same pretrained model and fine-tune selected later blocks at a small rate after head training. The first is cheaper and less prone to fitting all parameters from limited examples; the second can adapt features more closely to vegetables when validation supports it. --- ## MT2 · R24 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c14-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/14/mt2-c14-s7 ## Question **Question (10 marks).** Explain different RNN input/output sequence types with architecture sketches. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. ![One-to-many many-to-one and two many-to-many arrangements](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: One-to-many many-to-one and two many-to-many arrangements. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. One-to-many many-to-one and two many-to-many arrangements Tap the diagram to enlarge. One-to-many: one image representation generates a caption. Many-to-one: a review produces one sentiment. Aligned many-to-many: each video frame receives a label. Unaligned many-to-many: an encoder reads a source sentence and a decoder generates a translation of possibly different length. Specify which outputs carry losses during training; variable lengths do not require new weights for each position. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-intro The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper. --- ## MT2 · F22 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s0 ## Question **Question (4 marks).** A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3]. Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding. Compute the output and identify the feature. ## Answer — step by step ### Given A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3]. Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding. ### To find Find every convolution output cell. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{5+2 \times 0-3}{2}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.5 + 0 + (−0.3) = 0.2** Row 2**0.5 + 0 + (−0.3) = 0.2** Row 3**0.5 + 0 + (−0.3) = 0.2** **0.2 + 0.2 + 0.2 = 0.6** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 1, column 1. **Window row 1.** $$ R_1=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 2.** $$ R_2=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 3.** $$ R_3=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(0.2)+(0.2)+(0.2)=0.6 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.3 + 0 + (−0.3) = 0** Row 2**0.3 + 0 + (−0.3) = 0** Row 3**0.3 + 0 + (−0.3) = 0** **0 + 0 + 0 = 0** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 1, column 2. **Window row 1.** $$ R_1=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 3.** $$ R_3=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(0)+(0)+(0)=0 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.5 + 0 + (−0.3) = 0.2** Row 2**0.5 + 0 + (−0.3) = 0.2** Row 3**0.5 + 0 + (−0.3) = 0.2** **0.2 + 0.2 + 0.2 = 0.6** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 2, column 1. **Window row 1.** $$ R_1=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 2.** $$ R_2=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 3.** $$ R_3=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0.2)+(0.2)+(0.2)=0.6 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.3 + 0 + (−0.3) = 0** Row 2**0.3 + 0 + (−0.3) = 0** Row 3**0.3 + 0 + (−0.3) = 0** **0 + 0 + 0 = 0** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 2, column 2. **Window row 1.** $$ R_1=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 3.** $$ R_3=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(0)+(0)+(0)=0 $$ **Final output:** $\begin{bmatrix}0.6 & 0\\0.6 & 0\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. The filter compares left and right neighborhoods and responds to a vertical brightness boundary. Inputs are already normalized. No additional output-normalization rule is supplied, so the determined convolution is $\begin{bmatrix}0.6 & 0\\0.6 & 0\end{bmatrix}$ . If an examiner explicitly requests min-max normalization of this result, divide by 0.6 to obtain $\begin{bmatrix}1 & 0\\1 & 0\end{bmatrix}$ ; that is a separate stated operation. --- ## MT2 · F22 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s1 ## Question **Question (4 marks).** Weights are 0.55 and 0.65; their gradients are+0.3 and−0.25. With η = 0.1, compute gradient-descent changes and new weights. ## Answer — step by step ### Given Old weights = [0.55,0.65]. Corresponding supplied gradients = [0.3,−0.25]; η = 0.1. ### To find Calculate the signed weight changes and both new weights. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1: calculate the change in weight 1.** $$ \Delta w_1=-0.1(0.3)=-0.03 $$ **Step 2: add that change to old weight 1.** $$ w_1^{\mathrm{new}}=0.55+(-0.03)=0.52 $$ **Step 3: calculate the change in weight 2.** $$ \Delta w_2=-0.1(-0.25)=0.025 $$ **Step 4: add that change to old weight 2.** $$ w_2^{\mathrm{new}}=0.65+0.025=0.675 $$ **Final answer:** changes = [−0.03, +0.025]; new weights = [0.52, 0.675]. A negative gradient increases its weight under descent. --- ## MT2 · F22 Q1e Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s2 ## Question **Question (4 marks).** Suggest and sketch an RNN architecture for image captioning. ## Answer — step by step Use a CNN encoder to map the image to a visual representation and a recurrent decoder (vanilla RNN) to generate words one at a time. Condition the decoder’s initial state or its inputs on image features. At each step, vocabulary softmax predicts the next word based on the image and earlier words. Train using correct next-word targets and cross-entropy; stop generation at an end token. This is a one-to-many mapping from one image to a caption sequence. ![The one-to-many pattern models image captioning](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: The one-to-many pattern models image captioning. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. The one-to-many pattern models image captioning Tap the diagram to enlarge. --- ## MT2 · F22 Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s3 ## Question **Question (10 marks).** Explain bias, variance, underfitting and overfitting. A four-feature product-texture classifier has four grades including reject. Choose output activation/loss. Diagnose training error 15% and validation error 18% and suggest remedies. ## Answer — step by step Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data. ![Validation behavior distinguishes useful fitting from prolonged fitting to training details](https://deep-learning-study-room.vercel.app/assets/figures/overfitting.svg) Diagram: Validation behavior distinguishes useful fitting from prolonged fitting to training details. Labels: Loss | Epochs | Validation | Training | Best validation region | Save the best validation checkpoint; do not choose by test loss. Validation behavior distinguishes useful fitting from prolonged fitting to training details Tap the diagram to enlarge. Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws. Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate. Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting. **Classifier:** four mutually exclusive grades require 4 output scores, softmax probabilities and categorical cross-entropy. A reject class is still a labeled class in this formulation. **Diagnosis:** a modest 3 percentage-point gap with both errors appreciable suggests underfitting/high bias if a much better attainable baseline exists. Confirm learning curves and label quality. Improve relevant features or model capacity, train sufficiently, tune the optimizer and reduce excessive regularization. The percentages alone do not prove the cause. --- ## MT2 · F22 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s4 ## Question **Question (10 marks).** Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2; convolution 5 × 5: 16 filters, pooling 2; dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps. ## Answer — step by step ### Given Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2. convolution 5 × 5: 16 filters, pooling 2. dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**32 × 32 × 1**height × width × channels → Convolution**6 filters · 5 × 5 · S = 1 · P = 0** → Output shape**28 × 28 × 6**height × width × channels Weights**150** + Biases**6** = Trainable parameters**156** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **32 × 32 × 1**. Use 6 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{32+2(0)-5}{1}\right\rfloor+1=28 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **28 × 28 × 6**. **3. Count trainable values.** Each filter receives all 1 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times1\times6=150 $$ $$ N_{\mathrm{biases}}=6 $$ $$ P_1=150+6=156 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**28 × 28 × 6**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**14 × 14 × 6**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **28 × 28 × 6**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{28-2}{2}\right\rfloor+1=14 $$ Pooling acts independently on each channel. The output is **14 × 14 × 6**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**14 × 14 × 6**height × width × channels → Convolution**16 filters · 5 × 5 · S = 1 · P = 0** → Output shape**10 × 10 × 16**height × width × channels Weights**2,400** + Biases**16** = Trainable parameters**2,416** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **14 × 14 × 6**. Use 16 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{14+2(0)-5}{1}\right\rfloor+1=10 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **10 × 10 × 16**. **3. Count trainable values.** Each filter receives all 6 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times6\times16=2{,}400 $$ $$ N_{\mathrm{biases}}=16 $$ $$ P_3=2{,}400+16=2{,}416 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**10 × 10 × 16**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**5 × 5 × 16**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **10 × 10 × 16**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{10-2}{2}\right\rfloor+1=5 $$ Pooling acts independently on each channel. The output is **5 × 5 × 16**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — flatten Layer 5: what changes and what is learned Input shape**5 × 5 × 16**height × width × channels → Flatten**Rearrange activations** → Output shape**400**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **5 × 5 × 16**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=5\times5\times16=400 $$ Output: a vector with **400 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 6 — fully connected layer Layer 6: what changes and what is learned Input shape**400**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**48,000** + Biases**120** = Trainable parameters**48,120** Every output neuron receives every input feature, with one bias per output neuron. Input: **400 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=400\times120=48{,}000 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_6=48{,}000+120=48{,}120 $$ ### Layer 7 — fully connected layer Layer 7: what changes and what is learned Input shape**120**input features → Dense layer**84 neurons** → Output shape**84**output features Weights**10,080** + Biases**84** = Trainable parameters**10,164** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **84 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times84=10{,}080 $$ $$ N_{\mathrm{biases}}=84 $$ $$ P_7=10{,}080+84=10{,}164 $$ ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**84**input features → Dense layer**10 neurons** → Output shape**10**output features Weights**840** + Biases**10** = Trainable parameters**850** Every output neuron receives every input feature, with one bias per output neuron. Input: **84 features**. Output: **10 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=84\times10=840 $$ $$ N_{\mathrm{biases}}=10 $$ $$ P_8=840+10=850 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{32+2 \times 0-5}{1}\right\rfloor+1$ | 28 x 28 x 6 | 150 | 6 | 156 | | 2 pool | $\left\lfloor\frac{28-2}{2}\right\rfloor+1$ | 14 x 14 x 6 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{14+2 \times 0-5}{1}\right\rfloor+1$ | 10 x 10 x 16 | 2,400 | 16 | 2,416 | | 4 pool | $\left\lfloor\frac{10-2}{2}\right\rfloor+1$ | 5 x 5 x 16 | 0 | 0 | 0 | | 5 flat | 5 × 5 × 16 | 400 | 0 | 0 | 0 | | 6 dense | 400 → 120 | 120 | 48,000 | 120 | 48,120 | | 7 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 | | 8 dense | 84 → 10 | 10 | 840 | 10 | 850 | **Parameter substitutions, in layer order:** Layer 1: (5 x 5 x 1 + 1) x 6 = 156. Layer 3: (5 x 5 x 6 + 1) x 16 = 2,416. Layer 6: (400 + 1) x 120 = 48,120. Layer 7: (120 + 1) x 84 = 10,164. Layer 8: (84 + 1) x 10 = 850. **Total:** 156 + 2,416 + 48,120 + 10,164 + 850 = **61,706 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. This is the standard classroom full-channel interpretation with nontrainable pooling and a dense 10-class softmax. The historical original LeNet used partial channel connections and other details, so its published parameter count need not equal this diagram-based count. If an input label is read as 28, the stated later 28 map after valid 5 × 5 convolution is inconsistent; record the 32 input assumption rather than silently ignoring the mismatch. --- ## MT2 · F22 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s5 ## Question **Question (10 marks).** An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. Compute operations and explain 1 × 1 convolution. ## Answer — step by step ### Given An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. ### To find Find direct and bottleneck computation counts and the requested saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Direct path **Step 2: direct path.** output stays 28 × 28 with 32 channels. **Step 3: direct path.** MACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**. ### Bottleneck path with 16 reduction channels **Step 4: bottleneck path with 16 reduction channels.** 1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs. **Step 5: bottleneck path with 16 reduction channels.** The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs. **Step 6: bottleneck path with 16 reduction channels.** Total = **12,443,648**. ### Saving **Step 7: saving.** 120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r** **Step 8: saving.** there is no unique numeric bottleneck cost until r is chosen. **Step 9: calculate this quantity.** An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. **Step 10: calculate this quantity.** A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. --- ## MT2 · F22 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s6 ## Question **Question (10 marks).** Draw a 6-input, two-hidden-layer network with 5 neurons each and 4 classes, trained by AdaGrad. Identify learned parameters and tunable hyperparameters; no input-layer weights or biases. ## Answer — step by step ### Given 6 inputs → hidden 5 → hidden 5 → 4 output classes. AdaGrad is the specified training algorithm; one bias per receiving neuron. ### To find Draw the architecture and count its learned weights/biases. The optimizer name does not change the count. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step Architecture: 6 → 5 → 5 → 4. Each non-input neuron has one bias. ![F22 Q4b the input has six features and the output has four classes](https://deep-learning-study-room.vercel.app/assets/figures/dense_6_5_5_4.svg) Diagram: F22 Q4b the input has six features and the output has four classes. Labels: Input | 6 neurons | Hidden 1 | 5 neurons | Hidden 2 | 5 neurons | Output | 4 neurons | Every receiving neuron has one bias. All adjacent layers are fully connected. F22 Q4b the input has six features and the output has four classes Tap the diagram to enlarge. ### Layer 6 → 5 **Step 1: count connections.** $$ 6(5)=30 $$ **Step 2: count receiving-neuron biases.** There are 5 receiving neurons, so 5 biases. **Step 3: add weights and biases.** $$ 30+5=35 $$ ### Layer 5 → 5 **Step 1: count connections.** $$ 5(5)=25 $$ **Step 2: count receiving-neuron biases.** There are 5 receiving neurons, so 5 biases. **Step 3: add weights and biases.** $$ 25+5=30 $$ ### Layer 5 → 4 **Step 1: count connections.** $$ 5(4)=20 $$ **Step 2: count receiving-neuron biases.** There are 4 receiving neurons, so 4 biases. **Step 3: add weights and biases.** $$ 20+4=24 $$ **Total = 89**. Input values and activations are not trainable parameters. The learned parameters are input-to-hidden, hidden-to-hidden and hidden-to-output weights plus biases at receiving neurons. AdaGrad accumulators are optimizer state, not additional predictive weights. Hyperparameters include hidden widths/depth, learning rate η,ε, batch size, epochs, activations and regularization settings. Use 4-class softmax with categorical cross-entropy. AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise: $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change. Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864. Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average. --- ## MT2 · F22 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s7 ## Question **Question (10 marks).** Explain DL optimization challenges and the advantage of momentum over plain mini-batch gradient descent. ## Answer — step by step The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable. ![Illustrative full batch and stochastic traces distinguish smoothness from progress](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative full batch and stochastic traces distinguish smoothness from progress. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative full batch and stochastic traces distinguish smoothness from progress Tap the diagram to enlarge. Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory. Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories. Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD. Momentum uses vnew = βv−ηg and θ new = θ+vnew; consistent gradients accumulate direction, whereas some alternating components cancel. Plain mini-batch descent uses only the current batch gradient. Both can be combined: momentum with mini-batch gradients is a standard coherent choice. --- ## MT2 · F22 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s8 ## Question **Question (10 marks).** A CNN image classifier performs poorly because of vanishing gradients. Explain remedies. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. Also inspect saturated sigmoid/tanh hidden units, initialization and gradient magnitudes. Appropriate ReLU-family activations and initialization can help preserve usable gradients. Residual blocks require shape-compatible additions or projections. Verify that the diagnosis is correct; additional training time alone cannot repair gradients that are consistently negligible. --- ## MT2 · F22 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c15-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/15/mt2-c15-s9 ## Question **Question (10 marks).** Design a clothes classifier for 250,000 six-dimensional samples,6 classes and 2 hidden layers. Give train/validation/test split, activations/loss, actions for 15% training 20% validation error, interpret improvement to 1%/1.5%, then address 15% test error. ## Answer — step by step **One explicit design:** 6 → 16 → 8 → 6, ReLU hidden layers,6-unit softmax output, categorical cross-entropy. The hidden widths are chosen assumptions, not supplied numerical requirements. Architecture: 6 → 16 → 8 → 6. Each non-input neuron has one bias. ### Layer 6 → 16 **Step 1: count connections.** $$ 6(16)=96 $$ **Step 2: count receiving-neuron biases.** There are 16 receiving neurons, so 16 biases. **Step 3: add weights and biases.** $$ 96+16=112 $$ ### Layer 16 → 8 **Step 1: count connections.** $$ 16(8)=128 $$ **Step 2: count receiving-neuron biases.** There are 8 receiving neurons, so 8 biases. **Step 3: add weights and biases.** $$ 128+8=136 $$ ### Layer 8 → 6 **Step 1: count connections.** $$ 8(6)=48 $$ **Step 2: count receiving-neuron biases.** There are 6 receiving neurons, so 6 biases. **Step 3: add weights and biases.** $$ 48+6=54 $$ **Total = 302**. Input values and activations are not trainable parameters. **Split:** a stratified 70/15/15 split gives 175,000 training,37,500 validation,37,500 test samples. Fit preprocessing on training data only and keep related duplicates within a single split. **15%/20%:** inspect learning curves and baseline difficulty. Substantial training error suggests capacity/optimization or data-quality issues; the 5 point gap also suggests a generalization issue. Improve training fit, tune regularization and collect useful representative data instead of blindly increasing capacity. **1%/1.5%:** good comparable-set results with a small gap are encouraging; verify that validation has not leaked into training. **15% test:** inspect distribution shift, preprocessing differences, label quality, leakage or repeated validation overfitting. Compare per-class errors and collection conditions. Address the actual cause and collect representative data if necessary. If test results guide retuning, use a fresh untouched final test set. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-intro The source archive retains historical optimization, CNN and introductory RNN questions. Following the 12 October faculty update, the focused MT2 website presents the Unit 3–4 questions; retained RNN bookmarks lead to TEE. Original question IDs and marks are preserved. Full assessed tasks and numerical data are transcribed in readable language; worked solutions follow each question. This selection does not predict the paper. --- ## MT2 · R23 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s0 ## Question **Question (4 marks).** Suggest and sketch an RNN architecture to synthesize a musical tune. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. Represent notes as discrete tokens or suitable pitch-duration vectors. Train a recurrent decoder to predict the next note from preceding notes. A vocabulary softmax handles discrete note tokens; repeatedly feed generated notes to continue the tune until a stop condition. This is autoregressive sequence generation, typically a many-to-many next-token training pattern. ![Music generation can predict a next note at every recurrent step](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: Music generation can predict a next note at every recurrent step. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. Music generation can predict a next note at every recurrent step Tap the diagram to enlarge. --- ## MT2 · R23 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s1 ## Question **Question (4 marks).** Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below. Give the output and justify pooling. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_847554ea0e.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 2 | 3 | 2 | 5 | 5 | 3 | 4 | 6 | 3 | 7 | 4 | 8 | 5 | 9 | 8 | 5 | 10 | 4 | 2 | 9 | 10 | 3 | 4 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below. ### To find Find the requested pooled output(s). ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step **Shape:** $\left\lfloor\frac{5-3}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 2, 3, 5, 3, 4, 7, 4, 8] **Step 2: add all selected entries.** 1 + 2 + 3 + 5 + 3 + 4 + 7 + 4 + 8 = 37 **Step 3: divide by the 9 entries in this window.** 37/9 = **4.111111111** ### Pooling window (1,2) **Step 1: select the entries.** [3, 2, 5, 4, 6, 3, 8, 5, 9] **Step 2: add all selected entries.** 3 + 2 + 5 + 4 + 6 + 3 + 8 + 5 + 9 = 45 **Step 3: divide by the 9 entries in this window.** 45/9 = **5** ### Pooling window (2,1) **Step 1: select the entries.** [7, 4, 8, 8, 5, 10, 9, 10, 3] **Step 2: add all selected entries.** 7 + 4 + 8 + 8 + 5 + 10 + 9 + 10 + 3 = 64 **Step 3: divide by the 9 entries in this window.** 64/9 = **7.111111111** ### Pooling window (2,2) **Step 1: select the entries.** [8, 5, 9, 10, 4, 2, 3, 4, 3] **Step 2: add all selected entries.** 8 + 5 + 9 + 10 + 4 + 2 + 3 + 4 + 3 = 48 **Step 3: divide by the 9 entries in this window.** 48/9 = **5.333333333** **Average output:** $\begin{bmatrix}4.111111111 & 5\\7.111111111 & 5.333333333\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. --- ## MT2 · R23 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s2 ## Question **Question (4 marks).** Evaluate how mini-batch size affects optimization and parallel processing. ## Answer — step by step A mini-batch forms one gradient from B examples. Smaller B usually gives noisier gradients, lower activation memory and more updates per epoch. Larger B averages more examples, often uses vectorized hardware more efficiently up to resource limits, but needs more memory and gives fewer updates per epoch. N examples produce ceil(N/B) updates if the last partial batch is kept. Too-large B does not guarantee faster wall-clock progress or better generalization. Tune batch size with learning rate and resource constraints. Computation within a batch can be parallelized; successive optimizer steps remain dependent on the updated parameters. --- ## MT2 · R23 Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s3 ## Question **Question (10 marks).** A CNN with convolution, pooling, dense and output layers receives 127 × 127 images for 5 classes and trains with Adam. Identify learned parameters and tunable hyperparameters. ## Answer — step by step Convolutional kernel entries and filter biases are learned parameters. Dense weight matrices and biases are learned parameters. Pooling, flattening and ordinary activations have no learned weights. The output has 5 class scores and usually softmax with categorical cross-entropy. Adam’s m, v are running optimizer states, not extra predictive parameters. Hyperparameters include kernel sizes, filter counts, strides, padding, pooling configuration, dense widths, activations, learning rate, Adam β₁/β₂/ε, batch size, epochs and regularization. Architecture choices control representation capacity and resource demand; optimizer settings control update behavior. No filter counts or dense widths are specified, so a unique numerical parameter total cannot be calculated from 127 × 127 and 5 classes alone. Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g. For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9. Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum. --- ## MT2 · R23 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s4 ## Question **Question (10 marks).** Justify ReLU over sigmoid/tanh for deep hidden layers. For predicting brand ratings on a scale of 10, choose output activation and loss. Diagnose 2% training versus 18% validation error and suggest remedies. ## Answer — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. **Rating regression:** for a continuous score, one linear output with MSE is a standard baseline. If strict 0–10 bounds are required, use 10 × sigmoid(z) and the matching regression loss, noting saturation near the endpoints. If labels are categorical rating bins, a categorical formulation would be different; state the continuous-score assumption. **2% versus 18%:** if errors use the same comparable metric and distribution, this large generalization gap strongly suggests overfitting. Use appropriate regularization, early stopping, more representative data and possibly a smaller model; inspect leakage and preprocessing too. Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. --- ## MT2 · R23 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s5 ## Question **Question (10 marks).** Design employee face-recognition attendance using a CNN with too few training images. Explain solutions. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. --- ## MT2 · R23 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s6 ## Question **Question (10 marks).** Estimate parameters of the AlexNet diagram. Use Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units. ## Answer — step by step ### Given Use Input 227 × 227 × 3. Conv K = 11, filters = 96, stride = 4, padding = 0. Pool K = 3, stride = 2, padding = 0. Conv K = 5, filters = 256, stride = 1, padding = 2. Pool K = 3, stride = 2, padding = 0. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 256, stride = 1, padding = 1. Pool K = 3, stride = 2, padding = 0. Dense 4096 units. Dense 4096 units. Dense 1000 units. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**227 × 227 × 3**height × width × channels → Convolution**96 filters · 11 × 11 · S = 4 · P = 0** → Output shape**55 × 55 × 96**height × width × channels Weights**34,848** + Biases**96** = Trainable parameters**34,944** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **227 × 227 × 3**. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-11}{4}\right\rfloor+1=55 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **55 × 55 × 96**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=11\times11\times3\times96=34{,}848 $$ $$ N_{\mathrm{biases}}=96 $$ $$ P_1=34{,}848+96=34{,}944 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**55 × 55 × 96**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**27 × 27 × 96**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **55 × 55 × 96**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{55-3}{2}\right\rfloor+1=27 $$ Pooling acts independently on each channel. The output is **27 × 27 × 96**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**27 × 27 × 96**height × width × channels → Convolution**256 filters · 5 × 5 · S = 1 · P = 2** → Output shape**27 × 27 × 256**height × width × channels Weights**614,400** + Biases**256** = Trainable parameters**614,656** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **27 × 27 × 96**. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{27+2(2)-5}{1}\right\rfloor+1=27 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **27 × 27 × 256**. **3. Count trainable values.** Each filter receives all 96 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times96\times256=614{,}400 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_3=614{,}400+256=614{,}656 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**27 × 27 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**13 × 13 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **27 × 27 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{27-3}{2}\right\rfloor+1=13 $$ Pooling acts independently on each channel. The output is **13 × 13 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**884,736** + Biases**384** = Trainable parameters**885,120** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 256**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 256 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times256\times384=884{,}736 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_5=884{,}736+384=885{,}120 $$ ### Layer 6 — convolution Layer 6: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**1,327,104** + Biases**384** = Trainable parameters**1,327,488** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times384=1{,}327{,}104 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_6=1{,}327{,}104+384=1{,}327{,}488 $$ ### Layer 7 — convolution Layer 7: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**256 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 256**height × width × channels Weights**884,736** + Biases**256** = Trainable parameters**884,992** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 256**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times256=884{,}736 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_7=884{,}736+256=884{,}992 $$ ### Layer 8 — pooling Layer 8: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**6 × 6 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **13 × 13 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{13-3}{2}\right\rfloor+1=6 $$ Pooling acts independently on each channel. The output is **6 × 6 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 9 — flatten Layer 9: what changes and what is learned Input shape**6 × 6 × 256**height × width × channels → Flatten**Rearrange activations** → Output shape**9,216**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **6 × 6 × 256**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=6\times6\times256=9{,}216 $$ Output: a vector with **9,216 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**9,216**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**37,748,736** + Biases**4,096** = Trainable parameters**37,752,832** Every output neuron receives every input feature, with one bias per output neuron. Input: **9,216 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=9{,}216\times4{,}096=37{,}748{,}736 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_10=37{,}748{,}736+4{,}096=37{,}752{,}832 $$ ### Layer 11 — fully connected layer Layer 11: what changes and what is learned Input shape**4,096**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**16,777,216** + Biases**4,096** = Trainable parameters**16,781,312** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times4{,}096=16{,}777{,}216 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_11=16{,}777{,}216+4{,}096=16{,}781{,}312 $$ ### Layer 12 — fully connected layer Layer 12: what changes and what is learned Input shape**4,096**input features → Dense layer**1,000 neurons** → Output shape**1,000**output features Weights**4,096,000** + Biases**1,000** = Trainable parameters**4,097,000** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **1,000 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times1{,}000=4{,}096{,}000 $$ $$ N_{\mathrm{biases}}=1{,}000 $$ $$ P_12=4{,}096{,}000+1{,}000=4{,}097{,}000 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{227+2 \times 0-11}{4}\right\rfloor+1$ | 55 x 55 x 96 | 34,848 | 96 | 34,944 | | 2 pool | $\left\lfloor\frac{55-3}{2}\right\rfloor+1$ | 27 x 27 x 96 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{27+2 \times 2-5}{1}\right\rfloor+1$ | 27 x 27 x 256 | 614,400 | 256 | 614,656 | | 4 pool | $\left\lfloor\frac{27-3}{2}\right\rfloor+1$ | 13 x 13 x 256 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 884,736 | 384 | 885,120 | | 6 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 | | 7 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 256 | 884,736 | 256 | 884,992 | | 8 pool | $\left\lfloor\frac{13-3}{2}\right\rfloor+1$ | 6 x 6 x 256 | 0 | 0 | 0 | | 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 | | 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 | | 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 | | 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 | **Parameter substitutions, in layer order:** Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944. Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656. Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120. Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488. Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992. Layer 10: (9216 + 1) x 4096 = 37,752,832. Layer 11: (4096 + 1) x 4096 = 16,781,312. Layer 12: (4096 + 1) x 1000 = 4,097,000. **Total:** 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = **62,378,344 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. The diagram labels its final pooled map 6 × 6 × 256; a 3 × 3, stride 2 pool from 13 × 13 produces 6. If a small printed stride label is read as 3, it conflicts with the shown 6 × 6 map. The answer uses the consistent classic stride 2 interpretation. Original grouped AlexNet counts are a different architecture assumption. --- ## MT2 · R23 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s7 ## Question **Question (10 marks).** Suggest and justify remedies when (1) loss does not minimize after many epochs, (2) gradients are noisy and loss oscillates, (3) loss falls then rises because updates miss a minimum. ## Answer — step by step The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable. ![Illustrative full batch and stochastic traces distinguish smoothness from progress](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative full batch and stochastic traces distinguish smoothness from progress. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative full batch and stochastic traces distinguish smoothness from progress Tap the diagram to enlarge. Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory. Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories. Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD. **Scenario 1:** check correctness and gradient magnitudes, then tune learning rate, activation, initialization or optimizer. **Scenario 2:** consider an appropriate larger mini-batch or momentum; distinguish ordinary stochastic noise from divergence. **Scenario 3:** reduce learning rate or use a schedule; momentum/Nesterov can help but excessive momentum can worsen overshoot. Judge training and validation curves separately. --- ## MT2 · R23 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s8 ## Question **Question (10 marks).** A 28 × 28 × 192 input is convolved with 32 same-padded 5 × 5 filters to produce 28 × 28 × 32. Count computations. What happens if a 1 × 1 convolution precedes it? The reduction-channel count is not supplied. ## Answer — step by step ### Given Input = 28 × 28 × 192; 32 same-padded 5 × 5 filters. A preceding 1 × 1 convolution would reduce to C channels. C is not supplied, so first give the symbolic bottleneck count; 16 channels is only an illustrative assumption. ### To find Calculate direct cost and express bottleneck cost in C; explain when the bottleneck saves computation. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Direct path **Step 2: direct path.** output stays 28 × 28 with 32 channels. **Step 3: direct path.** MACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**. ### Bottleneck path with 16 reduction channels **Step 4: bottleneck path with 16 reduction channels.** 1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs. **Step 5: bottleneck path with 16 reduction channels.** The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs. **Step 6: bottleneck path with 16 reduction channels.** Total = **12,443,648**. ### Saving **Step 7: saving.** 120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r** **Step 8: saving.** there is no unique numeric bottleneck cost until r is chosen. The 16-channel bottleneck is an explicitly chosen illustrative value, consistent with a related paper; the symbolic 777,728 r answer is required when r has not been specified. **Step 9: calculate this quantity.** An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. **Step 10: calculate this quantity.** A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. --- ## MT2 · R23 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c16-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/16/mt2-c16-s9 ## Question **Question (10 marks).** Training performance is good but unseen-data performance degrades. Identify the issue and explain ways to improve it. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. A comparable-distribution train/unseen gap suggests overfitting. Also check whether the new data have shifted, preprocessing differs or labels are inconsistent. Select regularization using validation performance, not final-test tuning. Improvement must be demonstrated on representative held-out examples. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-intro These 45 original drills cover the required maths, neuron/gradient prerequisites, optimization, regularization, CNNs and basic RNN forward passes. Each has a full solution. They are practice questions, not predictions. First learn the method, then cover the answer and solve. Start with drills 4–6, 11, 14–39, then the mixed drills 40–45. Use the earlier arithmetic drills whenever a sign, bracket or calculator entry blocks you. --- ## MT2 · How to solve a mixed numerical without guessing Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s0 A mixed question is a chain of familiar operations. First circle the requested final quantity. Underline the given activation, loss, padding, stride, threshold, learning rate and any fixed parameters. Draw a small box for each stage. Write the output of one box as the input to the next. Never skip a stage because its title looks unfamiliar. **If the question says output or prediction:** do only the forward chain, unless training is separately requested. For a neuron, multiply and add before activating. For a CNN, determine spatial shapes before flattening. For an RNN, carry the previous state into the next time step. **If it says train or update:** finish the entire old forward pass, calculate the stated loss derivative, move backward, store every gradient, then update together. If there are several output branches, add their effects at the shared hidden unit. For a batch, follow the stated sum or mean loss before updating. **If it mixes dropout and a loss:** apply the given training mask and scaling first; then predict from the resulting activations. The gradient to a dropped path is zero for that mask. If it also specifies L2, add its gradient to the data gradient before the optimizer step. If it also specifies momentum or Adam, carry their stored states as well as the parameter. **If it mixes convolution, pooling and a dense classifier:** output shape → next layer input shape → flatten length → dense scores → activation → loss. Count learned parameters separately from output values. Pooling and flattening have zero learned parameters in these questions. **If it changes a familiar problem:** an extra input adds another product; an extra output adds another backward path; a new channel adds another kernel sum; a new time step adds another shared-state calculation. The method stays the same. The drills below deliberately change signs, targets, loss scales and layer combinations. **Your completion rule:** solve a representative problem today without the answer, repeat it after several hours, then solve a changed-number or mixed version tomorrow. Mark the exact line where your reasoning failed. Correcting that line is what builds reliable recall; reading an open solution does not test it. --- ## MT2 · Drill 1 Signs and brackets Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s1 ## Question **Question.** Evaluate 2−0.1 × (−3), then (2−0.1) × (−3). Explain why they differ. ## Answer — step by step ### Given Expressions: 2−0.1×(−3) and (2−0.1)×(−3). ### To find Evaluate both and explain the effect of grouping. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step ### Meaning of the calculation A minus sign can mean subtraction or a negative number. Brackets tell us which part belongs together. These two expressions have different brackets, so they describe different calculations. ### Rule used Do brackets first, then multiplication, then addition or subtraction. Subtracting a negative number means adding its positive size. **Step 1: First expression: multiply before subtracting.** $$ 2-0.1\times(-3) $$ First multiply: a positive number times a negative number is negative. $$ 0.1\times(-3)=-0.3 $$ Put this complete result back into the expression. Keep its negative sign inside brackets. $$ 2-(-0.3)=2+0.3=2.3 $$ **Step 2: Second expression: finish the bracket first.** $$ 2-0.1=1.9 $$ Now multiply the whole bracket result by negative 3. $$ 1.9\times(-3)=-5.7 $$ **Final answer:** first expression = 2.3; second expression = −5.7. In a weight update, multiply learning rate by gradient first, then subtract that product from the old weight. --- ## MT2 · Drill 2 Half sum versus mean Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s2 ## Question **Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE. ## Answer — step by step ### Given Predictions = [0.2, 0.7, 0.4]. Targets = [0, 1, 1]; number of outputs n = 3. ### To find Find SSE, half SSE, MSE and the prediction gradients requested in the question. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step ### Given values and loss definitions There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i. $$ \mathrm{SSE}=\sum_{i=1}^{n}e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}2 $$ $$ \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}n $$ SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively. ### Step 1: calculate each residual and its square | Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual | | --- | --- | --- | --- | --- | | 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 | | 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 | | 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 | ### Step 2: sum the squared residuals $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel. ### Step 3: apply each requested normalization For half SSE, the denominator is 2: $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245 $$ For MSE, the denominator is the number of pairs, n = 3: $$ \mathrm{MSE}=\frac{0.49}{3}=0.163333333\ldots $$ **Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question. --- ## MT2 · Drill 3 Natural log and probability Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s3 ## Question **Question.** Compute −ln(0.9),−ln(0.1), and $e^{-\ln 2}$ . ## Answer — step by step ### Given Probabilities: 0.9 and 0.1. Exponential expression: exp(−ln 2). ### To find Find −ln(0.9), −ln(0.1) and exp(−ln 2). ### Formula — which one and why **Natural logarithms.** The natural logarithm reverses the exponential. A negative log turns a small correct-class probability into a larger positive loss. ln uses base e ≈ 2.71828. exp(a) means e raised to the power a. Keep the minus outside ln when the question says −ln(p). $$ -\ln p=\ln\frac1p $$ $$ e^{\ln a}=a $$ $$ e^{-\ln a}=\frac1a $$ **Yaad rakhne ka cue:** exp banata hai power; ln us power ko wapas nikaalta hai. ### Solution — step by step ### Meaning and formulas The natural logarithm ln tells us the power of e needed to obtain a number. It reverses the exponential operation. A calculator evaluates ln; it is not ordinary division. $$ \ln(e^a)=a,\qquad e^{\ln a}=a\quad(a>0) $$ $$ e^{-a}=\frac1{e^a} $$ ### Step 1: calculate the first logarithm, then its minus sign $$ \ln(0.9)\approx-0.105360516 $$ $$ -\ln(0.9)=-(-0.105360516)=0.105360516 $$ ### Step 2: calculate the second logarithm $$ \ln(0.1)\approx-2.302585093 $$ $$ -\ln(0.1)=-(-2.302585093)=2.302585093 $$ When these are correct-class probabilities, 0.1 represents a worse prediction than 0.9, so its negative-log loss is larger. ### Step 3: simplify the exponential using the inverse rule $$ e^{-\ln2}=\frac1{e^{\ln2}}=\frac12=0.5 $$ **Final answers:** 0.105360516, 2.302585093 and 0.5. Use the ln key; base-10 log gives different first and second answers. --- ## MT2 · Drill 4 Dot product and bias Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s4 ## Question **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU. ## Answer — step by step ### Given Inputs x = [2, −1, 0.5]. Weights w = [0.3, 0.4, −0.2]; bias b = 0.1. Activation: ReLU. ### To find Find the weighted sum z and ReLU output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero. $$ z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z) $$ Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output. ### Step 1: multiply the matching pairs $$ w_1x_1=0.3\times2=0.6 $$ $$ w_2x_2=0.4\times(-1)=-0.4 $$ $$ w_3x_3=(-0.2)\times0.5=-0.1 $$ ### Step 2: add all products and add bias once $$ z=0.6+(-0.4)+(-0.1)+0.1 $$ $$ z=0.2-0.1+0.1=0.2 $$ ### Step 3: apply the activation $$ a=\max(0,0.2)=0.2 $$ **Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive. --- ## MT2 · Drill 5 Matrix rows receive inputs Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s5 ## Question **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ed1fcbfdf5.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ. Activation: ReLU on each result. ### To find Find Wx+b and the activated output. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formula Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row. $$ z=Wx+b,\qquad a_j=\max(0,z_j) $$ $$ W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix} $$ ### Step 1: calculate receiving neuron 1 from row 1 $$ z_1=(1\times2)+(2\times1)+0.5 $$ $$ z_1=2+2+0.5=4.5 $$ ### Step 2: calculate receiving neuron 2 from row 2 $$ z_2=((-1)\times2)+(3\times1)+(-0.5) $$ $$ z_2=-2+3-0.5=0.5 $$ ### Step 3: apply ReLU to each entry $$ a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5 $$ $$ z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix} $$ The shape is 2 rows by 1 column: one output for each receiving row. --- ## MT2 · Drill 6 Chain rule on a tiny loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s6/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s6 ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. --- ## MT2 · Drill 7 Sigmoid with binary cross entropy Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s7/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s7 ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. --- ## MT2 · Drill 8 Activation contrast Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s8/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s8 ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. --- ## MT2 · Drill 9 Softmax and true class loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s9/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s9 ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. --- ## MT2 · Drill 10 Sigmoid neuron gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s10/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s10 ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. --- ## MT2 · Drill 11 Full ReLU backpropagation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s11/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s11 ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## MT2 · Drill 12 Two output sigmoid backpropagation Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s12/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s12 ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## MT2 · Drill 13 A negative ReLU blocks this path Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s13/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s13 ## Question **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2. ## Answer — step by step ### Given x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2. Hidden activation: ReLU; output activation: linear. Target y = 1; loss: half squared error. ### To find Find the gradients of w₁ and w₂ and explain any zero gradients. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output. $$ z_h=w_1x+b_1,\quad h=\max(0,z_h),\quad o=w_2h+b_2 $$ $$ L=\frac12(o-y)^2,\quad \delta_o=o-y $$ $$ \frac{\partial L}{\partial w_2}=\delta_o h,\quad \delta_h=\delta_o w_2\operatorname{ReLU}'(z_h),\quad \frac{\partial L}{\partial w_1}=\delta_h x $$ ### Step 1: hidden weighted sum and activation $$ z_h=(-0.4)\times2+0.1=-0.8+0.1=-0.7 $$ $$ h=\max(0,-0.7)=0 $$ Because the raw hidden score is negative, ReLU's derivative at this point is 0. ### Step 2: output prediction and output delta $$ o=0.5\times0+0.2=0.2 $$ $$ \delta_o=0.2-1=-0.8 $$ ### Step 3: output weight gradient $$ \frac{\partial L}{\partial w_2}=(-0.8)\times0=0 $$ This is zero because the input to that connection, h, is zero. ### Step 4: hidden delta and input weight gradient $$ \delta_h=(-0.8)\times0.5\times0=0 $$ $$ \frac{\partial L}{\partial w_1}=0\times2=0 $$ **Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero. --- ## MT2 · Drill 14 Batch and sequential SGD Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s14/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s14 ## Question **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch. ## Answer — step by step ### Given Model ŷ = wx; initial w = 0; η = 0.1. Samples in order: (x,y) = (1,1), then (2,2). Batch objective: mean half squared loss. ### To find Compare the weight after one batch update and one sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### Meaning and formulas For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample. $$ \hat y=wx,\quad L=\frac12(\hat y-y)^2,\quad g=(\hat y-y)x $$ $$ g_{\mathrm{batch}}=\frac{g_1+g_2}{2},\qquad w_{\mathrm{new}}=w-\eta g $$ **Step 1: Part A: one batch update from w = 0.** For sample (1,1), prediction = 0 × 1 = 0, so: $$ g_1=(0-1)\times1=-1 $$ For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so: $$ g_2=(0-2)\times2=-4 $$ Average these two gradients, then update once. $$ g_{\mathrm{batch}}=\frac{-1+(-4)}2=-2.5 $$ $$ w_{\mathrm{batch}}=0-0.1(-2.5)=0.25 $$ **Step 2: Part B: one SGD epoch, restarting at w = 0.** The first sample has gradient −1, just as above. $$ w_1=0-0.1(-1)=0.1 $$ For the second sample use this new weight, so prediction is: $$ \hat y_2=0.1\times2=0.2 $$ $$ g_2=(0.2-2)\times2=(-1.8)\times2=-3.6 $$ $$ w_2=0.1-0.1(-3.6)=0.1+0.36=0.46 $$ **Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates. --- ## MT2 · Drill 15 Incomplete mini batch Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s15/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s15 ## Question **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it. ## Answer — step by step ### Given N = 103 samples; B = 20 samples per mini-batch. E = 3 epochs. Compare both remainder policies: keep or drop the final 3 examples. ### To find Find total updates for full-batch GD, SGD, and both mini-batch policies. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step ### Meaning and formulas An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs. $$ U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NE $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained. ### Step 1: batch GD All 103 samples form one batch per epoch. $$ U_{\mathrm{batch}}=3\times1=3 $$ ### Step 2: SGD Each sample gets its own update. $$ U_{\mathrm{SGD}}=103\times3=309 $$ ### Step 3: mini-batches with the remainder $$ 103=5\times20+3 $$ The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give: $$ U_{\mathrm{keep}}=6\times3=18 $$ ### Step 4: mini-batches without the remainder Only the five complete batches are used each epoch. $$ U_{\mathrm{drop}}=5\times3=15 $$ **Final answer:** 3, 309, 18 and 15 updates respectively. --- ## MT2 · Drill 16 GD variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s16/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s16 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ \theta_1=1-0.1(1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ \theta_2=0.9-0.1(0.9)=0.81 $$ **Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## MT2 · Drill 17 GD variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s17/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s17 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ \theta_1=-0.5-0.1(-1.5)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ \theta_2=-0.35-0.1(-1.35)=-0.215 $$ **Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## MT2 · Drill 18 Momentum variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s18/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s18 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ **Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## MT2 · Drill 19 Momentum variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s19/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s19 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ v_2=0.9(0.15)-0.1(-1.35)=0.27 $$ $$ \theta_2=-0.35+(0.27)=-0.08 $$ **Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## MT2 · Drill 20 Nesterov variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s20/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s20 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=1+0.9(0)=1 $$ $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81 $$ $$ g_2=0.81-(0)=0.81 $$ $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ $$ \theta_2=0.9+(-0.171)=0.729 $$ **Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. --- ## MT2 · Drill 21 Nesterov variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s21/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s21 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=-0.5+0.9(0)=-0.5 $$ $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=-0.35+0.9(0.15)=-0.215 $$ $$ g_2=-0.215-(1)=-1.215 $$ $$ v_2=0.9(0.15)-0.1(-1.215)=0.2565 $$ $$ \theta_2=-0.35+(0.2565)=-0.0935 $$ **Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. --- ## MT2 · Drill 22 AdaGrad variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s22/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s22 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_1=0+(2)^2=4 $$ $$ \sqrt{G_1}=\sqrt{4}=2 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** Add the new squared gradient to all previous squares. $$ G_2=4+(1)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=0.9-\frac{0.1\times(1)}{\sqrt{5}}=0.85527864 $$ --- ## MT2 · Drill 23 AdaGrad variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s23/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s23 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Add the new squared gradient to all previous squares. $$ G_1=0+(-1)^2=1 $$ $$ \sqrt{G_1}=\sqrt{1}=1 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_2=1+(2)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=-0.4-\frac{0.1\times(2)}{\sqrt{5}}=-0.489442719 $$ --- ## MT2 · Drill 24 RMSProp variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s24/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s24 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(2)^2=0.4 $$ $$ \sqrt{s_1}=\sqrt{0.4}=0.632455532 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{0.4}}=0.683772234 $$ **Step 2: Iteration 2 with supplied gradient 1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.4)+0.1(1)^2=0.46 $$ $$ \sqrt{s_2}=\sqrt{0.46}=0.678232998 $$ $$ \theta_2=0.683772234-\frac{0.1\times(1)}{\sqrt{0.46}}=0.536330278 $$ --- ## MT2 · Drill 25 RMSProp variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s25/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s25 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(-1)^2=0.1 $$ $$ \sqrt{s_1}=\sqrt{0.1}=0.316227766 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{0.1}}=-0.183772234 $$ **Step 2: Iteration 2 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.1)+0.1(2)^2=0.49 $$ $$ \sqrt{s_2}=\sqrt{0.49}=0.7 $$ $$ \theta_2=-0.183772234-\frac{0.1\times(2)}{\sqrt{0.49}}=-0.46948652 $$ --- ## MT2 · Drill 26 Adam variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s26/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s26 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ --- ## MT2 · Drill 27 Adam variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s27/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s27 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ --- ## MT2 · Drill 28 Regularized objective Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s28/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s28 ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. --- ## MT2 · Drill 29 L1 update on positive and negative weights Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s29/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s29 ## Question **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero. ## Answer — step by step ### Given Old weights = [−0.5,0.3]. Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2. ### To find Find total gradients and the two updated weights. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formula The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j) $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed. **Step 1: Weight 1: old value −0.5.** $$ \lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1 $$ $$ g_{1,\mathrm{total}}=0.2+(-0.1)=0.1 $$ $$ w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52 $$ **Step 2: Weight 2: old value 0.3.** $$ \lambda\operatorname{sign}(0.3)=0.1(1)=0.1 $$ $$ g_{2,\mathrm{total}}=-0.1+0.1=0 $$ $$ w_{2,\mathrm{new}}=0.3-0.2(0)=0.3 $$ **Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero. --- ## MT2 · Drill 30 L2 update Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s30/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s30 ## Question **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw². ## Answer — step by step ### Given Old weights = [0.5,−0.2]; data gradients = [0.1,0.3]. λ = 0.2; η = 0.1; penalty = (λ/2)Σw². ### To find Find penalty gradients, total gradients and updated weights. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Formula and its reason The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda w_j $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ **Step 1: Weight 1.** $$ \lambda w_1=0.2\times0.5=0.1 $$ $$ g_{1,\mathrm{total}}=0.1+0.1=0.2 $$ $$ w_{1,\mathrm{new}}=0.5-0.1(0.2)=0.48 $$ **Step 2: Weight 2.** $$ \lambda w_2=0.2\times(-0.2)=-0.04 $$ $$ g_{2,\mathrm{total}}=0.3+(-0.04)=0.26 $$ $$ w_{2,\mathrm{new}}=-0.2-0.1(0.26)=-0.226 $$ **Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative. --- ## MT2 · Drill 31 Inverted dropout Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s31/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s31 ## Question **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference. ## Answer — step by step ### Given Activations = [2,−1,0,3]. Keep probability q = 0.75; keep mask = [1,0,1,1]. ### To find Find training outputs and state the inference rule. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step ### Meaning and formula The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved. $$ a'_j=\frac{m_j a_j}{q} $$ The multiplication is entry by entry; there is no dot-product sum. ### Step 1: apply the mask $$ [1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3] $$ ### Step 2: divide each result by 0.75 $$ a'_1=\frac2{0.75}=2.666666667 $$ $$ a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0 $$ $$ a'_4=\frac3{0.75}=4 $$ **Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero. ### During inference Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training. --- ## MT2 · Drill 32 Convolution with a negative coefficient Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s32/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s32 ## Question **Question.** Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | --- | --- | | 1 | −1 | | 0 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel K are printed with the question. Stride S = 1; padding P = 0; bias = 0. Use the kernel without flipping it. ### To find Calculate all nine output cells. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−2) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(-1)+(1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 3 = 3** **2 + 3 = 5** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(-1)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(3)=5 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 2 = 2** **(−1) + 2 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(-1)+(2)=1 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(-1)+(1)=0 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −3 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−3) = −2** Row 2**0 + 0 = 0** **(−2) + 0 = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(-1)=-2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(-2)+(0)=-2 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + (−2) = 1** Row 2**0 + 1 = 1** **1 + 1 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(1)+(1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | −1 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + (−1) = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(1)+(0)=1 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 2 = 2** **1 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(2)=3 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 3 = 3** **(−1) + 3 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(-1)+(3)=2 $$ **Final output:** $\begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. --- ## MT2 · Drill 33 Padding stride and channels Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s33/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s33 ## Question **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total. ## Answer — step by step ### Given Input shape = 31 × 31 × 3. Filter size = 5 × 5; number of filters = 8. Stride S = 2; padding P = 1 per side; one bias per filter. ### To find Find output shape, weights, biases and total parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning of each given value The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels. ### Formulas used $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ \text{weights}=K^2C_{\mathrm{in}}C_{\mathrm{out}},\quad \text{biases}=C_{\mathrm{out}} $$ $$ \text{parameters}=\text{weights}+\text{biases} $$ ### Step 1: find the output height and width Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33. $$ N_{\mathrm{out}}=\left\lfloor\frac{31+2(1)-5}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{28}{2}\right\rfloor+1=14+1=15 $$ The height uses the same calculation. **Output shape = 15 × 15 × 8.** ### Step 2: count weights in one filter Each filter covers 5 × 5 positions in each of 3 channels. $$ 5\times5\times3=75\text{ weights per filter} $$ ### Step 3: count all filters and their biases $$ \text{weights}=75\times8=600 $$ $$ \text{biases}=8,\qquad \text{total}=600+8=608 $$ **Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights. --- ## MT2 · Drill 34 Max and average pooling Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s34/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s34 ## Question **Question.** Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_563e4ee8f7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | A | 1 | 4 | 2 | 3 | 0 | 2 | 5 | 1 | 6 | 3 | 2 | 0 | 1 | 7 | 4 | 8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input A is printed with the question. Window size = 2 × 2; stride = 2; no padding. ### To find Find the complete max-pooling and average-pooling outputs. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 4, 0, 2] **Step 2: add all selected entries.** 1 + 4 + 0 + 2 = 7 **Step 3: divide by the 4 entries in this window.** 7/4 = **1.75**. Maximum = **4** ### Pooling window (1,2) **Step 1: select the entries.** [2, 3, 5, 1] **Step 2: add all selected entries.** 2 + 3 + 5 + 1 = 11 **Step 3: divide by the 4 entries in this window.** 11/4 = **2.75**. Maximum = **5** ### Pooling window (2,1) **Step 1: select the entries.** [6, 3, 1, 7] **Step 2: add all selected entries.** 6 + 3 + 1 + 7 = 17 **Step 3: divide by the 4 entries in this window.** 17/4 = **4.25**. Maximum = **7** ### Pooling window (2,2) **Step 1: select the entries.** [2, 0, 4, 8] **Step 2: add all selected entries.** 2 + 0 + 4 + 8 = 14 **Step 3: divide by the 4 entries in this window.** 14/4 = **3.5**. Maximum = **8** **Average output:** $\begin{bmatrix}1.75 & 2.75\\4.25 & 3.5\end{bmatrix}$ . **Max output:** $\begin{bmatrix}4 & 5\\7 & 8\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. --- ## MT2 · Drill 35 Complete small CNN Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s35/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s35 ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. --- ## MT2 · Drill 36 Reverse a spatial formula Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s36/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s36 ## Question **Question.** A stride-1, valid 3 × 3 convolution outputs 18 × 18 × 12. What input spatial size is needed? Is input channel count determined? ## Answer — step by step ### Given Output shape = 18 × 18 × 12. Kernel = 3 × 3; stride = 1; valid padding P = 0. Input channel count is not supplied. ### To find Solve for input height/width; state whether input channels and parameter count are uniquely determined. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Formula used For stride 1 and valid padding, the output width is input width minus kernel width plus one. $$ N_{\mathrm{out}}=N-K+1 $$ ### Step 1: substitute the known output and kernel sizes $$ 18=N-3+1=N-2 $$ ### Step 2: isolate the unknown input size Add 2 to both sides. Doing the same operation to both sides preserves the equality. $$ 18+2=N-2+2 $$ $$ N=20 $$ ### Step 3: interpret the channel information The 12 output channels tell us there are 12 filters. They do not say how many channels the input has. An RGB input and a many-channel intermediate map could both feed 12 filters. With one bias per filter, the parameter formula would be: $$ P=(3\times3\times C_{\mathrm{in}}+1)\times12 $$ **Final answer:** input spatial shape 20 × 20; input channel count is undetermined from these givens. --- ## MT2 · Drill 37 Bottleneck cost with new values Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s37/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s37 ## Question **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first. ## Answer — step by step ### Given Input = 16 × 16 × 64. Direct branch: 32 same-padded 3 × 3 filters. Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer. Spatial size remains 16 × 16; exclude bias additions. ### To find Find both MAC counts, absolute saving and percentage saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Meaning and formula A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ ### Step 1: direct 3 × 3 convolution Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 reduction from 64 to 8 channels $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ ### Step 3: 3 × 3 convolution now receives only 8 channels $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: add the two bottleneck stages $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: calculate the saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32. --- ## MT2 · Drill 38 Residual addition Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s38/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s38 ## Question **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block. ## Answer — step by step ### Given x = [2,−3,1]; F(x) = [−1,1,0.5]. ### To find Find x+F(x), apply ReLU, and compare with ReLU(F(x)). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join. $$ H(x)=x+F(x),\qquad a_j=\max(0,H_j) $$ ### Step 1: add each pair $$ H_1=2+(-1)=1 $$ $$ H_2=-3+1=-2 $$ $$ H_3=1+0.5=1.5 $$ The residual output before ReLU is **[1, −2, 1.5]**. ### Step 2: apply ReLU $$ [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5] $$ ### Step 3: compare with the plain branch Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**. Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation. --- ## MT2 · Drill 39 Two RNN steps Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s39/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s39 ## Question **Question.** h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂. ## Answer — step by step ### Given Initial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0. Inputs: x₁ = 1, x₂ = 2; activation: tanh. ### To find Calculate h₁, then use it to calculate h₂. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ### Meaning and formulas At time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\qquad h_t=\tanh(a_t) $$ The raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward. $$ \tanh(a)=\frac{e^{2a}-1}{e^{2a}+1} $$ ### Step 1: calculate the first raw sum $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ ### Step 2: activate it to obtain the first state $$ h_1=\tanh(0.5)=\frac{e^1-1}{e^1+1} $$ $$ h_1=\frac{2.718281828-1}{2.718281828+1}=0.462117157 $$ ### Step 3: use that state in the second raw sum $$ a_2=0.5(2)+0.2(0.462117157)+0 $$ $$ a_2=1+0.092423431=1.092423431 $$ ### Step 4: activate the second sum $$ h_2=\tanh(1.092423431)=0.797760954 $$ **Final answer:** h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5. --- ## MT2 · Drill 40 Mixed convolution pool dense and loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s40/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s40 ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. --- ## MT2 · Drill 41 Mixed backpropagation and L2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s41/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s41 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. --- ## MT2 · Drill 42 Mixed dropout and a momentum update Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s42/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s42 ## Question **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1. ## Answer — step by step ### Given Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5. Linear-output weights = [0.2,0.3]; bias = 0; target = 1. Half squared loss; update only the output weights. Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1. ### To find Find dropped/scaled activations, prediction, gradients, new velocities and new output weights. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Formulas used Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients. $$ a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j $$ The signed-velocity momentum convention is: $$ v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}} $$ ### Step 1: apply inverted dropout $$ a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0 $$ ### Step 2: prediction and error $$ o=0.2(2)+0.3(0)+0=0.4 $$ $$ o-y=0.4-1=-0.6 $$ ### Step 3: current weight gradients $$ g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0 $$ ### Step 4: remembered movement plus new correction $$ v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21 $$ $$ v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09 $$ ### Step 5: add velocities to old weights $$ w_{1,\mathrm{new}}=0.2+0.21=0.41 $$ $$ w_{2,\mathrm{new}}=0.3-0.09=0.21 $$ **Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero. --- ## MT2 · Drill 43 Mixed CNN dimensions softmax and loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s43/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s43 ## Question **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE. ## Answer — step by step ### Given Input = 8 × 8 × 3. Conv: kernel 3, stride 1, padding 1, 2 filters. Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs. Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]). ### To find Find shapes, parameter counts, probabilities and CCE. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Formulas used Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}} $$ Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.367879441, 1, 0.135335283]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Correct class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**. --- ## MT2 · Drill 44 Inception concatenation and parameter count Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s44/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s44 ## Question **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters. ## Answer — step by step ### Given Input = 10 × 10 × 8. Branch A: 1 × 1 convolution, 4 filters. Branch B: 3 × 3 same convolution, 6 filters. Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters. ### To find Find each branch shape and parameters, then the concatenated shape and total parameters. ### Formula — which one and why **Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches. C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions. $$ C_{\mathrm{out}}=\sum_k C_k $$ $$ P_{\mathrm{total}}=\sum_k P_k $$ **Yaad rakhne ka cue:** Channels jodo; height-width same rakho. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values. $$ C_{\mathrm{combined}}=C_A+C_B+C_C $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Assume one bias per filter. Pooling has no learned parameters. **Step 1: Branch A.** Its 4 filters produce shape 10 × 10 × 4. $$ P_A=(1\times1\times8+1)\times4=9\times4=36 $$ **Step 2: Branch B.** Its 6 filters produce shape 10 × 10 × 6. $$ P_B=(3\times3\times8+1)\times6=73\times6=438 $$ **Step 3: Branch C.** Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2. $$ P_C=0+(1\times1\times8+1)\times2=18 $$ **Step 4: Join the shapes and add the parameters.** $$ C_{\mathrm{combined}}=4+6+2=12 $$ $$ P_{\mathrm{total}}=36+438+18=492 $$ **Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches. --- ## MT2 · Drill 45 Projection shortcut Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s45/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s45 ## Question **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters. ## Answer — step by step ### Given Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16. Shortcut kernel = 1 × 1; stride 2; padding 0. Use one bias per projection filter. ### To find Choose the number of filters and find projection output shape and parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ ### Step 1: choose 16 filters One filter creates one channel, so 16 filters give the required 16 output channels. ### Step 2: use the specified 1 × 1 kernel and stride 2 $$ N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8 $$ The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1. ### Step 3: count weights and biases $$ \text{weights}=1\times1\times8\times16=128 $$ $$ \text{biases}=16,\qquad \text{total}=128+16=144 $$ **Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid. --- ## MT2 · Revising solved examples Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c17-s46/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/17/mt2-c17-s46 Revisit a direct example and then its changed-number version. Write the given values, the formula and the substitutions on separate lines. If an answer differs, compare from the first line until the difference appears. Use the mixed questions after the individual methods make sense. --- ## MT2 · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c18-intro/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/18/mt2-c18-intro These mocks follow the teacher's Unit 3–4 guidance and clarification relayed on 12 October. The 2-mark parts can be theory or a theory–numerical combination. For Q2–Q4, practise both stated arrangements: two numerical questions plus one theory question; or one mixed theory–numerical question, one numerical question and one theory question. Prepare the theory of both units as well as the numerical methods. Each rehearsal uses the established 10-mark layout: Q1(a) and Q1(b), 2 marks each, are compulsory; answer any TWO of Q2–Q4, 3 marks each. Allow 60 minutes. Mock A uses numerical + numerical + theory in Q2–Q4, including two Unit 4 numericals. Mock B uses mixed + numerical + theory, including Unit 3 optimization. The topic assigned to each question, and the marks split inside Mock B's mixed question, are practice choices rather than a fixed official blueprint. Unit 5 RNN and standalone Unit 2 ANN questions are not included here. Every answer is immediately below its question. The worked answers deliberately show more explanation than you need to copy during the exam. In the exam, retain the method, substitutions, intermediate values and labelled final answer. --- ## MT2 · MT2 Mock A: full questions and answers Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c18-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/18/mt2-c18-s0 60-minute rehearsal; attempt 10 marks. Q1(a) and Q1(b) are compulsory. Answer any TWO of Q2–Q4. This practice arrangement contains Unit 3 theory and two Unit 4 long numericals. ### Q1a **Q1a compulsory (2 marks). Explain the key difference between the AdaGrad and RMSProp accumulators. Why can AdaGrad's effective learning rate become very small during prolonged training?** **Answer — step by step.** ### 1. AdaGrad stores a running sum Let gₜ be the current gradient and Gₜ the stored sum of squared gradients. AdaGrad updates that sum as follows: $$ G_t=G_{t-1}+g_t^2 $$ Every added square is nonnegative, so G cannot decrease. All past squared gradients remain in the sum. ### 2. RMSProp stores a decaying average Let sₜ be the stored squared-gradient average and ρ the fraction of the previous average retained. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ RMSProp gradually reduces the influence of older squared gradients. Its accumulator can rise or fall, depending on the new squared gradient. ### 3. Explain the effect on step size AdaGrad's effective learning rate is: $$ \eta_{effective}=\frac{\eta}{\sqrt{G_t}+\epsilon} $$ Here η is the base learning rate and ε is a small positive number that stabilizes division. When G becomes large, its square root becomes large, so dividing by it produces a smaller effective learning rate. RMSProp's decaying average avoids this particular permanent-accumulation problem. **Final answer:** AdaGrad uses a nondecreasing sum of all squared gradients; RMSProp uses a decaying average. AdaGrad's growing denominator can make future updates very small. **Yaad rakho:** AdaGrad sab squares jodta rehta hai; RMSProp purane squares ka asar dheere kam karta hai. ### Q1b **Q1b compulsory (2 marks). Explain local connectivity and parameter sharing in a CNN. How do they reduce the number of learned parameters compared with a fully connected mapping of an image?** **Answer — step by step.** ### 1. Local connectivity Each convolutional output depends on a small spatial neighbourhood of the input, called its receptive field, rather than every pixel in the entire image. The filter spans the input channels within that neighbourhood. This limits the connections used to calculate each output. ### 2. Parameter sharing The same filter weights and bias are reused at every spatial position where that filter is applied. A CNN therefore learns one set of parameters for that filter, rather than a separate set for each image location. ### 3. Concrete image example An edge-detecting filter can identify a vertical boundary near the left of an image and then slide to look for a similar boundary near the right. Both locations use the same learned filter. The network does not need to learn an independent edge detector for every position. ### 4. Comparison with a fully connected mapping A fully connected layer generally learns a separate weight for every input-to-output connection. Convolution reduces this requirement in two ways: local connectivity restricts each output's input region, while parameter sharing reuses one filter across locations. More output positions create more activations and computation, but do not create new parameters for that shared filter. **Final answer:** local connectivity uses a limited input neighbourhood; parameter sharing reuses the filter's learned values across spatial positions. Together they can require far fewer parameters than a fully connected image mapping. **Yaad rakho:** Filter ke numbers ek baar seekhte hain; image par baar-baar wahi use hote hain. ### Q2 **Q2 optional (3 marks). An input of size 7 × 7 × 3 passes through a convolutional layer with two 3 × 3 filters, stride 2 and padding 1 on each side; then a 2 × 2 pooling layer with stride 2 and no padding; then flattening; then a dense layer with three output units. Find the output shape after every layer and the total trainable parameters. Include one bias per convolutional filter and per dense unit.** **Answer — step by step.** ### Given - Input: height 7, width 7, channels 3. - Convolution: kernel 3 × 3, filters 2, stride 2, padding 1 per side. - Pooling: window 2 × 2, stride 2, padding 0. - Flattening followed by 3 dense output units. - Include all specified biases. Pooling and flattening have no learned parameters. ### To find The convolution output, pooling output, flatten length, dense output length and total trainable parameter count. ### Formula — which one and why For either spatial axis, use the output-size formula. N is that axis's input size, K the window size, S its stride and P padding per side. The floor means discard any fractional part because a full window must fit. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Each convolution filter spans all input channels; each dense unit connects to every flattened input. Therefore: $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}},\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flattening simply counts all values in the preceding feature maps: $$ n_{\mathrm{flat}}=HWC $$ ### Solution — step by step ### Step 1: convolution height and width $$ N_{\mathrm{conv}}=\left\lfloor\frac{7+2(1)-3}{2}\right\rfloor+1=3+1=4 $$ The numerator is 7 + 2 − 3 = 6. Divide by stride 2 to get 3, then add 1 to get 4 positions. The same calculation applies to height and width. Two filters create two output channels. **Convolution output: 4 × 4 × 2.** ### Step 2: convolution parameters $$ \text{weights}=3\times3\times3\times2=54 $$ $$ \text{biases}=2,\qquad P_{\mathrm{conv}}=54+2=56 $$ ### Step 3: pooling shape and parameters The pooling input size is now 4, not the original 7. $$ N_{\mathrm{pool}}=\left\lfloor\frac{4-2}{2}\right\rfloor+1=1+1=2 $$ Pooling operates separately on each channel, so the two channels remain two. **Pooling output: 2 × 2 × 2. Pooling parameters: 0.** ### Step 4: flattening $$ n_{\mathrm{flat}}=2\times2\times2=8 $$ There are eight values to send to the dense layer. Flattening rearranges them into a list and learns no parameters. **Flatten output: 8 values. Flatten parameters: 0.** ### Step 5: dense output and parameters Each of the three dense units receives eight inputs and has one bias. $$ \text{weights}=8\times3=24,\quad\text{biases}=3,\quad P_{\mathrm{dense}}=27 $$ **Dense output: 3 values.** ### Step 6: total trainable parameters $$ P_{\mathrm{total}}=56+0+0+27=83 $$ **Final answer:** 4 × 4 × 2 → 2 × 2 × 2 → 8 → 3. **Total: 83 trainable parameters.** **Yaad rakho:** Pehle shape; phir weights. Ek layer ka output hi next layer ka input hai. ### Q3 **Q3 optional (3 marks). Apply the printed kernel K to input I using the deep-learning cross-correlation convention, with stride 1, no padding and zero bias. Do not flip K. Apply ReLU to the resulting map, followed by 2 × 2 max pooling with stride 2 and no padding. Show every convolution entry and find the final output.** $$ I=\begin{bmatrix}1&2&1\\0&1&3\\2&1&0\end{bmatrix},\quad K=\begin{bmatrix}1&-1\\0&1\end{bmatrix} $$ **Answer — step by step.** ### Given - Input I is the printed 3 × 3 matrix; kernel K is the printed 2 × 2 matrix. - Convolution stride 1, padding 0, bias 0; use K in its printed orientation. - Activation: ReLU. - Pooling: max, 2 × 2 window, stride 2, padding 0. ### To find All four convolution values, the ReLU output matrix and the final pooled output. ### Formula — which one and why For each input window, multiply entries at matching positions, then add all products and the bias once. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ X is the input, S is stride, u and v select a kernel entry, and b is bias. This formula uses zero-based array indices; the written solution labels positions from 1 for readability. ReLU is required because the question requests activation after convolution. It replaces a negative value by zero. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Max pooling selects the largest value in its current window V. $$ y_{\max}=\max V $$ ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. ### Step 1: determine the convolution shape For each axis: (3 − 2)/1 + 1 = 2. Therefore calculate a 2 × 2 output matrix. ### Step 2: top-left output, position (1,1) The window is $\begin{bmatrix}1 & 2\\0 & 1\end{bmatrix}$ . Multiply matching cells: 1 × 1 = 1; 2 × (−1) = −2; 0 × 0 = 0; 1 × 1 = 1. Add the products and bias: 1 − 2 + 0 + 1 + 0 = **0**. ### Step 3: top-right output, position (1,2) Move the window one column to the right. The window is $\begin{bmatrix}2 & 1\\1 & 3\end{bmatrix}$ . Multiply matching cells: 2 × 1 = 2; 1 × (−1) = −1; 1 × 0 = 0; 3 × 1 = 3. Add: 2 − 1 + 0 + 3 + 0 = **4**. ### Step 4: bottom-left output, position (2,1) The window is $\begin{bmatrix}0 & 1\\2 & 1\end{bmatrix}$ . Multiply matching cells: 0 × 1 = 0; 1 × (−1) = −1; 2 × 0 = 0; 1 × 1 = 1. Add: 0 − 1 + 0 + 1 + 0 = **0**. ### Step 5: bottom-right output, position (2,2) The window is $\begin{bmatrix}1 & 3\\1 & 0\end{bmatrix}$ . Multiply matching cells: 1 × 1 = 1; 3 × (−1) = −3; 1 × 0 = 0; 0 × 1 = 0. Add: 1 − 3 + 0 + 0 + 0 = **−2**. **Convolution map: $\begin{bmatrix}0 & 4\\0 & -2\end{bmatrix}$ .** ### Step 6: apply ReLU separately to every entry - max(0,0) = 0. - max(0,4) = 4. - max(0,0) = 0. - max(0,−2) = 0. **ReLU map: $\begin{bmatrix}0 & 4\\0 & 0\end{bmatrix}$ .** ### Step 7: max pooling The 2 × 2 pooling window covers all four activated entries. Compare 0, 4, 0 and 0. Their maximum is **4**. **Final answer:** convolution $\begin{bmatrix}0 & 4\\0 & -2\end{bmatrix}$ → ReLU $\begin{bmatrix}0 & 4\\0 & 0\end{bmatrix}$ → pooled output ** $\begin{bmatrix}4\end{bmatrix}$ **, a 1 × 1 map. **Yaad rakho:** Multiply-add se convolution; negative hatao ReLU; biggest uthao max pool. ### Q4 **Q4 optional (3 marks). What is overfitting? Explain how dropout and early stopping help improve generalization. State which dataset should select the stopping point.** **Answer — step by step.** ### 1. Definition and identification Overfitting occurs when a model learns training-specific details or noise that do not generalize to new data. It commonly shows low training error together with substantially higher validation error. Generalization means performing well on new examples drawn from the task's data distribution. ### 2. Dropout During training, dropout randomly sets selected activations to zero. The active subnetwork changes across training passes, discouraging reliance on a fixed combination of hidden units. Under inverted dropout, retained activations are divided by the keep probability to preserve their expected scale. During ordinary inference, the masking is disabled and all units are used without that training-time rescaling. Dropout is a regularizer; it can reduce overfitting, but excessive dropout can cause underfitting. ### 3. Early stopping Train on the training set and monitor performance on a separate **validation set**. Save the model whenever validation loss improves. Stop after the chosen number of consecutive non-improving evaluations, called patience, then restore the saved best model. For example, validation losses 0.8, 0.6, 0.5, 0.53 and 0.55 have their minimum at epoch 3. With patience 2, stop after epoch 5 and restore epoch 3. The stopping epoch and saved-model epoch are different. ![Early stopping retains the model with the best validation performance, not necessarily the final trained model.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: Early stopping retains the model with the best validation performance, not necessarily the final trained model.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. Early stopping retains the model with the best validation performance, not necessarily the final trained model. Tap the diagram to enlarge. **Final answer:** dropout restricts dependence on particular training-time activations; early stopping limits training after validation performance stops improving. Choose the stopping point using validation data and reserve the test set for final evaluation. **Yaad rakho:** Training se seekho, validation se best model chuno, test se final result batao. --- ## MT2 · MT2 Mock B: full questions and answers Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c18-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/18/mt2-c18-s1 60-minute rehearsal; attempt 10 marks. Q1(a) and Q1(b) are compulsory. Answer any TWO of Q2–Q4. Q2 is mixed theory + numerical, Q3 is numerical and Q4 is theory. The 1 + 2 split within Q2 is a practice allocation. The specific topics are not predictions of the actual paper. ### Q1a **Q1a compulsory (2 marks). Explain the purpose of scaling retained activations in inverted dropout. For activations [1,2,3,4], keep probability 0.5 and mask [1,0,1,0], calculate the activations used during this training pass. A mask value of 1 means keep and 0 means drop.** **Answer — step by step.** ### Theory: why scale retained values? Randomly dropping units would reduce the average activation. Inverted dropout divides every retained activation by the keep probability. Averaged over the random masks, this preserves the original activation's expected value. A dropped activation is still zero. ### Given - Original activations a = [1,2,3,4]. - Keep probability q = 0.5, meaning a unit has a 50% chance of being kept. - This pass's supplied mask m = [1,0,1,0]. ### To find The four post-dropout activations used in this training pass. ### Formula — which one and why Multiply the original activation aᵢ by its mask mᵢ, then divide by keep probability q. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ ### Solution — step by step ### Step 1: first activation Mask 1 keeps the value. Multiply 1 × 1 = 1; divide 1 by 0.5 to get **2**. ### Step 2: second activation Mask 0 drops the value. Multiply 0 × 2 = 0; divide 0 by 0.5 to get **0**. ### Step 3: third activation Mask 1 keeps the value. Multiply 1 × 3 = 3; divide 3 by 0.5 to get **6**. ### Step 4: fourth activation Mask 0 drops the value. Multiply 0 × 4 = 0; divide 0 by 0.5 to get **0**. **Final answer: [2,0,6,0].** This is the training-pass result. In ordinary inference with inverted dropout, disable the mask and its scaling. **Yaad rakho:** Mask lagao; bache hue values ko keep probability se divide karo. ### Q1b **Q1b compulsory (2 marks). Explain a ResNet shortcut. Why must the two tensors being added have matching shapes, and what can be done if their shapes differ?** **Answer — step by step.** ### 1. Residual addition A residual block combines a learned transformation F(x) with a shortcut carrying the input x: $$ H(x)=F(x)+x $$ F(x) is the output of the learned residual branch. Adding x provides a direct path for information and gradients, helping optimize deep networks. If the branch produces zero, the addition stage returns x. A post-addition activation, if specified by the architecture, is applied afterward. ### 2. Why matching shapes are needed The addition is entry by entry. Each element in F(x) needs a corresponding element in x, so height, width and channel count must match. For example, a 14 × 14 × 32 tensor cannot be directly added to a 14 × 14 × 64 tensor in a standard residual shortcut. ### 3. How a projection helps A learned projection, commonly a 1 × 1 convolution with an appropriate number of filters and stride, maps the shortcut to the required shape. Then add the projected shortcut and the residual branch. This is addition, not channel concatenation. **Final answer:** a shortcut adds input information to the learned residual branch. Use an identity shortcut when shapes match and a suitable projection when they do not. **Yaad rakho:** Same-size entries jodte hain; mismatch ho toh shortcut ka size pehle match karo. ### Q2 **Q2 optional (3 marks). (a) Explain why Adam uses bias correction for its initially zero moment estimates. (1 mark) (b) Adam starts with θ₀ = 1 and m₀ = v₀ = 0. Use learning rate η = 0.1, β₁ = 0.9 and β₂ = 0.99. The supplied gradients are g₁ = 2 and g₂ = −1. Perform two bias-corrected updates. Show the raw first and second moments, corrected moments and updated parameter after each iteration. Set ε = 0 only for this nonzero-denominator hand calculation. (2 marks)** **Answer — step by step.** ### Part (a): why Adam corrects the moments Adam initializes both moving averages to zero. At early iterations, too little gradient history has accumulated, so the raw estimates are biased toward zero. Dividing mₜ by 1 − β₁ᵗ and vₜ by 1 − β₂ᵗ corrects this initialization bias before the parameter update. The power uses the current iteration number t. As t increases, these denominators approach 1. ### Part (b): numerical solution ### Given - Starting parameter θ₀ = 1. - Starting first moment m₀ = 0 and second moment v₀ = 0. - Learning rate η = 0.1. - Decay factors β₁ = 0.9 and β₂ = 0.99. - Supplied gradients, in order: 2, then −1. - ε = 0 for this exercise; both denominators will be nonzero. ### To find For iterations 1 and 2, calculate m, v, their bias-corrected values and the new parameter θ. ### Formula — which one and why Adam keeps two moving averages: m stores signed gradients and v stores squared gradients. Zero initialization makes these averages initially too small, so divide by the correction factors for the current iteration t. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\quad\hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ The hat marks a corrected value. Carry the **raw** m and v into the next moving-average recurrence, not the corrected values. Keep full calculator precision until the final answer. ### Solution — step by step ### Step 1: iteration 1, raw first moment The current gradient is 2. Retain 90% of the old first moment and add 10% of this gradient. $$ m_1=0.9(0)+0.1(2)=0.2 $$ ### Step 2: iteration 1, raw second moment Square the gradient first: 2² = 4. Retain 99% of the old second moment and add 1% of that square. $$ v_1=0.99(0)+0.01(2)^2=0.04 $$ ### Step 3: iteration 1, bias correction Use t = 1. The correction denominators are 1 − 0.9 = 0.1 and 1 − 0.99 = 0.01. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.04}{1-0.99^{1}}=4 $$ ### Step 4: iteration 1, parameter update The square root of the corrected second moment is √4 = 2. The step is 0.1 × 2 ÷ 2 = 0.1. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ ### Step 5: iteration 2, raw first moment Use supplied gradient −1, previous raw m = 0.2 and previous raw v = 0.04. $$ m_2=0.9(0.2)+0.1(-1)=0.08 $$ The two contributions are 0.18 and −0.1; their sum is 0.08. ### Step 6: iteration 2, raw second moment The square of −1 is +1. $$ v_2=0.99(0.04)+0.01(-1)^2=0.0496 $$ The two contributions are 0.0396 and 0.01; their sum is 0.0496. ### Step 7: iteration 2, bias correction Use t = 2. The correction denominators are 1 − 0.9² = 0.19 and 1 − 0.99² = 0.0199. $$ \widehat m_2=\frac{0.08}{1-0.9^{2}}=0.421052632 $$ $$ \widehat v_2=\frac{0.0496}{1-0.99^{2}}=2.492462312 $$ ### Step 8: iteration 2, parameter update Start from the updated parameter 0.9. Take the square root of the corrected second moment, divide the corrected first moment by it, multiply by 0.1 and subtract. $$ \theta_2=0.9-\frac{0.1\times(0.421052632)}{\sqrt{2.492462312}}=0.873330057 $$ The displayed decimals are rounded; evaluate using the unrounded fractions when possible. **Final answer:** θ₁ = **0.9** and θ₂ ≈ **0.873330057**. The second supplied gradient is negative, but the first-moment estimate is still positive because it retains part of the previous positive gradient. **Yaad rakho:** Do raw memories banao → current t se hats nikalo → update; next round mein raw memories le jao. ### Q3 **Q3 optional (3 marks). A 16 × 16 × 64 feature map is processed by 32 filters of size 3 × 3 using stride 1 and same padding. Compare the direct convolution's MAC count with a bottleneck that first applies eight 1 × 1 filters, then the same 32-filter 3 × 3 convolution. Both stages keep spatial size 16 × 16. Find both costs, the absolute saving and percentage saving. Count one multiply–accumulate as one MAC and omit bias additions.** **Answer — step by step.** ### Given - Input map: 16 × 16 with 64 channels. - Direct layer: 32 filters, each 3 × 3, receiving 64 channels. - Bottleneck stage 1: eight 1 × 1 filters, receiving 64 channels. - Bottleneck stage 2: 32 filters, each 3 × 3, now receiving only 8 channels. - Every layer has output height and width 16; ignore bias additions. ### To find Direct MACs, both bottleneck-stage MACs, their total, absolute saving and saving percentage. ### Formula — which one and why Each output entry uses K²Cᵢₙ weight–input pairs. There are HₒᵤₜWₒᵤₜCₒᵤₜ output entries. Multiply these counts: $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ The bottleneck has two layers, so calculate each layer's work and add them. Saving is direct work minus reduced work, expressed as a fraction of direct work. $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ ### Solution — step by step ### Step 1: direct convolution Each output entry uses 3 × 3 × 64 = **576** weight–input pairs. There are 16 × 16 × 32 = **8,192** output entries. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 bottleneck reduction Each reduction output uses 1 × 1 × 64 = **64** weight–input pairs. There are 16 × 16 × 8 = **2,048** reduction outputs. $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ The new map is **16 × 16 × 8**. This changes the next layer's input channels to 8. ### Step 3: following 3 × 3 convolution Each output now uses 3 × 3 × 8 = **72** weight–input pairs, instead of 576. The number of outputs remains 16 × 16 × 32 = **8,192**. $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: total bottleneck cost Include both stages; do not compare the direct layer against only the second stage. $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: absolute saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ ### Step 6: percentage saving Divide the saving by the original direct cost, then multiply by 100. $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct **4,718,592 MACs**; bottleneck **720,896 MACs**; saving **3,997,696 MACs**, approximately **84.72%**. Both routes end in a 16 × 16 × 32 output, though they are different learned transformations. **Yaad rakho:** 1 × 1 se channels kam; expensive 3 × 3 ka kaam kam. Dono stages ka cost jodna hai. ### Q4 **Q4 optional (3 marks). Explain transfer learning for image classification with limited labelled data. Distinguish feature extraction from fine-tuning and describe a suitable training procedure.** **Answer — step by step.** ### 1. Definition Transfer learning reuses knowledge learned on a source task to help solve a target task. For image classification, a pretrained CNN can provide visual features that a new classifier uses for the target classes. This can reduce the amount of target data and training needed, provided the source features are useful for the target task. ### 2. Feature extraction and fine-tuning | Method | Which parameters learn? | Main effect | | --- | --- | --- | | Feature extraction | Keep pretrained backbone weights fixed; train a new classification head | Learn to classify using fixed pretrained features | | Fine-tuning | Update selected pretrained layers as well as the new head | Adapt representations to the target task | Freezing a layer stops its parameter updates; it does not remove its stored weights. Fine-tuning typically exposes more trainable parameters and therefore needs particular care with small datasets. ![Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update. Tap the diagram to enlarge. ### 3. Training procedure 1. Create separate training, validation and test splits for the target problem. 2. Choose a suitable pretrained CNN and apply its required input size and preprocessing. 3. Replace the original classifier with a new head matching the target classes. 4. Freeze the backbone and train the new head on the training set. 5. If validation results justify it, unfreeze selected later layers and continue with an appropriately smaller learning rate. 6. Select the best checkpoint using validation performance; evaluate the final model on the held-out test set. ### 4. Concrete example For a small dataset containing cats and dogs, reuse a pretrained image backbone and replace its original classifier with a binary-classification head. First learn the new head with the backbone frozen. Later, selected high-level layers may be fine-tuned if this improves validation performance. **Final answer:** feature extraction learns a new classifier over fixed features; fine-tuning also adapts selected pretrained features. The correct choice depends on target data quantity, similarity to the source data and validation results. **Yaad rakho:** Freeze mein purane features fixed; fine-tune mein selected purane features bhi seekhte hain. --- ## MT2 · The one page method map Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c19-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/19/mt2-c19-s0 | Question clue | First action | Core formula or check | | --- | --- | --- | | Neuron output | Multiply each input by its matching weight | $z=\sum_iw_ix_i+b$ , then activate | | Network prediction | Finish one whole layer before the next | $h=f(Wx+b)$ | | Backpropagation | State loss then prediction-minus-target | delta times sending activation | | Optimizer update | Identify current gradient and stored state | old parameter minus update | | CNN dimensions | Write N K S P and channels | $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ | | CNN parameters | Count one filter then all filters | $(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}$ | | Pooling | Draw every valid window | Maximum, or mean of the window | | Dense parameters | Count connections and receiving biases | $(N_{\mathrm{in}}+1)N_{\mathrm{out}}$ | --- ## MT2 · All formulas: meanings, use and substitution Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c19-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/19/mt2-c19-s1 These are the formulas used by this course. Each entry names its symbols and the type of calculation it serves. Numerical practice in the teaching chapters shows the longer working. Follow any different convention explicitly stated in the examination. ### Formula 1: Mean, variance and standard deviation $$ \mu=\frac{1}{n}\sum_i x_i,\quad v=\frac{1}{n}\sum_i(x_i-\mu)^2,\quad s=\sqrt{v} $$ **Symbols:** n is the count; mu is the mean; v is the population variance; s is the standard deviation. **Use:** Use when calculating a population mean/variance or standardization. Sample-variance estimates may use n−1 when explicitly requested. **Substitution example:** For [2,4]: mean = 3; deviations = −1,1; squares = 1,1; variance = 1; standard deviation = 1. ### Formula 2: Standardization $$ x_{scaled}=\frac{x-\mu_{train}}{s_{train}} $$ **Symbols:** Use the mean and standard deviation estimated from training data. **Use:** Use to express a value relative to the training distribution. The standard deviation must be nonzero. **Substitution example:** For x = 7, mean = 5, standard deviation = 2: (7−5)/2 = 1. ### Formula 3: Power, constant and sum derivatives $$ \frac{d}{dx}(cx^n)=cnx^{n-1},\quad\frac{d}{dx}c=0,\quad (f+g)\prime=f\prime+g\prime $$ **Symbols:** c is a constant, x is the changing variable and n is the power. **Use:** Use to differentiate the loss supplied in a gradient-descent question. **Substitution example:** For 0.7x³: derivative = 0.7 × 3 × x² = 2.1x². ### Formula 4: Chain rule $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ **Symbols:** z is the intermediate value linking w to L. **Use:** Multiply derivatives along a path; add contributions from distinct paths that meet. **Substitution example:** For z = 3w+1 and L = z² at w = 1: z = 4; derivative = (2×4)×3 = 24. ### Formula 5: Dot product and a neuron $$ z=\sum_i w_i x_i+b,\qquad a=f(z) $$ **Symbols:** x is the input; w is its matching weight; b is one bias; z is the weighted sum; a is the activated output. **Use:** Multiply each matching pair, add the products, add bias once, then activate. **Substitution example:** x = [2,1], w = [0.5,−0.2], b = 0.1: z = 1−0.2+0.1 = 0.9. ### Formula 6: Matrix multiplication $$ (AB)_{ij}=\sum_k A_{ik}B_{kj} $$ **Symbols:** i selects a row of A; j selects a column of B; k runs along the matching entries. **Use:** Use row-times-column sums. A with shape m×n and B with n×p produces m×p. **Substitution example:** Row [1,2] times column [3,4] gives 1×3+2×4 = 11. ### Formula 7: Layer forward pass $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ **Symbols:** l labels the layer; the previous layer activations become inputs to the next layer. **Use:** For receiving-neuron rows, compute each row dot product and add that receiving neuron’s bias. Finish the whole layer before moving on. ### Formula 8: Sigmoid and its derivative $$ \sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ **Symbols:** e is approximately 2.71828; sigma is the output. **Use:** Use for a sigmoid forward pass and for its local backpropagation derivative. **Substitution example:** At z = 0: output = 1/(1+1) = 0.5; derivative = 0.5×0.5 = 0.25. ### Formula 9: Tanh and its derivative $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1},\quad \tanh\prime(z)=1-\tanh^2(z) $$ **Symbols:** Tanh gives an output between −1 and 1. **Use:** Use for a stated tanh activation, commonly an RNN hidden state or recurrent candidate. **Substitution example:** At z = 0: numerator = 0, denominator = 2; output = 0; derivative = 1. ### Formula 10: ReLU and its derivative $$ f(z)=\max(0,z),\quad f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ **Symbols:** max means choose the larger value. **Use:** Use for ReLU. At zero the derivative is mathematically undefined; use the convention stated by the question or implementation. **Substitution example:** For −2, output = 0 and derivative = 0. For 3, output = 3 and derivative = 1. ### Formula 11: Leaky ReLU $$ f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases} $$ **Symbols:** a is the small negative-side slope, commonly 0.01 unless specified. **Use:** For negative z multiply by a; the output remains negative. **Substitution example:** For z = −2 and a = 0.01: output = −0.02; derivative on this side is 0.01. ### Formula 12: Stable softmax $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_j z_j $$ **Symbols:** z values are class scores; p values are probabilities. **Use:** Subtract the same maximum from every score, exponentiate, add, divide each by that total. **Substitution example:** For [0,0]: exponentials = [1,1]; total = 2; probabilities = [0.5,0.5]. ### Formula 13: Squared and absolute losses $$ \mathrm{SSE}=\sum_i(o_i-y_i)^2 $$ $$ L_{\mathrm{half}}=\frac{\mathrm{SSE}}2 $$ $$ L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n $$ $$ L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i| $$ **Symbols:** o is prediction; y is target; n is the number of averaged terms. **Use:** Read whether the question uses a half sum, a mean, or an unhalved sum. The scaling changes gradients. **Substitution example:** For errors [−1,2]: half sum = (1+4)/2 = 2.5; MSE = 2.5; MAE = (1+2)/2 = 1.5. The first two coincide only because n = 2. ### Formula 14: Binary cross entropy $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Symbols:** y is 0 or 1; p is the predicted probability for class 1; ln is the natural logarithm. **Use:** Use for a sigmoid binary prediction. If y = 1 use −ln(p); if y = 0 use −ln(1−p). **Substitution example:** For y = 1 and p = 0.8: loss = 0.223144. ### Formula 15: Categorical cross entropy $$ H(p,q)=-\sum_i p_i\ln q_i,\qquad\log_2q=\frac{\ln q}{\ln2} $$ For a soft target distribution, retain every nonzero target-weighted term. Natural logs give nats; base-2 logs give bits. $$ L=-\sum_i y_i\ln p_i=-\ln p_{true} $$ **Symbols:** The final equality assumes one-hot labels, with exactly one y equal to 1. **Use:** Use for mutually exclusive multiclass labels. Select the probability of the true class. **Substitution example:** True class 2 and probabilities [0.1,0.8,0.1]: loss = −ln(0.8) = 0.223144. ### Formula 16: Output delta for half squared loss $$ \delta^{out}=(o-y)f\prime(z) $$ **Symbols:** Delta is the loss derivative with respect to the preactivation z. **Use:** Use with half squared loss. For unhalved one-output MSE multiply by 2; for an MSE averaged over m outputs multiply the half-sum delta by 2/m. **Substitution example:** Sigmoid output 0.5 and target 1: delta = (−0.5)×0.25 = −0.125. ### Formula 17: Sigmoid plus BCE; softmax plus CCE $$ \delta^{out}=p-y $$ **Symbols:** p and y can be vectors for softmax. **Use:** This is the combined derivative with respect to the logits. Do not multiply by the activation derivative again. **Substitution example:** Binary p = 0.8, y = 1: delta = −0.2. ### Formula 18: Hidden delta, weight gradient and bias gradient $$ \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out},\quad\frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\quad\frac{\partial L}{\partial b_j}=\delta_j $$ **Symbols:** a\_i is the activation sending a signal along the weight; delta\_j belongs to the receiving neuron. **Use:** Use OLD downstream weights throughout one backward pass. Add all outgoing-path contributions. **Substitution example:** If receiving delta = −0.2 and sending activation = 3: weight gradient = −0.6; bias gradient = −0.2. ### Formula 19: Gradient descent $$ \theta_{new}=\theta-\eta g,\qquad g=\frac{\partial J}{\partial\theta} $$ **Symbols:** theta is the parameter; eta is learning rate; g is the current gradient. **Use:** Recalculate g at the current parameter unless the question supplies gradients directly. **Substitution example:** theta = 1, eta = 0.1, g = −2 gives 1−0.1(−2) = 1.2. ### Formula 20: Batch and mini-batch gradient $$ g_{batch}=\frac1B\sum_{i=1}^{B}g_i,\quad U_{epoch}=\left\lceil\frac NB\right\rceil $$ **Symbols:** B is batch size; N is sample count; U is updates per epoch when the final incomplete batch is retained. **Use:** SGD uses B = 1. Full batch uses B = N. Average only when the stated objective is a mean. **Substitution example:** N = 105, B = 20: six updates, with five samples in the last batch. ### Formula 21: Momentum: signed velocity $$ v_t=\beta v_{t-1}-\eta g_t,\qquad\theta_t=\theta_{t-1}+v_t $$ **Symbols:** v stores signed movement; beta retains part of old movement. **Use:** Use this convention consistently. Other textbooks may define an unsigned gradient accumulator and subtract it instead. **Substitution example:** beta = 0.8, old v = −0.2, eta = 0.1, g = 1.8 gives v = −0.34. ### Formula 22: Nesterov: look-ahead gradient $$ \theta_{look}=\theta+\beta v,\quad v_{new}=\beta v-\eta g(\theta_{look}),\quad\theta_{new}=\theta+v_{new} $$ **Symbols:** The look-ahead point is temporary; the update is added to the original theta. **Use:** Evaluate the gradient at the look-ahead point, not at the old theta. **Substitution example:** theta = 1.8, beta = 0.8 and v = −0.2 gives look-ahead = 1.64. ### Formula 23: AdaGrad $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Symbols:** G is accumulated squared gradient; epsilon prevents division by zero. **Use:** Accumulate every squared gradient, separately for each parameter. Use the epsilon placement stated by the problem. **Substitution example:** Old G = 4 and g = −3 gives G = 13. ### Formula 24: RMSProp $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Symbols:** rho is the decay; s is a decaying mean of squared gradients. **Use:** Calculate the new square memory first, then update the parameter. **Substitution example:** rho = 0.9, old s = 4, g = −3 gives s = 3.6+0.9 = 4.5. ### Formula 25: Adam: both memories $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ **Symbols:** m remembers signed gradients; v remembers squared gradients. **Use:** Start with the initial states specified. Common defaults are beta1 = 0.9 and beta2 = 0.999. ### Formula 26: Adam: correction and update $$ \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ **Symbols:** t counts updates starting at 1, not 0. **Use:** Correct both zero-initialized memories before the update; do not multiply by the gradient a second time. **Substitution example:** At t = 1, m = 0.2 and beta1 = 0.9: corrected m = 0.2/0.1 = 2. ### Formula 27: L1 penalty and update $$ J=L+\lambda\sum_j|w_j|,\quad w_{new}=w-\eta\left[g_{data}+\lambda\operatorname{sign}(w)\right] $$ **Symbols:** lambda is regularization strength; sign(w) is +1 for positive w and −1 for negative w. **Use:** At zero, L1 has a subgradient rather than a unique derivative; use the specified convention. Include only parameters named in the penalty. **Substitution example:** w = −2, lambda = 0.1, data gradient = 0 gives penalty gradient = −0.1. ### Formula 28: L2 penalty and update with a half $$ J=L+\frac\lambda2\sum_jw_j^2,\quad w_{new}=w-\eta(g_{data}+\lambda w) $$ **Symbols:** The half cancels the 2 produced by differentiating a square. **Use:** If the objective uses lambda times the sum of squares WITHOUT a half, use 2 lambda w instead. **Substitution example:** w = 2, lambda = 0.1, data gradient = 0.3 gives total gradient = 0.5. ### Formula 29: Inverted dropout $$ q=1-p,\quad a_{train}=\frac{m\odot a}{q},\quad a_{test}=a $$ **Symbols:** p is drop probability; q is keep probability; mask m contains zeros and ones; the circled dot means elementwise multiplication. **Use:** During training, zero dropped entries and divide kept entries by q. Disable dropout at inference. **Substitution example:** a = 3, q = 0.5, mask = 1 gives 3/0.5 = 6. ### Formula 30: Convolution output dimensions $$ N_{out}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Symbols:** N is input height or width; P is padding on each side; K is kernel size; S is stride; floor rounds down. **Use:** Apply separately to height and width. Output channels equal the number of filters. This formula assumes dilation 1. **Substitution example:** N = 7, P = 1, K = 3, S = 2 gives floor(6/2)+1 = 4. ### Formula 31: Convolution value $$ Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o $$ **Symbols:** c runs over input channels; u,v run over kernel positions; o selects one filter. **Use:** This is the usual deep-learning cross-correlation operation. Use zeros outside the padded image. Multiply matching cells, add all channels, then add the filter bias once. ### Formula 32: Max, min and average pooling Min pooling keeps the smallest entry in each window. For a four-entry window: $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{max}=\max_{(i,j)\in W}x_{ij},\quad y_{avg}=\frac1{|W|}\sum_{(i,j)\in W}x_{ij} $$ **Symbols:** W is one pooling window; its size is the number of included cells. **Use:** Move the window by the stated stride; process channels separately. For padding, follow the stated average-pooling denominator convention. **Substitution example:** For [1,4,2,3]: maximum = 4; average = 10/4 = 2.5. ### Formula 33: Convolution parameter count $$ P_{conv}=(K_hK_wC_{in}+1)C_{out} $$ **Symbols:** Kernel height and width times input channels gives weights per filter; +1 is one bias; Cout is filter count. **Use:** Use for a fully connected, ungrouped convolution with bias. Output height and width do not multiply the parameter count. **Substitution example:** 3×3 kernel, 3 input channels, 2 filters: (27+1)×2 = 56. ### Formula 34: Flatten and dense parameter count $$ N_{flat}=HWC,\qquad P_{dense}=(N_{in}+1)N_{out} $$ **Symbols:** H,W,C are activation dimensions; Nin and Nout are dense-layer input and output counts. **Use:** Flatten has no trainable parameters. Dense biases belong to receiving neurons. **Substitution example:** 2×2×2 flattens to 8; dense 3 then has (8+1)×3 = 27 parameters. ### Formula 35: Convolution multiply-accumulate count $$ MAC=H_{out}W_{out}K_hK_wC_{in}C_{out} $$ **Symbols:** Each output value uses kernel area times input-channel count multiplications. **Use:** Used for Inception bottleneck comparisons. This counts one multiply-accumulate as one MAC and ignores bias additions. Some FLOP conventions count multiply and add separately. ### Formula 36: Concatenation and residual addition $$ C_{concat}=\sum_b C_b,\qquad y_{res}=F(x)+x $$ **Symbols:** b labels branches with matching spatial dimensions. **Use:** Inception concatenates channels; residual addition requires matching full shapes. Use a projection when needed. **Substitution example:** Channels 16+32+8+8 give 64. Adding two 14×14×64 tensors keeps 14×14×64. ### Formula 37: Vanilla RNN $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad o_t=W_{hy}h_t+b_y $$ **Symbols:** x\_t is current input; h\_previous is previous state; the same matrices are reused at every time step. **Use:** Calculate the two matrix-vector products, add bias, then tanh. Apply the stated output activation to o\_t when required. ### Formula 46: Batch normalization and learned scale/shift Use during training to normalize intermediate activations before a learned scale and shift. m is the number of values; μ\_B and σ²\_B are the batch mean and population variance; γ and β are learned. Epsilon is inside the square root. Ordinary inference uses stored running statistics. The linked full lesson calculates every stage for [2,4]. $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ ### Formula 47: Global average pooling Take a separate average for each full spatial feature map. H and W count rows and columns; c selects one of C channels; a is an activation value. Output has C values. The linked lesson computes two channel averages and the following dense-layer parameters. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ ### Formula 48: Top-k error percentage N is the total number of evaluated examples. C\_k counts those whose true class is among the k highest-scoring predictions. Count failures, divide by total, then convert to percent. The worked lesson contrasts top-1 and top-5 using the same 100 images. $$ E_k=\frac{N-C_k}{N}\times100\% $$ ## Expanded formula guide and worked examples ## Mean, variance and standard deviation The mean is a measure of location; variance and standard deviation quantify dispersion. Distinguish population and sample conventions before calculating these statistics. $$ \mu=\frac{1}{n}\sum_i x_i $$ $$ v=\frac{1}{n}\sum_i(x_i-\mu)^2 $$ $$ s=\sqrt{v} $$ ## Mean, variance and standard deviation from ordinary arithmetic **Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i,\qquad \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2,\qquad \sigma=\sqrt{\sigma^2} $$ These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable. **Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].** ### Given Observations = [3,3,7,7]; N = 4. Use population variance (divide by N). ### To find Find the population mean μ, variance σ² and standard deviation σ. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3. ### Population mean and spread: [1, 2, 3] **Question. For the population [1, 2, 3], find mean, population variance and standard deviation.** ### Given Values = [1, 2, 3]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 1+2+3=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (1-2)=-1 $$ $$ (-1)^2=1 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (3-2)=1 $$ $$ (1)^2=1 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{1+0+1}{3}\approx0.666666667 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.666666667}\approx0.816496581 $$ **Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581. ### Population mean and spread: [2, 2, 2] **Question. For the population [2, 2, 2], find mean, population variance and standard deviation.** ### Given Values = [2, 2, 2]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 2+2+2=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{0+0+0}{3}\approx0.000000000 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.000000000}\approx0.000000000 $$ **Final answer:** μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation. ## Standardization Standardization has a meaning before it has a formula: measure how far an input is from its training mean, then express that distance in standard-deviation units. The worked question below calculates the mean and standard deviation first, so neither number appears without explanation. $$ x_{scaled}=\frac{x-\mu_{train}}{s_{train}} $$ ## Standardization: what the answer 1 actually means **Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units. $$ z=\frac{x-\mu_{\mathrm{train}}}{\sigma_{\mathrm{train}}},\qquad \sigma_{\mathrm{train}}>0 $$ Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1]. **Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.** ### Given Training observations = [3,3,7,7]. New observation = 9. Estimate the population standard deviation from the training values. ### To find Find the training mean and standard deviation; standardize all four training values and the new observation. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. **Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units. x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Step 8: standardize each original training observation.** $$ \frac{3-5}{2}=-1 $$ $$ \frac{7-5}{2}=1 $$ The repeated observations give [-1, −1, 1, 1]. **Step 9: subtract the training mean from the new observation.** $$ 9-5=4 $$ **Step 10: divide by the training standard deviation.** $$ z_{\mathrm{new}}=\frac42=2 $$ **Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy. ### Standardization with x = 3 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.** ### Given x = 3; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=3-5=-2 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{-2}{2}=-1 $$ **Step 3: interpret sign and size.** The value is 1 standard-deviation unit(s) below the training mean. **Final answer:** z = -1. Use the same training statistics when transforming a new test input. ### Standardization with x = 5 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.** ### Given x = 5; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=5-5=0 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{0}{2}=0 $$ **Step 3: interpret sign and size.** The value is 0 standard-deviation unit(s) away from the training mean. **Final answer:** z = 0. Use the same training statistics when transforming a new test input. ### Standardization with x = 9 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.** ### Given x = 9; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=9-5=4 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{4}{2}=2 $$ **Step 3: interpret sign and size.** The value is 2 standard-deviation unit(s) above the training mean. **Final answer:** z = 2. Use the same training statistics when transforming a new test input. ## Power, constant and sum derivatives Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization. $$ \frac{d}{dx}(cx^n)=cnx^{n-1} $$ $$ \frac{d}{dx}c=0 $$ $$ (f+g)\prime=f\prime+g\prime $$ ## Derivatives without a calculus course **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector. $$ \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix} $$ The rules needed for the supplied neural-network numericals are: | Function or operation | Derivative | | --- | --- | | Constant c | 0 | | xⁿ | nxⁿ⁻¹ | | c f(x) | c f′(x) | | f(x) + g(x) | f′(x) + g′(x) | | f(x)g(x) | f′(x)g(x) + f(x)g′(x) | | eˣ | eˣ | | ln x, x > 0 | 1/x | | f(g(x)) | f′(g(x))g′(x) | ### A parameter derivative is not an input derivative For z = wx + b, the variable named in the denominator determines which derivative is required. $$ \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1 $$ At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities. ### Gradient direction For a small parameter displacement Δθ, first-order Taylor expansion gives: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large. ### Differentiate before substituting: 3w^2+5 **Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.** ### Given J(w) = 3w^2+5; w = 2. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(2)=3(2)^2+5=17 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=3(2)w^{2-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(2)=3(2)(2)^{1}=12 $$ **Final answer:** J = 17; derivative = 12. ### Differentiate before substituting: 0.7w^3+0 **Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.** ### Given J(w) = 0.7w^3+0; w = 1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(1)=0.7(1)^3+0=0.7 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=0.7(3)w^{3-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(1)=0.7(3)(1)^{2}=2.1 $$ **Final answer:** J = 0.7; derivative = 2.1. ### Differentiate before substituting: 2w^1+4 **Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.** ### Given J(w) = 2w^1+4; w = -1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(-1)=2(-1)^1+4=2 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=2(1)w^{1-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(-1)=2(1)(-1)^{0}=2 $$ **Final answer:** J = 2; derivative = 2. ## Chain rule The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ ## The chain rule is multiplying linked effects **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions. ### First understand why the rates multiply Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency. The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w. **Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add. ### Chain-rule example 1: two linear links **Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.** ### Given w = 1; u = 3w+1; v = 2u. ### To find The two function values and the rate dv/dw. ### Formula — which one and why v depends on w through u. Both links must be included, so multiply the two local derivatives. du/dw is the slope of 3w+1. dv/du is the slope of 2u. $$ \frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw} $$ **Yaad rakhne ka cue:** Andar ka slope × bahar ka slope. ### Solution — step by step **Step 1: calculate the intermediate value.** $$ u=3(1)+1=4 $$ **Step 2: calculate the final value.** $$ v=2(4)=8 $$ **Step 3: differentiate the first link.** $$ \frac{du}{dw}=3 $$ **Step 4: differentiate the second link.** $$ \frac{dv}{du}=2 $$ **Step 5: multiply along the path.** $$ \frac{dv}{dw}=2(3)=6 $$ **Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different. ### 2. One output: calculate both parameter gradients **Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1. ### Given Input x = 2; target y = 4. Old weight w = 1; old bias b = 0. Linear prediction z = wx+b; loss L = ½(z−y)². Learning rate η = 0.1; update both parameters once. ### To find Find the loss gradient for w and for b, then w\_new and b\_new. Finally evaluate the changed prediction and loss. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ![The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.](https://deep-learning-study-room.vercel.app/assets/figures/chain_linear_revision.svg) Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1). The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient. Tap the diagram to enlarge. **Step 1 — evaluate the original model.** Use the given parameters before applying any update. $$ z=wx+b=1(2)+0=2 $$ $$ e=z-y=2-4=-2 $$ $$ L=\frac12e^2=\frac12(-2)^2=2 $$ Forward pass: three distinct quantities Prediction z**2**1 × 2 + 0 → Error e**−2**Prediction 2 − target 4 → Half squared loss**2**½ × (−2)² The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide. **Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it. $$ \frac{\partial L}{\partial e}=\frac12(2e)=e $$ $$ \frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1 $$ $$ \delta_z=\frac{\partial L}{\partial z}=e(1)=-2 $$ The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection. **Step 3 — differentiate the neuron's affine expression.** When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1. $$ \frac{\partial z}{\partial w}=x=2 $$ $$ \frac{\partial z}{\partial b}=1 $$ **Step 4 — multiply along each parameter's path.** $$ g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4 $$ $$ g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2 $$ Backward pass: a separate derivative for each parameter Shared loss derivative δz**−2**Both parameter paths start here. Weight path: multiply by x = 2**−4**∂L/∂w = (−2) × 2 Bias path: multiply by 1**−2**∂L/∂b = (−2) × 1 Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate. **Step 5 — update both parameters simultaneously.** Gradient descent subtracts the learning rate times the corresponding gradient. $$ w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4 $$ $$ b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2 $$ One simultaneous parameter update Weight**1 → 1.4**Subtract 0.1 × (−4) Bias**0 → 0.2**Subtract 0.1 × (−2) Both updates use gradients from the same original forward pass. Re-evaluate the model afterward. Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure. **Step 6 — perform a new forward pass to evaluate the update.** $$ z_{\mathrm{new}}=1.4(2)+0.2=3 $$ $$ L_{\mathrm{new}}=\frac12(3-4)^2=0.5 $$ **Final answer:** weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3. ### 3. Two outputs: why backward contributions are added A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward. **Question.** A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed. ### Given h = wx+b; outputs o₁ = 2h and o₂ = −h. x = 2; w = 0.5; b = 0.1; targets = [1,0]. Loss = half the sum of the two squared output errors. Output multipliers 2 and −1 stay fixed. ### To find Find total loss and its derivatives with respect to h, w and b. ### Formula — which one and why **Chain rule with more than one path.** The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b. oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h} $$ $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h} $$ **Yaad rakhne ka cue:** Har raaste ka effect pehle multiply, junction par sab add. ### Solution — step by step ![The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.](https://deep-learning-study-room.vercel.app/assets/figures/chain_branch_revision.svg) Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5 The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias. Tap the diagram to enlarge. **Step 1 — forward values.** $$ h=0.5(2)+0.1=1.1 $$ $$ o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1 $$ $$ L_1=\frac12(2.2-1)^2=0.72 $$ $$ L_2=\frac12(-1.1-0)^2=0.605 $$ $$ L=L_1+L_2=0.72+0.605=1.325 $$ **Step 2 — output derivatives.** Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output. $$ \frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2 $$ $$ \frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1 $$ **Step 3 — calculate the contribution of each complete path.** The derivative of 2h with respect to h is 2; the derivative of −h is −1. $$ \frac{\partial L_1}{\partial h}=(1.2)(2)=2.4 $$ $$ \frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1 $$ **Step 4 — add the two contributions at the shared hidden value.** $$ \frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h} $$ $$ \frac{\partial L}{\partial h}=2.4+1.1=3.5 $$ Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path. **Step 5 — continue to the hidden weight and bias.** Since h = wx + b, its derivatives with respect to w and b are x and 1. $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7 $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5 $$ **Final answer:** L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation. ### Chain-rule example 4: a square after a linear link **Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.** ### Given w = 1; z = 3w+1; L = z². ### To find z, L and the derivative of L with respect to w. ### Formula — which one and why The square acts on z, not directly on w. Differentiate the square first and include the change of z with w. dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ $$ \frac{d(z^2)}{dz}=2z $$ **Yaad rakhne ka cue:** Square ka 2z; andar ka 3; dono multiply. ### Solution — step by step **Step 1: evaluate the inner expression.** $$ z=3(1)+1=4 $$ **Step 2: evaluate the loss expression.** $$ L=4^2=16 $$ **Step 3: differentiate the square and use z = 4.** $$ \frac{dL}{dz}=2z=2(4)=8 $$ **Step 4: differentiate the inner expression.** $$ \frac{dz}{dw}=3 $$ **Step 5: multiply, because the changes occur in sequence.** $$ \frac{dL}{dw}=8(3)=24 $$ **Final answer:** z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. ## Dot product and a neuron A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions. $$ z=\sum_i w_i x_i+b $$ $$ a=f(z) $$ ## Model answer: explain an artificial neuron ## Question **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example. ## Answer — step by step An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers. ![Follow each input through its weight, add one bias at the receiving neuron, then apply activation.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Follow each input through its weight, add one bias at the receiving neuron, then apply activation. Tap the diagram to enlarge. ### Formula and symbols $$ z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z) $$ | Part | Meaning | What it does | | --- | --- | --- | | x₁, x₂, … | Input features | Supply the information for this calculation | | w₁, w₂, … | Weights | Control the contribution of each matching input | | b | Bias | Shifts the weighted sum independently of the inputs | | z | Net input | The number before activation | | f | Activation rule | Transforms z, often nonlinearly | | a | Activated output | The number sent onward | ### Worked ReLU neuron example **Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU. ### To find Preactivation z and activated output a. ### Formula — which one and why Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total. z means before activation; a means after activation. $$ z=w_1x_1+w_2x_2+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Product, product, sum, bias, phir activation. ### Solution — step by step **Step 1: multiply the first pair.** $$ 0.5(2)=1 $$ **Step 2: multiply the second pair.** $$ (-0.2)(1)=-0.2 $$ **Step 3: add both contributions.** $$ 1+(-0.2)=0.8 $$ **Step 4: add the bias once.** $$ z=0.8+0.1=0.9 $$ **Step 5: apply ReLU.** $$ a=\max(0,0.9)=0.9 $$ **Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged. The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once. ## What one artificial neuron does **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output. ### What goes into a neuron? An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive. The **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**. **Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output. ### Mathematical model and notation $$ z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b $$ $$ a=f(z) $$ | Symbol | Meaning and role | | --- | --- | | xᵢ | Input component i; the neuron receives d such components | | wᵢ | Trainable weight multiplying that input component | | b | Trainable bias added once to this neuron's total | | z | Preactivation: the result before applying the activation function | | f | Chosen activation function, such as sigmoid or ReLU | | a | Activation: the neuron's output after applying f | The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs. ### Why both a bias and an activation are needed The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin. The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values. ![Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron. Tap the diagram to enlarge. ### Worked numerical **Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output. ### Given Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5]. Bias = 0.1; activation = sigmoid. ### To find Find the preactivation z, then the sigmoid output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. ### From one neuron to a layer With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron: $$ z=Wx+b,\qquad a=f(z) $$ Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together. ### Neuron example 2: ReLU output **Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2]. Weights = [3]; bias = 1. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(3)=6 $$ **Step 2: add the input contributions.** $$ s=(6)=6 $$ **Step 3: add the bias once.** $$ z=6+(1)=7 $$ **Step 4: apply the stated activation.** $$ a=\max(0,7)=7 $$ **Final answer:** z = 7; a = 7. ### Neuron example 3: ReLU output **Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.** ### Given Inputs = [2, 1]. Weights = [-1, 0.5]; bias = 0.2. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(-1)=-2 $$ **Step 2: multiply input 2 by its own weight.** $$ (1)(0.5)=0.5 $$ **Step 3: add the input contributions.** $$ s=(-2)+(0.5)=-1.5 $$ **Step 4: add the bias once.** $$ z=-1.5+(0.2)=-1.3 $$ **Step 5: apply the stated activation.** $$ a=\max(0,-1.3)=0 $$ **Final answer:** z = -1.3; a = 0. ### Neuron example 4: sigmoid output **Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.** ### Given Inputs = [1, 2]. Weights = [0.2, -0.1]; bias = 0. Activation = sigmoid. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (1)(0.2)=0.2 $$ **Step 2: multiply input 2 by its own weight.** $$ (2)(-0.1)=-0.2 $$ **Step 3: add the input contributions.** $$ s=(0.2)+(-0.2)=0 $$ **Step 4: add the bias once.** $$ z=0+(0)=0 $$ **Step 5: apply the stated activation.** $$ a=\frac1{1+e^{0}}=\frac1{1+1}=0.5 $$ **Final answer:** z = 0; a = 0.5. ## Drill 4 Dot product and bias ## Question **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU. ## Answer — step by step ### Given Inputs x = [2, −1, 0.5]. Weights w = [0.3, 0.4, −0.2]; bias b = 0.1. Activation: ReLU. ### To find Find the weighted sum z and ReLU output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero. $$ z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z) $$ Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output. ### Step 1: multiply the matching pairs $$ w_1x_1=0.3\times2=0.6 $$ $$ w_2x_2=0.4\times(-1)=-0.4 $$ $$ w_3x_3=(-0.2)\times0.5=-0.1 $$ ### Step 2: add all products and add bias once $$ z=0.6+(-0.4)+(-0.1)+0.1 $$ $$ z=0.2-0.1+0.1=0.2 $$ ### Step 3: apply the activation $$ a=\max(0,0.2)=0.2 $$ **Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive. ## Matrix multiplication Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias. $$ (AB)_{ij}=\sum_k A_{ik}B_{kj} $$ ## Matrix multiplication: every row has a job ### Row, column, vector and matrix — read these first A **row** runs horizontally, from left to right. A **column** runs vertically, from top to bottom. A **matrix** is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows. A **vector** is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors. ### Transpose: rows become columns The superscript T means **transpose**, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m. $$ [A^T]_{ij}=A_{ji} $$ The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1. **Yaad rakhne ka cue:** transpose = row ko column bana; number aur sign ko haath mat laga. ### Transpose example 1 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}2&1&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 1 row(s) and 3 column(s), so its size is 1 × 3. **Step 2: move original row 1 into new column 1.** Copy [2, 1, 3] down column 1, in the same order. **Step 3: write the transposed matrix.** $$ A^T=\begin{bmatrix}2\\1\\3\end{bmatrix} $$ **Step 4: state its new size.** The result has 3 row(s) and 1 column(s): 3 × 1. **Final answer:** Aᵀ has size 3 × 1. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 1 rows × 3 columns A: 1 rows × 3 columns | | | | | --- | --- | --- | | 2 | 1 | 3 | Read each colored row left to right. Aᵀ: 3 rows × 1 columns Aᵀ: 3 rows × 1 columns | | | --- | | 2 | | 1 | | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 2 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 3 column(s), so its size is 2 × 3. **Step 2: move original row 1 into new column 1.** Copy [1, 2, 3] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4, 5, 6] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix} $$ **Step 5: state its new size.** The result has 3 row(s) and 2 column(s): 3 × 2. **Final answer:** Aᵀ has size 3 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 3 columns A: 2 rows × 3 columns | | | | | --- | --- | --- | | 1 | 2 | 3 | | 4 | 5 | 6 | Read each colored row left to right. Aᵀ: 3 rows × 2 columns Aᵀ: 3 rows × 2 columns | | | | --- | --- | | 1 | 4 | | 2 | 5 | | 3 | 6 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 3 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&-2\\0&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 2 column(s), so its size is 2 × 2. **Step 2: move original row 1 into new column 1.** Copy [1, -2] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [0, 3] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix} $$ **Step 5: state its new size.** The result has 2 row(s) and 2 column(s): 2 × 2. **Final answer:** Aᵀ has size 2 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 2 columns A: 2 rows × 2 columns | | | | --- | --- | | 1 | −2 | | 0 | 3 | Read each colored row left to right. Aᵀ: 2 rows × 2 columns Aᵀ: 2 rows × 2 columns | | | | --- | --- | | 1 | 0 | | −2 | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 4 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}-1\\4\\2\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 3 row(s) and 1 column(s), so its size is 3 × 1. **Step 2: move original row 1 into new column 1.** Copy [-1] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4] down column 2, in the same order. **Step 4: move original row 3 into new column 3.** Copy [2] down column 3, in the same order. **Step 5: write the transposed matrix.** $$ A^T=\begin{bmatrix}-1&4&2\end{bmatrix} $$ **Step 6: state its new size.** The result has 1 row(s) and 3 column(s): 1 × 3. **Final answer:** Aᵀ has size 1 × 3. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 3 rows × 1 columns A: 3 rows × 1 columns | | | --- | | −1 | | 4 | | 2 | Read each colored row left to right. Aᵀ: 1 rows × 3 columns Aᵀ: 1 rows × 3 columns | | | | | --- | --- | --- | | −1 | 4 | 2 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Dot product: multiply matching entries, then add A **dot product** turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed. $$ w^Tx=w_1x_1+w_2x_2+w_3x_3 $$ This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products. ### Dot-product example 1 **Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.** ### Given Weights w = [2, 3]. Matching inputs x = [4, 1]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (2)(4)=8 $$ **Step 2: multiply matching pair 2.** $$ (3)(1)=3 $$ **Step 3: add the products.** $$ w^Tx=(8)+(3)=11 $$ **Final answer:** wᵀx = 11. ### Dot-product example 2 **Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.** ### Given Weights w = [1, -2]. Matching inputs x = [2, 3]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (1)(2)=2 $$ **Step 2: multiply matching pair 2.** $$ (-2)(3)=-6 $$ **Step 3: add the products.** $$ w^Tx=(2)+(-6)=-4 $$ **Final answer:** wᵀx = -4. ### Dot-product example 3 **Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.** ### Given Weights w = [0.5, 1, -1]. Matching inputs x = [4, 2, 3]. Both lists contain 3 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (0.5)(4)=2 $$ **Step 2: multiply matching pair 2.** $$ (1)(2)=2 $$ **Step 3: multiply matching pair 3.** $$ (-1)(3)=-3 $$ **Step 4: add the products.** $$ w^Tx=(2)+(2)+(-3)=1 $$ **Final answer:** wᵀx = 1. ### Matrix multiplication: each row makes one output To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B. $$ C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj} $$ For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once. **Yaad rakhne ka cue:** multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column. **Definition.** A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation. $$ z=Wx+b $$ For d input features and m receiving neurons, the dimensions are: $$ W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1} $$ The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product. ![Each row of the weight matrix calculates one receiving neuron's preactivation.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge. **Question.** Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b. ### Given Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ. ### To find Find both components of z = Wx+b. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. ### Solution — step by step **Step 1: verify compatible dimensions.** W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column. **Step 2: calculate the first receiving row products.** $$ 1(2)=2 $$ $$ (-2)(1)=-2 $$ **Step 3: sum row 1 and add its bias.** $$ z_1=2+(-2)+0.1=0.1 $$ **Step 4: calculate the second receiving row products.** $$ 0.5(2)=1 $$ $$ 3(1)=3 $$ **Step 5: sum row 2 and add its bias.** $$ z_2=1+3-0.2=3.8 $$ $$ z_1=1(2)+(-2)(1)+0.1=0.1 $$ Row 1 produces output component z1 1. Select weight row 1 1. Select weight row 1 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 2 | −2 | First × first; second × second. Sum the products**0** + Bias for this row**0.1** = z1**0.1** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z_2=0.5(2)+3(1)-0.2=3.8 $$ Row 2 produces output component z2 1. Select weight row 2 1. Select weight row 2 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 1 | 3 | First × first; second × second. Sum the products**4** + Bias for this row**−0.2** = z2**3.8** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z=\begin{bmatrix}0.1\\3.8\end{bmatrix} $$ An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W. ### Two output columns **Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.** ### Given $$ A=\begin{bmatrix}1&2\\3&4\end{bmatrix} $$ $$ B=\begin{bmatrix}2&0\\1&3\end{bmatrix} $$ ### To find The 2 × 2 matrix C = AB. ### Formula — which one and why Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication. Cᵢⱼ is the entry in output row i, column j. $$ C_{ij}=\sum_k A_{ik}B_{kj} $$ **Yaad rakhne ka cue:** Output ki position dekho: wahi row, wahi column. ### Solution — step by step **Step 1: check the dimensions.** (2 × 2)(2 × 2) is valid; output size = 2 × 2. **Step 2: row 1 with column 1.** $$ C_{11}=1(2)+2(1)=2+2=4 $$ **Step 3: row 1 with column 2.** $$ C_{12}=1(0)+2(3)=0+6=6 $$ **Step 4: row 2 with column 1.** $$ C_{21}=3(2)+4(1)=6+4=10 $$ **Step 5: row 2 with column 2.** $$ C_{22}=3(0)+4(3)=0+12=12 $$ **Step 6: assemble the output by position.** $$ C=\begin{bmatrix}4&6\\10&12\end{bmatrix} $$ **Final answer:** C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2. ### Two products that must not be confused **Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.** ### Given a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5. ### To find The 2 × 1 entry-by-entry product and the 2 × 2 outer product. ### Formula — which one and why ⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons. ⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row. $$ (a\odot b)_i=a_i b_i $$ $$ (ab^T)_{ij}=a_i b_j $$ **Yaad rakhne ka cue:** ⊙ mein same jagah; outer product mein har pair. ### Solution — step by step **Step 1: multiply matching entries.** $$ a\odot b=\begin{bmatrix}8\\15\end{bmatrix} $$ **Step 2: transpose b.** $$ b^T=\begin{bmatrix}4&5\end{bmatrix} $$ **Step 3: form row 1 of the outer product.** $$ 2(4)=8,\qquad2(5)=10 $$ **Step 4: form row 2 of the outer product.** $$ 3(4)=12,\qquad3(5)=15 $$ **Step 5: write both rows.** $$ ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix} $$ **Final answer:** Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ . ## Drill 5 Matrix rows receive inputs ## Question **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ed1fcbfdf5.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ. Activation: ReLU on each result. ### To find Find Wx+b and the activated output. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formula Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row. $$ z=Wx+b,\qquad a_j=\max(0,z_j) $$ $$ W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix} $$ ### Step 1: calculate receiving neuron 1 from row 1 $$ z_1=(1\times2)+(2\times1)+0.5 $$ $$ z_1=2+2+0.5=4.5 $$ ### Step 2: calculate receiving neuron 2 from row 2 $$ z_2=((-1)\times2)+(3\times1)+(-0.5) $$ $$ z_2=-2+3-0.5=0.5 $$ ### Step 3: apply ReLU to each entry $$ a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5 $$ $$ z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix} $$ The shape is 2 rows by 1 column: one output for each receiving row. ## Layer forward pass A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f(z^{(l)}) $$ ## Forward propagation through multiple neurons **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations. ### Input, hidden and output layers The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task. In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections. ![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg) Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights. A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge. ### General layer equations and dimensions Let a⁽⁰⁾ = x. For layer l: $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f^{(l)}(z^{(l)}) $$ If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j. $$ W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1} $$ For one hidden layer, these computations compose to give: $$ h=f^{(1)}(W^{(1)}x+b^{(1)}) $$ $$ \hat y=f^{(2)}(W^{(2)}h+b^{(2)}) $$ The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer. ### Forward propagation versus training Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture. ### Complete two-layer numerical **Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.** ### Given A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. ### To find Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg) Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge. **Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output. **Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias. $$ z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3 $$ **Step 3: calculate the second hidden preactivation.** $$ z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5 $$ **Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them. $$ a^{(1)}_1=\max(0,0.3)=0.3 $$ $$ a^{(1)}_2=\max(0,0.5)=0.5 $$ $$ a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix} $$ **Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs. $$ z^{(2)}=0.5(0.3)+0.6(0.5)+0.1 $$ $$ z^{(2)}=0.15+0.30+0.10=0.55 $$ **Step 6: apply the output activation.** $$ \hat y=\max(0,0.55)=0.55 $$ **Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass. ## Model answer: forward propagation, backpropagation and updating ## Question **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations. ## Answer — step by step Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs. ![Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge. | Operation | Direction or action | Result | | --- | --- | --- | | Forward propagation | Inputs move through successive layers | Predictions and intermediate activations | | Loss calculation | Compare predictions with targets | A scalar objective for the example or batch | | Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters | | Optimizer update | Use those gradients to change parameters | A new set of weights and biases | ### 1. Forward propagation At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ ### 2. Loss Use the loss specified by the task. For one regression output, half squared loss is one possible choice: $$ L=\frac12(a-y)^2 $$ Here a is the prediction and y is the target. Other stated losses produce different derivatives. ### 3. Backpropagation Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it. ### 4. Update and repeat $$ w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w} $$ Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 11 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 12 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Sigmoid and its derivative Sigmoid takes a raw score and changes it to a number between 0 and 1. Its derivative is a different number: it tells backpropagation how sensitive that output is to its raw score. The complete example below calculates both and shows where each is used. $$ \sigma(z)=\frac{1}{1+e^{-z}} $$ $$ \sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ### sigmoid example 1 **Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.** ### Given z = -2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(-2)=2 $$ **Step 2: calculate the exponential.** $$ e^{2}\approx7.389056099 $$ **Step 3: add one in the denominator.** $$ 1+7.389056099\approx8.389056099 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: use the activated output in the slope.** $$ f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585 $$ **Final answer:** a ≈ 0.119202922. Derivative ≈ 0.104993585. ### sigmoid example 2 **Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.** ### Given z = 0; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: add one in the denominator.** $$ 1+1.000000000\approx2.000000000 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: use the activated output in the slope.** $$ f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000 $$ **Final answer:** a ≈ 0.500000000. Derivative ≈ 0.250000000. ### sigmoid example 3 **Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.** ### Given z = 2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(2)=-2 $$ **Step 2: calculate the exponential.** $$ e^{-2}\approx0.135335283 $$ **Step 3: add one in the denominator.** $$ 1+0.135335283\approx1.135335283 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: use the activated output in the slope.** $$ f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585 $$ **Final answer:** a ≈ 0.880797078. Derivative ≈ 0.104993585. ## Tanh and its derivative Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input. $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ \tanh\prime(z)=1-\tanh^2(z) $$ **Question. Calculate tanh(0.5) and its derivative with respect to the input.** ### Given z = 0.5. ### To find Find tanh(z) and its derivative. ### Formula — which one and why **Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a². z = value entering tanh; a = its output. A prime means derivative, or local rate of change. $$ a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}} $$ $$ \frac{da}{dz}=1-a^2 $$ **Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square. ### Solution — step by step **Step 1: identify the input.** The input z is 0.5. In this version of the tanh formula, the exponential uses 2z. $$ 2z=2(0.5)=1,\qquad e^{2z}=e^1\approx2.718281828 $$ **Step 2: calculate the numerator.** Subtract 1 from that exponential. $$ e^{2z}-1=2.718281828-1=1.718281828 $$ **Step 3: calculate the denominator.** Add 1 to the same exponential. $$ e^{2z}+1=2.718281828+1=3.718281828 $$ **Step 4: divide the top by the bottom.** $$ a=\tanh(0.5)=\frac{1.718281828}{3.718281828}\approx0.462117157 $$ **Step 5: calculate the local derivative.** Square the output a and subtract it from 1. Keep the calculator's unrounded result. $$ a^2\approx0.213552267,\qquad1-a^2\approx0.786447733 $$ **Final answer:** tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward. ### tanh example 1 **Question. For preactivation z = -2, calculate the tanh output and state its derivative.** ### Given z = -2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(-2)=-4 $$ **Step 2: calculate the exponential.** $$ e^{-4}\approx0.018315639 $$ **Step 3: calculate numerator and denominator separately.** $$ 0.018315639-1\approx-0.981684361 $$ $$ 0.018315639+1\approx1.018315639 $$ **Step 4: divide.** $$ a=\frac{-0.981684361}{1.018315639}\approx-0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(-2)=1-(-0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ -0.964027580. Derivative ≈ 0.070650825. ### tanh example 2 **Question. For preactivation z = 0, calculate the tanh output and state its derivative.** ### Given z = 0; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: calculate numerator and denominator separately.** $$ 1.000000000-1\approx0.000000000 $$ $$ 1.000000000+1\approx2.000000000 $$ **Step 4: divide.** $$ a=\frac{0.000000000}{2.000000000}\approx0.000000000 $$ **Step 5: square the activated output for its slope.** $$ f\prime(0)=1-(0.000000000)^2\approx1.000000000 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 1.000000000. ### tanh example 3 **Question. For preactivation z = 2, calculate the tanh output and state its derivative.** ### Given z = 2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(2)=4 $$ **Step 2: calculate the exponential.** $$ e^{4}\approx54.598150033 $$ **Step 3: calculate numerator and denominator separately.** $$ 54.598150033-1\approx53.598150033 $$ $$ 54.598150033+1\approx55.598150033 $$ **Step 4: divide.** $$ a=\frac{53.598150033}{55.598150033}\approx0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(2)=1-(0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ 0.964027580. Derivative ≈ 0.070650825. ## ReLU and its derivative ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side. $$ f(z)=\max(0,z) $$ $$ f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### ReLU example 1 **Question. For preactivation z = -2, calculate the ReLU output and state its derivative.** ### Given z = -2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 0.000000000. ### ReLU example 2 **Question. For preactivation z = 0, calculate the ReLU output and state its derivative.** ### Given z = 0; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### ReLU example 3 **Question. For preactivation z = 2, calculate the ReLU output and state its derivative.** ### Given z = 2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ## Leaky ReLU Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function. $$ f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### Leaky ReLU example 1 **Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = -2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=-0.2 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0.1 $$ **Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000. ### Leaky ReLU example 2 **Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 0; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### Leaky ReLU example 3 **Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ## Stable softmax Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ $$ c=\max_j z_j $$ ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. ### Softmax for [0, 0, 0] **Question. Find stable softmax probabilities for logits [0, 0, 0].** ### Given Logits = [0, 0, 0]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}0&0&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{0}\approx1.000000000 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{0}\approx1.000000000 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=1.000000000+1.000000000+1.000000000\approx3.000000000 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding. ### Softmax for [1, 2, 3] **Question. Find stable softmax probabilities for logits [1, 2, 3].** ### Given Logits = [1, 2, 3]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}-2&-1&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{-2}\approx0.135335283 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{-1}\approx0.367879441 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=0.135335283+0.367879441+1.000000000\approx1.503214724 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{0.135335283}{1.503214724}\approx0.090030573 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{0.367879441}{1.503214724}\approx0.244728471 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{1.503214724}\approx0.665240956 $$ **Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding. ## Squared and absolute losses SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate. $$ \mathrm{SSE}=\sum_i(o_i-y_i)^2 $$ $$ L_{\mathrm{half}}=\frac{\mathrm{SSE}}2 $$ $$ L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n $$ $$ L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i| $$ ## Loss: how one wrong prediction becomes a number **Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative. $$ e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2 $$ For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective. ### Derivatives and their scaling When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_j}=2e_j,\qquad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_j}=e_j,\qquad \frac{\partial\mathrm{MSE}}{\partial\hat y_j}=\frac{2}{n}e_j $$ The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight. **Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.** ### Given Predictions = [0.2,0.7,0.4]. Targets = [0,1,1]; n = 3 scalar predictions. ### To find Find SSE, half SSE and MSE, then all three prediction-gradient lists. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step **Step 1: calculate the three signed prediction errors.** $$ e=[0.2-0,\ 0.7-1,\ 0.4-1]=[0.2,-0.3,-0.6] $$ **Step 2: square every error.** $$ e\odot e=[0.04,0.09,0.36] $$ **Step 3: add the three squared errors.** $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ **Step 4: apply the requested loss divisors.** $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245,\qquad \mathrm{MSE}=\frac{0.49}{3}\approx0.163333333 $$ **Step 5: multiply every error by 2 for the SSE gradient.** $$ \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2] $$ **Step 6: use each error itself for the half-SSE gradient.** $$ \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6] $$ **Step 7: multiply every error by 2/3 for the MSE gradient.** $$ \nabla_{\hat y}\mathrm{MSE}=\frac23[0.2,-0.3,-0.6]=[0.133333333,-0.2,-0.4] $$ The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate. ## Drill 2 Half sum versus mean ## Question **Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE. ## Answer — step by step ### Given Predictions = [0.2, 0.7, 0.4]. Targets = [0, 1, 1]; number of outputs n = 3. ### To find Find SSE, half SSE, MSE and the prediction gradients requested in the question. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step ### Given values and loss definitions There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i. $$ \mathrm{SSE}=\sum_{i=1}^{n}e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}2 $$ $$ \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}n $$ SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively. ### Step 1: calculate each residual and its square | Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual | | --- | --- | --- | --- | --- | | 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 | | 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 | | 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 | ### Step 2: sum the squared residuals $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel. ### Step 3: apply each requested normalization For half SSE, the denominator is 2: $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245 $$ For MSE, the denominator is the number of pairs, n = 3: $$ \mathrm{MSE}=\frac{0.49}{3}=0.163333333\ldots $$ **Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question. ### Absolute error uses a different operation **Question. Targets are [2, 4] and predictions are [1, 6]. Find mean absolute error.** ### Given Targets = [2,4]; predictions = [1,6]; n = 2. ### To find Find mean absolute error. ### Formula — which one and why **Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes. n = number of predictions. |e| means the nonnegative magnitude of error e. $$ \mathrm{MAE}=\frac1n\sum_i|\hat y_i-y_i| $$ **Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square. ### Solution — step by step The signed errors are 1 − 2 = −1 and 6 − 4 = 2. Absolute value keeps the size of a number without its sign, so their absolute errors are 1 and 2. Add them to obtain 3. There are two prediction–target pairs, so divide by 2. $$ \mathrm{MAE}=\frac{|1-2|+|6-4|}{2}=\frac{1+2}{2}=1.5 $$ **Final answer: MAE = 1.5.** Do not square the errors for MAE. ## Binary cross entropy Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### Binary cross-entropy: target 1, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.8 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.223143551)=0.223143551 $$ **Final answer:** Loss ≈ 0.223143551 nats. ### Binary cross-entropy: target 0, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 0; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ 1-p=1-0.8=0.2 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.2)\approx-1.609437912 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-1.609437912)=1.609437912 $$ **Final answer:** Loss ≈ 1.609437912 nats. ### Binary cross-entropy: target 1, predicted positive-class probability 0.5 **Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.5; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.5 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.693147181)=0.693147181 $$ **Final answer:** Loss ≈ 0.693147181 nats. ## Categorical cross entropy For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ \log_2q=\frac{\ln q}{\ln2} $$ $$ L=-\sum_i y_i\ln p_i=-\ln p_{true} $$ ## Cross-entropy for probability distributions **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities. $$ H(p,q)=-\sum_{i=1}^{K}p_i\log q_i $$ Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms. ### Logarithm base and units Natural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2: $$ \log_2 q=\frac{\ln q}{\ln 2} $$ For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities. ### A target distribution with two nonzero entries **Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.** ### Given A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms. ### To find Find cross-entropy in the requested units. ### Formula — which one and why **Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute. pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ H_2(p,q)=\frac{H(p,q)}{\ln2} $$ **Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai. ### Solution — step by step **Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded. $$ H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)] $$ **Step 2: calculate the logarithms.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: weight each negative logarithm by its target probability.** $$ -0.75\ln(0.5)\approx0.519860385 $$ $$ -0.25\ln(0.5)\approx0.173286795 $$ **Step 4: add the contributions, retaining calculator precision.** $$ H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats} $$ **Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1. $$ H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit} $$ **Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale. ### Cross-entropy, entropy and relative entropy These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p) $$ For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus. **Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.** ### Given For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats. ### To find Find entropy and KL divergence in nats. ### Formula — which one and why **Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p. p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p) $$ **Yaad rakhne ka cue:** KL = cross-entropy minus target ki own entropy. ### Solution — step by step **Step 1: calculate the target entropy.** $$ H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)] $$ $$ \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361 $$ $$ H(p)\approx0.215761554+0.346573590=0.562335145 $$ **Step 2: subtract target entropy from cross-entropy.** $$ D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036 $$ **Final answer:** H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ. ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. ## Output delta for half squared loss Delta is a short name for the loss derivative with respect to a neuron’s raw input z. For half squared loss, multiply prediction minus target by the activation’s derivative. The result is then multiplied by an incoming activation to obtain a weight gradient. $$ \delta^{out}=(o-y)f\prime(z) $$ ## Loss scaling: derive the gradient instead of guessing a factor ### 1. One error, three possible objectives Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square: ![Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge. $$ \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1 $$ Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below. | Stated loss over n outputs | Derivative with respect to one prediction aⱼ | | --- | --- | | SSE: sum of all squared errors | 2(aⱼ − yⱼ) | | Half SSE: half of that sum | aⱼ − yⱼ | | MSE: that sum divided by n | 2(aⱼ − yⱼ)/n | | Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n | ### 2. Why only the matching output appears Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor. Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages. ### 3. The activation contributes another factor If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore: $$ \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a) $$ For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question. ### 4. A numerical comparison For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042. The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used. ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ## Drill 11 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 12 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Sigmoid plus BCE; softmax plus CCE Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect. $$ \delta^{out}=p-y $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### The multiclass version, entry by entry **Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.** ### Given Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0]. One-example categorical cross-entropy. ### To find Find all three logit derivatives. ### Formula — which one and why **Softmax with categorical cross-entropy.** For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target. zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry. $$ \frac{\partial L}{\partial z_i}=p_i-y_i $$ **Yaad rakhne ka cue:** Softmax + CCE together: p minus y. Extra sigmoid slope nahi. ### Solution — step by step A logit is the raw score before softmax. Subtract each target entry from its matching probability entry. $$ \frac{\partial L}{\partial z_1}=0.1-0=0.1 $$ $$ \frac{\partial L}{\partial z_2}=0.8-1=-0.2 $$ $$ \frac{\partial L}{\partial z_3}=0.1-0=0.1 $$ **Final answer: [0.1, −0.2, 0.1].** Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor. ### Sigmoid+BCE logit gradient: p=0.8, y=1 **Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 1; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(-1.25)(0.16)=-0.2 $$ **Final answer:** dL/dz = -0.2. No further sigmoid derivative is needed. ### Sigmoid+BCE logit gradient: p=0.8, y=0 **Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 0; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-0}{0.16}=5 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(5)(0.16)=0.8 $$ **Final answer:** dL/dz = 0.8. No further sigmoid derivative is needed. ## Hidden delta, weight gradient and bias gradient A hidden neuron can influence several outputs. Add all those backward contributions before multiplying by its own activation derivative. For each connection, use the receiving neuron’s delta and the sending neuron’s activation. Keep the old weights unchanged until all gradients have been found. $$ \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out} $$ $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ ## More outputs and vectorized backpropagation **Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path. ### Why the hidden derivative is a sum If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives: $$ \frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2} $$ For any number of connected outputs, sum their contributions and then differentiate the hidden activation: $$ \delta_{h_j}=\left(\sum_k W_{2,kj}\delta_{o_k}\right)f'(z_{h_j}) $$ Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss. ![The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_branch_gradient.svg) Diagram: The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.. Labels: Two output paths both affect the hidden neuron | Two output paths both affect the hidden neuron | Example: hidden activation h = 0.6, using sigmoid in the hidden neuron. | h = 0.6 | Output 1 delta = 0.2 | Connecting weight = 0.4 | Output 2 delta = −0.1 | Connecting weight = 0.5 | Downstream sum = 0.2 × 0.4 + (−0.1) × 0.5 = 0.03 | Hidden delta = 0.03 × 0.6 × (1 − 0.6) = 0.0072 | Add the two downstream paths first. Then multiply by the hidden activation | derivative. The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative. Tap the diagram to enlarge. ### Matrix equations for one training sample Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂: $$ L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2 $$ $$ \delta_2=(\hat y-y)\odot f_2'(z_2) $$ $$ \frac{\partial L}{\partial W_2}=\delta_2a_1^T,\qquad\frac{\partial L}{\partial b_2}=\delta_2 $$ $$ \delta_1=(W_2^T\delta_2)\odot f_1'(z_1) $$ $$ \frac{\partial L}{\partial W_1}=\delta_1x^T,\qquad\frac{\partial L}{\partial b_1}=\delta_1 $$ Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation. | Quantity | Shape | Reason | | --- | --- | --- | | W₁ and its gradient | m × d | m receiving hidden neurons, d inputs | | W₂ and its gradient | k × m | k receiving output neurons, m hidden activations | | δ₂ | k × 1 | One derivative per output preactivation | | W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit | | δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ | ### How the loss changes the output delta For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target: $$ \delta_{\mathrm{output}}=\hat y-y $$ That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required. For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 11 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 12 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 13 A negative ReLU blocks this path ## Question **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2. ## Answer — step by step ### Given x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2. Hidden activation: ReLU; output activation: linear. Target y = 1; loss: half squared error. ### To find Find the gradients of w₁ and w₂ and explain any zero gradients. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output. $$ z_h=w_1x+b_1,\quad h=\max(0,z_h),\quad o=w_2h+b_2 $$ $$ L=\frac12(o-y)^2,\quad \delta_o=o-y $$ $$ \frac{\partial L}{\partial w_2}=\delta_o h,\quad \delta_h=\delta_o w_2\operatorname{ReLU}'(z_h),\quad \frac{\partial L}{\partial w_1}=\delta_h x $$ ### Step 1: hidden weighted sum and activation $$ z_h=(-0.4)\times2+0.1=-0.8+0.1=-0.7 $$ $$ h=\max(0,-0.7)=0 $$ Because the raw hidden score is negative, ReLU's derivative at this point is 0. ### Step 2: output prediction and output delta $$ o=0.5\times0+0.2=0.2 $$ $$ \delta_o=0.2-1=-0.8 $$ ### Step 3: output weight gradient $$ \frac{\partial L}{\partial w_2}=(-0.8)\times0=0 $$ This is zero because the input to that connection, h, is zero. ### Step 4: hidden delta and input weight gradient $$ \delta_h=(-0.8)\times0.5\times0=0 $$ $$ \frac{\partial L}{\partial w_1}=0\times2=0 $$ **Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero. ## Gradient descent A gradient tells us the local direction in which loss rises. Gradient descent subtracts a small multiple of that gradient. Learning rate η controls the size of this correction. After the first update, calculate the next gradient at the new parameter. $$ \theta_{new}=\theta-\eta g $$ $$ g=\frac{\partial J}{\partial\theta} $$ ## Reading an optimizer formula without getting lost An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated. ![Old value, current gradient and new value belong to different moments in the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. Old value, current gradient and new value belong to different moments in the update. Tap the diagram to enlarge. ### Old, current and new At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position. $$ \theta_t=\theta_{t-1}-\eta g_t $$ η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition. ### What a stored state means An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares. ### A weighted average from ordinary arithmetic If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square. $$ 0.9\times4=3.6,\qquad(1-0.9)\times9=0.9 $$ $$ s_{\mathrm{new}}=3.6+0.9=4.5 $$ The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum. ### Squared memory and signed direction For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g². ### Epsilon and hats ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second. ## The ordinary gradient descent update **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate. ### Explanation and working $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent. Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction. ![Calculate the slope first, multiply it by the learning rate, then subtract that correction.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: Calculate the slope first, multiply it by the learning rate, then subtract that correction.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. Calculate the slope first, multiply it by the learning rate, then subtract that correction. Tap the diagram to enlarge. **Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations. ### Given J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1. ### To find Differentiate J and perform two gradient-descent iterations. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ². **Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79. **Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061. **Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**. The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum. **Common error:** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients. ## Drill 16 GD variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ \theta_1=1-0.1(1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ \theta_2=0.9-0.1(0.9)=0.81 $$ **Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 17 GD variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ \theta_1=-0.5-0.1(-1.5)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ \theta_2=-0.35-0.1(-1.35)=-0.215 $$ **Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. ## Batch and mini-batch gradient A batch is a group of training examples used together for one update. An epoch is one pass through the dataset. Averaging gradients and counting updates are different questions, so both receive complete worked examples below. $$ g_{batch}=\frac1B\sum_{i=1}^{B}g_i $$ $$ U_{epoch}=\left\lceil\frac NB\right\rceil $$ ## Model answer: batch, stochastic and mini-batch training ## Question **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration. ## Answer — step by step All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update. ![All methods visit the same six examples. Their grouping determines how often they update.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: All methods visit the same six examples. Their grouping determines how often they update.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge. A **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations. | Basis | Batch GD | Stochastic GD | Mini-batch GD | | --- | --- | --- | --- | | Examples in one gradient | Entire training dataset | One example | A smaller group of B examples | | Updates per epoch | One | N | Ceiling of N/B if the remainder is retained | | Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging | | Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation | ### Example to distinguish the three counts With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch. The basic parameter update is unchanged: $$ \theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g $$ For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples. ## A tiny dataset shows why the methods differ **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order. ### Given Model ŷ = wx; initial w = 1; η = 0.1. Samples in order: (1,2), (2,4). Mean half squared loss for the batch. ### To find Compare one batch update with a sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### One batch update **Step 1: predict sample 1 with the original weight.** $$ \hat y_1=1(1)=1 $$ **Step 2: calculate its error.** $$ e_1=1-2=-1 $$ **Step 3: multiply its error by its input.** $$ g_1=(-1)(1)=-1 $$ **Step 4: predict sample 2 with the same original weight.** $$ \hat y_2=1(2)=2 $$ **Step 5: calculate its error.** $$ e_2=2-4=-2 $$ **Step 6: multiply its error by its input.** $$ g_2=(-2)(2)=-4 $$ **Step 7: average the two gradients.** $$ g=\frac{-1+(-4)}2=-2.5 $$ **Step 8: update once.** $$ w_{\mathrm{new}}=1-0.1(-2.5)=1.25 $$ ![This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge. ### One sequential SGD epoch **Step 1: use the first sample gradient.** $$ g_1=-1 $$ **Step 2: update after the first sample.** $$ w_1=1-0.1(-1)=1.1 $$ **Step 3: predict sample 2 using the changed weight.** $$ \hat y_2=1.1(2)=2.2 $$ **Step 4: find sample 2 error.** $$ e_2=2.2-4=-1.8 $$ **Step 5: find the new sample 2 gradient.** $$ g_2=(-1.8)(2)=-3.6 $$ **Step 6: update after sample 2.** $$ w_2=1.1-0.1(-3.6)=1.46 $$ **Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples. Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim. ## Drill 14 Batch and sequential SGD ## Question **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch. ## Answer — step by step ### Given Model ŷ = wx; initial w = 0; η = 0.1. Samples in order: (x,y) = (1,1), then (2,2). Batch objective: mean half squared loss. ### To find Compare the weight after one batch update and one sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### Meaning and formulas For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample. $$ \hat y=wx,\quad L=\frac12(\hat y-y)^2,\quad g=(\hat y-y)x $$ $$ g_{\mathrm{batch}}=\frac{g_1+g_2}{2},\qquad w_{\mathrm{new}}=w-\eta g $$ **Step 1: Part A: one batch update from w = 0.** For sample (1,1), prediction = 0 × 1 = 0, so: $$ g_1=(0-1)\times1=-1 $$ For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so: $$ g_2=(0-2)\times2=-4 $$ Average these two gradients, then update once. $$ g_{\mathrm{batch}}=\frac{-1+(-4)}2=-2.5 $$ $$ w_{\mathrm{batch}}=0-0.1(-2.5)=0.25 $$ **Step 2: Part B: one SGD epoch, restarting at w = 0.** The first sample has gradient −1, just as above. $$ w_1=0-0.1(-1)=0.1 $$ For the second sample use this new weight, so prediction is: $$ \hat y_2=0.1\times2=0.2 $$ $$ g_2=(0.2-2)\times2=(-1.8)\times2=-3.6 $$ $$ w_2=0.1-0.1(-3.6)=0.1+0.36=0.46 $$ **Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates. ## Drill 15 Incomplete mini batch ## Question **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it. ## Answer — step by step ### Given N = 103 samples; B = 20 samples per mini-batch. E = 3 epochs. Compare both remainder policies: keep or drop the final 3 examples. ### To find Find total updates for full-batch GD, SGD, and both mini-batch policies. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step ### Meaning and formulas An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs. $$ U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NE $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained. ### Step 1: batch GD All 103 samples form one batch per epoch. $$ U_{\mathrm{batch}}=3\times1=3 $$ ### Step 2: SGD Each sample gets its own update. $$ U_{\mathrm{SGD}}=103\times3=309 $$ ### Step 3: mini-batches with the remainder $$ 103=5\times20+3 $$ The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give: $$ U_{\mathrm{keep}}=6\times3=18 $$ ### Step 4: mini-batches without the remainder Only the five complete batches are used each epoch. $$ U_{\mathrm{drop}}=5\times3=15 $$ **Final answer:** 3, 309, 18 and 15 updates respectively. ## Momentum: signed velocity Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ## Model answer: momentum and Nesterov ## Question **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ? ## Answer — step by step Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions. ![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge. ### Momentum formulas This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained. $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ### Nesterov's change Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look}}) $$ The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point. | Basis | Momentum | Nesterov | | --- | --- | --- | | Gradient evaluated at | Current parameter | Lookahead parameter | | Information retained | Previous velocity | Previous velocity | | Intuition | Continue useful movement | Inspect the slope where old momentum points | | Initial zero velocity | No movement contribution yet | First lookahead equals current parameter | Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently. ## Momentum remembers movement **Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions. ### Update equations and convention This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter: $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters. ![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg) Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients. Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration. **Step 1: Iteration 1: gradient, velocity, parameter.** $$ g_1=J'(1)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Iteration 2: use θ₁ and v₁.** $$ g_2=J'(0.9)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ | Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter | | --- | --- | --- | --- | --- | | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.9 | −0.18 | 0.72 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient. ### Properties and alternative notation Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective. Some texts store a positive gradient accumulator u instead: $$ u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t $$ With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions. ## Drill 18 Momentum variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ **Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 19 Momentum variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ v_2=0.9(0.15)-0.1(-1.35)=0.27 $$ $$ \theta_2=-0.35+(0.27)=-0.08 $$ **Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update. ## Nesterov: look-ahead gradient Nesterov first uses the old movement to choose a temporary look-ahead position. It measures the gradient there, calculates the new movement, then adds that movement to the original parameter. The temporary look-ahead position does not replace the original parameter in the final addition. $$ \theta_{look}=\theta+\beta v $$ $$ v_{new}=\beta v-\eta g(\theta_{look}) $$ $$ \theta_{new}=\theta+v_{new} $$ ## Nesterov looks before correcting **Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update. ### What changes relative to momentum Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured: $$ \theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look},t}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁. ![Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective. **Iteration 1 — step 1: locate the lookahead point.** $$ \theta_{\mathrm{look},1}=1+0.9(0)=1 $$ **Step 2: evaluate the gradient there.** $$ g_1=J'(1)=1 $$ **Step 3: calculate the new velocity and actual parameter.** $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Iteration 2 — step 1: use the previous actual parameter and velocity.** $$ \theta_{\mathrm{look},2}=0.9+0.9(-0.1)=0.81 $$ **Step 2: evaluate the gradient at 0.81, not at 0.9.** $$ g_2=J'(0.81)=0.81 $$ **Step 3: calculate the new velocity.** $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ **Step 4: update the old actual parameter.** $$ \theta_2=0.9+(-0.171)=0.729 $$ | Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ | | --- | --- | --- | --- | --- | --- | | 1 | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice. ## Drill 20 Nesterov variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=1+0.9(0)=1 $$ $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81 $$ $$ g_2=0.81-(0)=0.81 $$ $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ $$ \theta_2=0.9+(-0.171)=0.729 $$ **Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. ## Drill 21 Nesterov variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=-0.5+0.9(0)=-0.5 $$ $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=-0.35+0.9(0.15)=-0.215 $$ $$ g_2=-0.215-(1)=-1.215 $$ $$ v_2=0.9(0.15)-0.1(-1.215)=0.2565 $$ $$ \theta_2=-0.35+(0.2565)=-0.0935 $$ **Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. ## AdaGrad AdaGrad stores a running sum of squared gradients for each parameter. Squaring removes the sign only in this stored total. The unsquared gradient in the numerator still decides the direction of the update. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ ## AdaGrad adds every squared gradient **Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator. ### State, effective learning rate and update For one parameter θ, let gₜ be its current gradient and s₀ = 0: $$ s_t=s_{t-1}+g_t^2 $$ $$ \eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon} $$ $$ \theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t $$ s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign. ![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg) Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update. The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge. ### Complete numerical: supplied gradients **Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter. **Iteration 1 — step 1: update the accumulator.** $$ s_1=0+2^2=4 $$ **Step 2: calculate the effective learning rate.** $$ \eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05 $$ **Step 3: apply the signed gradient.** $$ \theta_1=1-0.05(2)=0.9 $$ **Iteration 2 — step 1: retain s₁ and add the new squared gradient.** $$ s_2=4+1^2=5 $$ **Step 2: calculate the new coefficient.** $$ \eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360 $$ **Step 3: update from θ₁.** $$ \theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640 $$ **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5. ### Advantage, limitation and epsilon convention Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes. ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical. ## Drill 22 AdaGrad variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_1=0+(2)^2=4 $$ $$ \sqrt{G_1}=\sqrt{4}=2 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** Add the new squared gradient to all previous squares. $$ G_2=4+(1)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=0.9-\frac{0.1\times(1)}{\sqrt{5}}=0.85527864 $$ ## Drill 23 AdaGrad variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Add the new squared gradient to all previous squares. $$ G_1=0+(-1)^2=1 $$ $$ \sqrt{G_1}=\sqrt{1}=1 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_2=1+(2)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=-0.4-\frac{0.1\times(2)}{\sqrt{5}}=-0.489442719 $$ ## RMSProp RMSProp remembers a weighted average of squared gradients. With decay 0.9, keep 90% of the old stored value and add 10% of the new squared gradient. Calculate this new stored value before taking its square root and updating the parameter. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ ## A moving average remembers the recent past **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients. $$ m_t=\beta m_{t-1}+(1-\beta)g_t,\qquad0\le\beta<1 $$ For m₀ = 0, expanding the recurrence gives: $$ m_t=(1-\beta)\sum_{k=1}^{t}\beta^{t-k}g_k $$ The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history. ### Signed gradients versus squared gradients An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form. **Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂. ### Given β = 0.9; initial m₀ = 0. Gradients in order: g₁ = 2, g₂ = −1. ### To find Find m₁ and m₂, preserving the sign of each gradient. ### Formula — which one and why **Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state. β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state. $$ m_t=\beta m_{t-1}+(1-\beta)g_t $$ **Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai. ### Solution — step by step **Step 1: find the recent-gradient coefficient.** $$ 1-0.9=0.1 $$ **Step 2: retain the old memory for step 1.** $$ 0.9(0)=0 $$ **Step 3: weight the first gradient.** $$ 0.1(2)=0.2 $$ **Step 4: add to obtain the first memory.** $$ m_1=0+0.2=0.2 $$ **Step 5: retain the first memory for step 2.** $$ 0.9(0.2)=0.18 $$ **Step 6: weight the second gradient.** $$ 0.1(-1)=-0.1 $$ **Step 7: add to obtain the second memory.** $$ m_2=0.18+(-0.1)=0.08 $$ **Final answer:** m₁ = 0.2; m₂ = 0.08. The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable. ## RMSProp remembers recent squared gradients **Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator. ### Recurrence and interpretation Using decay β and zero initial state: $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update. If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever. ![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg) Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient. Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β = 0.9. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step **Iteration 1 — step 1: calculate the squared-gradient average.** $$ s_1=0.9(0)+0.1(2^2)=0.4 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_1}=\sqrt{0.4}\approx0.632455532 $$ $$ \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766 $$ **Step 3: update the parameter.** $$ \theta_1=1-0.316227766\approx0.683772234 $$ **Iteration 2 — step 1: use s₁, not the original zero state.** $$ s_2=0.9(0.4)+0.1(1^2)=0.46 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_2}=\sqrt{0.46}\approx0.678232998 $$ $$ \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956 $$ **Step 3: update from θ₁, retaining calculator precision.** $$ \theta_2\approx0.683772234-0.147441956\approx0.536330278 $$ **Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46. ### Comparison with AdaGrad and Adam AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum. ## Drill 24 RMSProp variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(2)^2=0.4 $$ $$ \sqrt{s_1}=\sqrt{0.4}=0.632455532 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{0.4}}=0.683772234 $$ **Step 2: Iteration 2 with supplied gradient 1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.4)+0.1(1)^2=0.46 $$ $$ \sqrt{s_2}=\sqrt{0.46}=0.678232998 $$ $$ \theta_2=0.683772234-\frac{0.1\times(1)}{\sqrt{0.46}}=0.536330278 $$ ## Drill 25 RMSProp variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(-1)^2=0.1 $$ $$ \sqrt{s_1}=\sqrt{0.1}=0.316227766 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{0.1}}=-0.183772234 $$ **Step 2: Iteration 2 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.1)+0.1(2)^2=0.49 $$ $$ \sqrt{s_2}=\sqrt{0.49}=0.7 $$ $$ \theta_2=-0.183772234-\frac{0.1\times(2)}{\sqrt{0.49}}=-0.46948652 $$ ## Adam: both memories Adam keeps two separate running averages. The first remembers signed gradients and can be negative. The second remembers squared gradients and is nonnegative. These stored averages are not weights, and they are not extra gradients to multiply together. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ ## Optimizer notation: old value, memory, new value ### 1. A parameter and a stored state are different θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries. ![The time label tells you which value to use. Carry the previous memory into the current update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: The time label tells you which value to use. Carry the previous memory into the current update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. The time label tells you which value to use. Carry the previous memory into the current update. Tap the diagram to enlarge. At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values. ### 2. A supplied gradient versus a supplied objective If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead. For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64. ### 3. A squared gradient keeps size and loses sign Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square. ### 4. Square root and epsilon belong to the denominator If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero. ### 5. A table prevents mixing iterations Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1. The worked examples below use this exact order, with every state shown. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 26 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 27 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ ## Adam: correction and update Adam starts its two averages at zero. Early averages therefore contain less total weight on observed gradients. Bias correction compensates for that initialization. The power in βᵗ uses the current update number, and both averages must be corrected before the parameter update. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \widehat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ ## Adam bias correction: why the denominator is one minus beta to a power **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate. ### Explanation and working ### 1. What the moving average actually contains Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization. ![The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam_correction.svg) Diagram: The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.. Labels: Correct the early zero-initialized memory | Correct the early zero-initialized memory | Example: β₁ = 0.9, first gradient = 2, initial m = 0. | Raw memory | m₁ = 0.9 × 0 + 0.1 × | 2 = 0.2 | Correction factor | 1 − 0.9¹ = 0.1 | Corrected memory | m̂₁ = 0.2 ÷ 0.1 = 2 | The small raw value is partly caused by the starting zero. Bias correction | removes that effect. The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor. Tap the diagram to enlarge. $$ m_1=(1-\beta)g_1 $$ At the second step, substitute the first expression into the recurrence: $$ m_2=\beta(1-\beta)g_1+(1-\beta)g_2 $$ ### 2. Add the coefficients After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term: $$ \beta(1-\beta)+(1-\beta)=(1-\beta)(1+\beta)=1-\beta^2 $$ After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ The squared-gradient memory uses its own decay β₂ and the same reasoning: $$ \widehat s_t=\frac{s_t}{1-\beta_2^t} $$ ### 3. A constant-gradient example makes the purpose visible Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38. $$ \widehat m_1=\frac{0.2}{1-0.9}=2 $$ $$ \widehat m_2=\frac{0.38}{1-0.9^2}=\frac{0.38}{0.19}=2 $$ Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient. ### 4. Two betas mean two separate powers At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators. ### 5. Correct first, then combine The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows. This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 26 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 27 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ ## L1 penalty and update L1 adds a cost based on the absolute sizes of weights. Its derivative is positive for a positive weight and negative for a negative weight. The optimizer adds this penalty gradient to the data gradient before applying the usual update. $$ J=L+\lambda\sum_j|w_j| $$ $$ w_{new}=w-\eta\left[g_{data}+\lambda\operatorname{sign}(w)\right] $$ ## Model answer: regularization and the L1–L2 difference ## Question **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength. ## Answer — step by step Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost. ![Follow the stated penalty convention, including lambda and any half factor, when you form the objective.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: Follow the stated penalty convention, including lambda and any half factor, when you form the objective.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge. ### Objectives used in this course $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question. | Basis | L1 | L2 | | --- | --- | --- | | Quantity penalized | Absolute weight magnitude | Squared weight magnitude | | Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention | | Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes | | Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights | The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total. Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective. ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 29 L1 update on positive and negative weights ## Question **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero. ## Answer — step by step ### Given Old weights = [−0.5,0.3]. Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2. ### To find Find total gradients and the two updated weights. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formula The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j) $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed. **Step 1: Weight 1: old value −0.5.** $$ \lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1 $$ $$ g_{1,\mathrm{total}}=0.2+(-0.1)=0.1 $$ $$ w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52 $$ **Step 2: Weight 2: old value 0.3.** $$ \lambda\operatorname{sign}(0.3)=0.1(1)=0.1 $$ $$ g_{2,\mathrm{total}}=-0.1+0.1=0 $$ $$ w_{2,\mathrm{new}}=0.3-0.2(0)=0.3 $$ **Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero. ## Drill 28 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 41 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. ## L2 penalty and update with a half L2 adds a cost based on squared weights. Differentiating a square produces a factor of 2. If the objective includes a half, that half cancels the 2; if it does not, the 2 remains. Read the objective before choosing the gradient. $$ J=L+\frac\lambda2\sum_jw_j^2 $$ $$ w_{new}=w-\eta(g_{data}+\lambda w) $$ ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 30 L2 update ## Question **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw². ## Answer — step by step ### Given Old weights = [0.5,−0.2]; data gradients = [0.1,0.3]. λ = 0.2; η = 0.1; penalty = (λ/2)Σw². ### To find Find penalty gradients, total gradients and updated weights. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Formula and its reason The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda w_j $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ **Step 1: Weight 1.** $$ \lambda w_1=0.2\times0.5=0.1 $$ $$ g_{1,\mathrm{total}}=0.1+0.1=0.2 $$ $$ w_{1,\mathrm{new}}=0.5-0.1(0.2)=0.48 $$ **Step 2: Weight 2.** $$ \lambda w_2=0.2\times(-0.2)=-0.04 $$ $$ g_{2,\mathrm{total}}=0.3+(-0.04)=0.26 $$ $$ w_{2,\mathrm{new}}=-0.2-0.1(0.26)=-0.226 $$ **Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative. ## Drill 28 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 41 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. ## Inverted dropout Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated. $$ q=1-p $$ $$ a_{train}=\frac{m\odot a}{q} $$ $$ a_{test}=a $$ ## Model answer: dropout with training and inference explained ## Question **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling. ## Answer — step by step Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights. ![The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge. Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop. ### Inverted dropout during training $$ a'=\frac{ma}{q} $$ A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks. For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is: $$ 0.5(6)+0.5(0)=3 $$ ### Inference Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions. Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data. ## Dropout removes activations during training **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training. ### Explanation and working Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count. ![Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout.svg) Diagram: Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.. Labels: Dropout temporarily removes selected activations | Dropout temporarily removes selected activations | Example: keep probability q = 0.5. Inverted dropout during training. | Original | 0.8 | -0.4 | 0.6 | 0.2 | Mask | 1 | 0 | 1 | 0 | After mask ÷ q | 1.6 | 0 | 1.2 | 0 | The kept values double: 0.8 ÷ 0.5 = 1.6 and 0.6 ÷ 0.5 = 1.2. At inference, no | mask is used. Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge. Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions. $$ a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout} $$ $$ a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout} $$ **Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs. ### Given Activations = [0.8,−0.4,0.6,0.2]. Dropout rate = 50%, so keep probability q = 0.5. Chosen mask = [1,0,1,0]. ### To find Find training outputs under inverted dropout. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step **Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key. **Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer. Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition. ## Drill 31 Inverted dropout ## Question **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference. ## Answer — step by step ### Given Activations = [2,−1,0,3]. Keep probability q = 0.75; keep mask = [1,0,1,1]. ### To find Find training outputs and state the inference rule. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step ### Meaning and formula The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved. $$ a'_j=\frac{m_j a_j}{q} $$ The multiplication is entry by entry; there is no dot-product sum. ### Step 1: apply the mask $$ [1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3] $$ ### Step 2: divide each result by 0.75 $$ a'_1=\frac2{0.75}=2.666666667 $$ $$ a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0 $$ $$ a'_4=\frac3{0.75}=4 $$ **Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero. ### During inference Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training. ## Drill 42 Mixed dropout and a momentum update ## Question **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1. ## Answer — step by step ### Given Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5. Linear-output weights = [0.2,0.3]; bias = 0; target = 1. Half squared loss; update only the output weights. Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1. ### To find Find dropped/scaled activations, prediction, gradients, new velocities and new output weights. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Formulas used Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients. $$ a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j $$ The signed-velocity momentum convention is: $$ v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}} $$ ### Step 1: apply inverted dropout $$ a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0 $$ ### Step 2: prediction and error $$ o=0.2(2)+0.3(0)+0=0.4 $$ $$ o-y=0.4-1=-0.6 $$ ### Step 3: current weight gradients $$ g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0 $$ ### Step 4: remembered movement plus new correction $$ v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21 $$ $$ v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09 $$ ### Step 5: add velocities to old weights $$ w_{1,\mathrm{new}}=0.2+0.21=0.41 $$ $$ w_{2,\mathrm{new}}=0.3-0.09=0.21 $$ **Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero. ## Convolution output dimensions This formula counts how many legal filter positions fit along one image dimension. Padding adds space at both ends, the kernel occupies K positions, and stride controls how far the next window moves. The floor keeps only complete valid positions. $$ N_{out}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ ## Reading image shapes before doing CNN arithmetic **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension. ### Explanation and working ### Height, width and channels The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors. ![Count height, width and channels separately. Three channels are stacked at the same spatial positions.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_shape.svg) Diagram: Count height, width and channels separately. Three channels are stacked at the same spatial positions.. Labels: Read height, width and channels separately | Read height, width and channels separately | The text uses 4 × 5 × 3: four rows, five columns and three channels. | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 3 stacked channels | H × W × C | H = 4 rows | W = 5 columns | C = 3 channels | Total entries = 60 | A filter spans the incoming channels. Its spatial width is a different quantity. Count height, width and channels separately. Three channels are stacked at the same spatial positions. Tap the diagram to enlarge. ### One position versus the whole map A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels. ### Why filter depth matches input channels A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels. ### Reading a complete shape calculation The spatial-size formula computes height and width. It does not compute the channel count. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position. ### Shape, parameters and computation are three different counts **Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights. In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count. ## Stride padding and floor **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output. ### Explanation and working Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension. ![The border adds to both sides of the input. The stride changes the next starting position of the filter window.](https://deep-learning-study-room.vercel.app/assets/figures/v_stride_padding.svg) Diagram: The border adds to both sides of the input. The stride changes the next starting position of the filter window.. Labels: Padding adds a border. Stride moves the window. | Padding adds a border. Stride moves the window. | Example: a 3 × 3 input becomes 5 × 5 after one zero border. | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 3 | 0 | 0 | 4 | 5 | 6 | 0 | 0 | 7 | 8 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | Padding P = 1 | 0 | 1 | 2 | 3 | 4 | Stride S = 2 | Starting positions: 0, 2, … | For each spatial dimension: add 2P, subtract K, divide by S, take floor, then add | 1. The border adds to both sides of the input. The stride changes the next starting position of the filter window. Tap the diagram to enlarge. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: $\left\lfloor 2.5\right\rfloor$ = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels. **Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1. ### Output dimension with a fractional intermediate result **Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.** ### Given N = 6; K = 3; S = 2; P = 0; dilation = 1. ### To find The integer output side length. ### Formula — which one and why The formula counts complete kernel placements. Flooring removes an incomplete final placement. N is input side; P is padding on each side; K is kernel side; S is stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1. ### Solution — step by step **Step 1: include padding and subtract kernel size.** $$ 6+2(0)-3=3 $$ **Step 2: divide by the stride.** $$ \frac32=1.5 $$ **Step 3: take the floor.** $$ \lfloor1.5\rfloor=1 $$ **Step 4: count the first placement.** $$ N_{\mathrm{out}}=1+1=2 $$ **Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter. ### Padding on both sides **Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.** ### Given N = 5; K = 3; S = 1; P = 1 on each side. ### To find Padded input and output dimensions. ### Formula — which one and why Padding adds P entries on each of two opposite sides; the output counts complete kernel placements. N = unpadded side; K = kernel side; S = stride; P = per-side padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\lfloor(N+2P-K)/S\rfloor+1 $$ **Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain. ### Solution — step by step **Step 1: count both padding edges.** $$ 2P=2(1)=2 $$ **Step 2: calculate the padded input side.** $$ N+2P=5+2=7 $$ **Step 3: subtract kernel width.** $$ 7-3=4 $$ **Step 4: divide by stride and take the floor.** $$ \lfloor4/1\rfloor=4 $$ **Step 5: include the first placement.** $$ N_{out}=4+1=5 $$ **Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter. “Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case. ## Even-sized kernels: SAME padding is not always symmetric **Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides. For a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention. **Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho. ### Example 1: one entry on every side **Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)** ### Given Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter. ### To find The padded input dimensions and the convolution output dimensions. ### Formula — which one and why The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel. ### Solution — step by step **Step 1: count the two padding borders.** $$ 2P=2(1)=2 $$ **Step 2: find the padded side length.** $$ N_{\mathrm{padded}}=5+2=7 $$ The padded input is 7 × 7 × 1. **Step 3: subtract the kernel size.** $$ 7-2=5 $$ **Step 4: divide by stride, take the floor and count the first placement.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5}{1}\right\rfloor+1=5+1=6 $$ **Step 5: write height, width and channels.** Both spatial dimensions equal 6. There is one filter, so there is one output channel. **Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel. ### Example 2: preserve the size with SAME padding **Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)** ### Given Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found. ### To find Padding above, below, left and right; padded-input shape; final output shape. ### Formula — which one and why For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+P_{\mathrm{before}}+P_{\mathrm{after}}-K}{S}\right\rfloor+1 $$ At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1. $$ P_{\mathrm{total}}=P_{\mathrm{before}}+P_{\mathrm{after}}=K-1 $$ ### Solution — step by step **Step 1: calculate total padding along the height.** $$ P_{\mathrm{total}}=2-1=1 $$ Only one extra row in total is needed. **Step 2: place the row using the stated convention.** Padding above = 0; padding below = 1. **Step 3: apply the same calculation to width.** Only one extra column in total is needed. Padding left = 0; padding right = 1. **Step 4: find the padded-input shape.** $$ H_{\mathrm{padded}}=5+0+1=6 $$ $$ W_{\mathrm{padded}}=5+0+1=6 $$ The padded input is 6 × 6 × 1. **Step 5: count the output positions along each axis.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5+0+1-2}{1}\right\rfloor+1=4+1=5 $$ **Step 6: include the output channel.** One filter creates one output channel. **Final answer:** add **0 above, 1 below, 0 left and 1 right**; padded input = **6 × 6 × 1**; output = **5 × 5 × 1**. The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula. ## Drill 33 Padding stride and channels ## Question **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total. ## Answer — step by step ### Given Input shape = 31 × 31 × 3. Filter size = 5 × 5; number of filters = 8. Stride S = 2; padding P = 1 per side; one bias per filter. ### To find Find output shape, weights, biases and total parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning of each given value The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels. ### Formulas used $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ \text{weights}=K^2C_{\mathrm{in}}C_{\mathrm{out}},\quad \text{biases}=C_{\mathrm{out}} $$ $$ \text{parameters}=\text{weights}+\text{biases} $$ ### Step 1: find the output height and width Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33. $$ N_{\mathrm{out}}=\left\lfloor\frac{31+2(1)-5}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{28}{2}\right\rfloor+1=14+1=15 $$ The height uses the same calculation. **Output shape = 15 × 15 × 8.** ### Step 2: count weights in one filter Each filter covers 5 × 5 positions in each of 3 channels. $$ 5\times5\times3=75\text{ weights per filter} $$ ### Step 3: count all filters and their biases $$ \text{weights}=75\times8=600 $$ $$ \text{biases}=8,\qquad \text{total}=600+8=608 $$ **Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights. ## Convolution value One convolution output is a sum of matched input–kernel products. Move the window and repeat that exact rule for the next output. With several input channels, add contributions from every channel and add the filter bias once. $$ Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o $$ ## One filter over multiple channels: all contributions go into one cell **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels. ### Explanation and working ### 1. Separate spatial size from depth Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3. ![Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.](https://deep-learning-study-room.vercel.app/assets/figures/v_channel_products.svg) Diagram: Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.. Labels: A 1 × 1 filter still mixes all three channels | A 1 × 1 filter still mixes all three channels | Use input [2, 5, 1], weights [0.1, −0.2, 0.5] and one bias 0.3. | Channel 1 | 2 × 0.1 = 0.2 | Channel 2 | 5 × (−0.2) = −1 | Channel 3 | 1 × 0.5 = 0.5 | One filter → one output value | 0.2 + (−1) + 0.5 + 0.3 = 0 | One spatial position can contain several channel values. The one-by-one filter | mixes them. Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example. Tap the diagram to enlarge. $$ z=0.1(2)+(-0.2)(5)+0.5(1)+0.3 $$ $$ z=0.2-1+0.5+0.3=0 $$ The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels. ### 2. A larger filter has one spatial grid per input channel A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias. The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified. ### 3. What changes when there are several filters? Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth. For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters. ### 4. One-by-one does not mean one weight A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed. This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately. ## Faculty example: a complete binary-image convolution **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)** ### Given Image X = $\begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}$ . Filter K = $\begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}$ . These are the values in the Unit 4 lecture example. ### To find The output shape and all nine feature-map values. ### Formula — which one and why The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window. ![Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.](https://deep-learning-study-room.vercel.app/assets/figures/u4_binary_windows.svg) Diagram: Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.. Labels: u4 binary windows | One filter, nine window positions | Input: 5 × 5 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | Shared filter: 3 × 3 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | → | Output: 3 × 3 | 4 | 3 | 4 | 2 | 4 | 3 | 2 | 3 | 4 | Orange window → orange output cell = 4. Move one cell at a time. Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions. Tap the diagram to enlarge. ### Solution — step by step **Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3. **Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**. **Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**. **Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**. **Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**. **Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**. **Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**. **Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**. **Final answer:** Y = $\begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}$ . All nine outputs use the same nine filter weights. They do not introduce nine different filters. **Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain. ## Faculty example: combine three RGB channel sums **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)** ### Given Red patch = $\begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}$ ; red kernel = $\begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}$ . Green patch = $\begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}$ ; green kernel = $\begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}$ . Blue patch = $\begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}$ ; blue kernel = $\begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}$ . Bias = 1. These are three slices of **one** full-depth filter, not three independent output filters. ### To find The contribution of each input channel and the final scalar output at this position. ### Formula — which one and why Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions. $$ z=s_R+s_G+s_B+b $$ The quantities s\_R, s\_G and s\_B are intermediate weighted sums, not separate final output channels. ### Solution — step by step **Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1. **Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**. **Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1). **Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**. **Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1. **Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**. **Step 7: add the channel contributions.** 308 −498 +164 = −26. **Step 8: add one filter bias.** −26 +1 = **−25**. **Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages. **Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo. ## The convolution calculation **Definition.** A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention. ### Explanation and working In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it. **Question.** Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias. ### Given Input X = $\begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}$ . Kernel K = $\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$ . Stride = 1; padding = 0; bias = 0; no kernel flip. ### To find Find the 2 × 2 output map. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_cd39cabe48.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 0 | 0 | −1 | Matrix 2 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 Given matrices for this question Tap the diagram to enlarge. $$ X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix} $$ **Step 1. Top-left window.** 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4. **Step 2. Move right by one.** 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4. **Step 3. Move down and return left.** 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4. **Step 4. Bottom-right window.** 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is $\begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}$ . If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested. ![Four windows produce four output cells](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: Four windows produce four output cells. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. Four windows produce four output cells Tap the diagram to enlarge. ## Drill 32 Convolution with a negative coefficient ## Question **Question.** Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | --- | --- | | 1 | −1 | | 0 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel K are printed with the question. Stride S = 1; padding P = 0; bias = 0. Use the kernel without flipping it. ### To find Calculate all nine output cells. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−2) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(-1)+(1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 3 = 3** **2 + 3 = 5** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(-1)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(3)=5 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 2 = 2** **(−1) + 2 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(-1)+(2)=1 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(-1)+(1)=0 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −3 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−3) = −2** Row 2**0 + 0 = 0** **(−2) + 0 = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(-1)=-2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(-2)+(0)=-2 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + (−2) = 1** Row 2**0 + 1 = 1** **1 + 1 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(1)+(1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | −1 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + (−1) = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(1)+(0)=1 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 2 = 2** **1 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(2)=3 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 3 = 3** **(−1) + 3 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(-1)+(3)=2 $$ **Final output:** $\begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ## Drill 40 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. ## Max, min and average pooling Pooling reduces each window to one value. Max pooling keeps the largest entry; average pooling adds all included entries and divides by their count. Pooling is performed independently in each channel. $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{max}=\max_{(i,j)\in W}x_{ij} $$ $$ y_{avg}=\frac1{|W|}\sum_{(i,j)\in W}x_{ij} $$ ## Pooling needs comparisons or averages **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases. ### Explanation and working Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged. **Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding: ### Given The 4 × 4 input map is printed below. Window = 2 × 2; stride = 2; padding = 0. ### To find Find the pooled output by the operation(s) requested. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step $$ A=\begin{bmatrix}21&11&15&10\\12&9&10&11\\11&13&18&12\\7&4&15&12\end{bmatrix} $$ ### Top-left window **Step 1: select the four entries.** [21, 11, 12, 9] **Step 2: choose the largest entry.** $$ y_{\max}=21 $$ **Step 3: add all entries.** $$ 21+11+12+9=53 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{53}4=13.25 $$ ### Top-right window **Step 1: select the four entries.** [15, 10, 10, 11] **Step 2: choose the largest entry.** $$ y_{\max}=15 $$ **Step 3: add all entries.** $$ 15+10+10+11=46 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{46}4=11.5 $$ ### Bottom-left window **Step 1: select the four entries.** [11, 13, 7, 4] **Step 2: choose the largest entry.** $$ y_{\max}=13 $$ **Step 3: add all entries.** $$ 11+13+7+4=35 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{35}4=8.75 $$ ### Bottom-right window **Step 1: select the four entries.** [18, 12, 15, 12] **Step 2: choose the largest entry.** $$ y_{\max}=18 $$ **Step 3: add all entries.** $$ 18+12+15+12=57 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{57}4=14.25 $$ Outputs are max $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ and average $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . ![Each pooling window reduces several values to one](https://deep-learning-study-room.vercel.app/assets/figures/pooling.svg) Diagram: Each pooling window reduces several values to one. Labels: INPUT 4 × 4 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 | MAX POOL | 21 | 15 | 13 | 18 | AVERAGE POOL | 13.25 | 11.5 | 8.75 | 14.25 | 2 × 2 windows | stride 2 Each pooling window reduces several values to one Tap the diagram to enlarge. Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance. ## Max, min and average pooling on the same lecture grid **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases. **Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)** ### Given X = $\begin{bmatrix}12 & 20 & 30 & 0\\8 & 12 & 2 & 0\\34 & 70 & 37 & 4\\112 & 100 & 25 & 12\end{bmatrix}$ . Window = 2 × 2; stride = 2; padding = 0. ### To find The shared output shape and each of the four cells for each pooling method. ### Formula — which one and why Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap. $$ y_{\max}=\max(x_1,x_2,x_3,x_4) $$ $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{\mathrm{avg}}=\frac{x_1+x_2+x_3+x_4}{4} $$ ![The four colours identify four separate pooling windows. Each window creates one cell in each output map.](https://deep-learning-study-room.vercel.app/assets/figures/u4_pooling_windows.svg) Diagram: The four colours identify four separate pooling windows. Each window creates one cell in each output map.. Labels: u4 pooling windows | Same four windows, three different summaries | Input: 4 × 4 | 12 | 20 | 30 | 0 | 8 | 12 | 2 | 0 | 34 | 70 | 37 | 4 | 112 | 100 | 25 | 12 | Max | 20 | 30 | 112 | 37 | Min | 8 | 0 | 34 | 4 | Average | 13 | 8 | 79 | 19.5 | Bottom-right average: (37 + 4 + 25 + 12) ÷ 4 = 19.5 | Pooling changes values and shape; ordinary pooling learns no parameters. The four colours identify four separate pooling windows. Each window creates one cell in each output map. Tap the diagram to enlarge. ### Solution — step by step **Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2. **Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13. **Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8. **Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79. **Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20. **Step 6: put each result back in its window's position.** Max = $\begin{bmatrix}20 & 30\\112 & 37\end{bmatrix}$ . Min = $\begin{bmatrix}8 & 0\\34 & 4\end{bmatrix}$ . Average = $\begin{bmatrix}13 & 8\\79 & 19.5\end{bmatrix}$ . **Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation. ### Why the choice matters Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour. Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter. **Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide. ## Drill 34 Max and average pooling ## Question **Question.** Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_563e4ee8f7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | A | 1 | 4 | 2 | 3 | 0 | 2 | 5 | 1 | 6 | 3 | 2 | 0 | 1 | 7 | 4 | 8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input A is printed with the question. Window size = 2 × 2; stride = 2; no padding. ### To find Find the complete max-pooling and average-pooling outputs. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 4, 0, 2] **Step 2: add all selected entries.** 1 + 4 + 0 + 2 = 7 **Step 3: divide by the 4 entries in this window.** 7/4 = **1.75**. Maximum = **4** ### Pooling window (1,2) **Step 1: select the entries.** [2, 3, 5, 1] **Step 2: add all selected entries.** 2 + 3 + 5 + 1 = 11 **Step 3: divide by the 4 entries in this window.** 11/4 = **2.75**. Maximum = **5** ### Pooling window (2,1) **Step 1: select the entries.** [6, 3, 1, 7] **Step 2: add all selected entries.** 6 + 3 + 1 + 7 = 17 **Step 3: divide by the 4 entries in this window.** 17/4 = **4.25**. Maximum = **7** ### Pooling window (2,2) **Step 1: select the entries.** [2, 0, 4, 8] **Step 2: add all selected entries.** 2 + 0 + 4 + 8 = 14 **Step 3: divide by the 4 entries in this window.** 14/4 = **3.5**. Maximum = **8** **Average output:** $\begin{bmatrix}1.75 & 2.75\\4.25 & 3.5\end{bmatrix}$ . **Max output:** $\begin{bmatrix}4 & 5\\7 & 8\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. ## Drill 40 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. ## Convolution parameter count A filter has one learned weight for each spatial position in each input channel. Count the weights in one filter, add its one bias, then multiply by the number of filters. Sliding a filter to many positions reuses these weights; it does not create new parameters. $$ P_{conv}=(K_hK_wC_{in}+1)C_{out} $$ ## Output size and parameter count are different **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters. ### Explanation and working **Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias. ### Given Input = 7 × 7 × 3; two 3 × 3 filters. Stride = 2; padding = 1; include biases. ### To find Find the output shape and trainable parameter count. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step **Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2. ![Output size counts produced values; parameter count counts adjustable weights and biases.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Output size counts produced values; parameter count counts adjustable weights and biases.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Output size counts produced values; parameter count counts adjustable weights and biases. Tap the diagram to enlarge. **Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters. ## Model answer: shape, activations and parameters ## Question **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them. ## Answer — step by step Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions. ![Keep output shape, activation count and parameter count as three separate quantities.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Keep output shape, activation count and parameter count as three separate quantities.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Keep output shape, activation count and parameter count as three separate quantities. Tap the diagram to enlarge. ### Spatial output shape For an ordinary convolution with dilation 1: $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters. ### Output activations $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ This is the number of output values for one example, excluding any batch dimension. ### Trainable parameters $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions. ### Small example A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels. ## Faculty CNN table: shapes, activations and 62958 parameters **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)** ### Given Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question. ### To find Output shape and number of output values for each layer, its trainable parameters, and the total parameters. ### Formula — which one and why Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ A counts produced activations. P\_conv and P\_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here. ### Solution — step by step **Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0. **Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272. **Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**. **Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight. **Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600. **Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**. **Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters. **Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations. **Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations. **Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation. **Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**. | Layer | Output shape | Activations | Parameters | | --- | --- | --- | --- | | Input | 32 × 32 × 3 | 3,072 | 0 | | Conv1 | 28 × 28 × 8 | 6,272 | 608 | | Pool1 | 14 × 14 × 8 | 1,568 | 0 | | Conv2 | 10 × 10 × 16 | 1,600 | 3,216 | | Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 | | Dense 120 | 120 | 120 | 48,120 | | Dense 84 | 84 | 84 | 10,164 | | Dense 10 → softmax | 10 | 10 | 850 + 0 | **Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network. **Yaad rakho:** Shape aage carry karo; weights aur activations ko alag columns mein rakho. ## Parameters are different from activation values **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters. ### Explanation and working An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights. ![The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations. Tap the diagram to enlarge. $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation. ### One filter, then all filters **Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.** ### Given Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter. ### To find Weights, biases, total trainable parameters and output channels. ### Formula — which one and why Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights. Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters. $$ P=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply. ### Solution — step by step **Step 1: count weights in one filter.** $$ 3(3)(3)=27 $$ **Step 2: include its one bias.** $$ 27+1=28 $$ **Step 3: count all weights.** $$ 27(16)=432 $$ **Step 4: count all biases.** $$ 1(16)=16 $$ **Step 5: add and state channel count.** $$ P=432+16=448 $$ Each filter produces one channel, so output channels = 16. **Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels. $$ P_{\mathrm{pool}}=0,\quad P_{\mathrm{flatten}}=0,\quad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values. ## Drill 35 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. ## Flatten and dense parameter count Flattening rearranges all activation values into a list. It learns no new numbers. A dense layer then connects every value in that list to every receiving neuron; each receiving neuron also has one bias. $$ N_{flat}=HWC $$ $$ P_{dense}=(N_{in}+1)N_{out} $$ ## Counting parameters in a dense network **Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases. ### Deriving the parameter count For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters: $$ P_{\mathrm{dense}}=mn+n=(m+1)n $$ Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram. ![Count all input-to-neuron connections and then one bias per receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/v_dense_count.svg) Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters. Count all input-to-neuron connections and then one bias per receiving neuron. Tap the diagram to enlarge. ### Complete numerical **Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters. ### Given Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs. Every hidden/output neuron has one bias. ### To find Count weights and biases at each receiving layer, then the total. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step **Step 1: identify the consecutive layer widths.** $$ 10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4 $$ There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters. **Step 2: count weights and biases for each transformation.** | Transformation | Weights | Biases | Layer total | | --- | --- | --- | --- | | Input → hidden 1 | 10 × 5 = 50 | 5 | 55 | | Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 | | Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 | | Hidden 3 → output | 5 × 4 = 20 | 4 | 24 | **Step 3: add the layer totals.** $$ P=55+30+30+24=139 $$ Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139. **Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network. ## One complete CNN pipeline **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron. ### Given Input = 128 × 128 × 3. Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2. Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2. Flatten → dense 240 → dense 120 → dense 5. One bias per learned filter/dense neuron. ### To find Find every output shape and layer parameter count, then the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step **Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448. ![Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.](https://deep-learning-study-room.vercel.app/assets/figures/v_cnn_pipeline.svg) Diagram: Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.. Labels: Follow the complete CNN question, one layer at a time | Follow the complete CNN question, one layer at a time | Start with 128 × 128 × 3. Shape and learned-parameter count are separate. | LAYER | OUTPUT SHAPE / LENGTH | PARAMETERS | 1. Convolution 1 | 126 × 126 × 16 | 448 | 2. Pooling 1 | 63 × 63 × 16 | 0 | 3. Convolution 2 | 61 × 61 × 32 | 4,640 | 4. Pooling 2 | 30 × 30 × 32 | 0 | 5. Flatten | 28,800 | 0 | 6. Dense 240 | 240 | 6,912,240 | 7. Dense 120 | 120 | 28,920 | 8. Classifier | 5 | 605 | Total learned parameters = 6,946,853 | Each output is the next layer’s input. Pooling and flattening add no learned | parameters. Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count. Tap the diagram to enlarge. **Step 2. Pool 1 size.** $\left\lfloor\frac{126 - 2}{2}\right\rfloor+1$ = 63; output 63 × 63 × 16. Parameters = 0. **Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640. **Step 4. Pool 2 size.** $\left\lfloor\frac{61 - 2}{2}\right\rfloor+1$ = $\left\lfloor 29.5\right\rfloor$ + 1 = 30; output 30 × 30 × 32. Parameters = 0. **Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0. **Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240. **Step 7. Dense 120.** (240 + 1) × 120 = 28,920. **Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer. **Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer. ## Drill 35 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. ## Drill 43 Mixed CNN dimensions softmax and loss ## Question **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE. ## Answer — step by step ### Given Input = 8 × 8 × 3. Conv: kernel 3, stride 1, padding 1, 2 filters. Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs. Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]). ### To find Find shapes, parameter counts, probabilities and CCE. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Formulas used Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}} $$ Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.367879441, 1, 0.135335283]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Correct class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**. ## Convolution multiply-accumulate count A parameter count counts learned numbers. A multiply-accumulate count counts how many times input–weight products are used. One filter is reused at every output position, so computation includes output height and width even though its parameter count does not. $$ MAC=H_{out}W_{out}K_hK_wC_{in}C_{out} $$ ## Why one by one convolution saves work **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts. ![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg) Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications. Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge. **Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.** ### Given Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size. ### To find Direct MACs, total two-stage bottleneck MACs and percentage saved. ### Formula — which one and why Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs. ### Solution — step by step **Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400 $$ **Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels. $$ \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448 $$ **Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels. $$ \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200 $$ **Step 4: add the two bottleneck costs.** $$ \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648 $$ **Step 5: calculate the saving.** $$ \mathrm{saving}=120422400-12443648=107978752 $$ $$ \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\% $$ **Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption. ### A second bottleneck with different dimensions **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1. ### To find Direct MACs, bottleneck MACs and saving. ### Formula — which one and why Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers. ### Solution — step by step **Step 1: direct spatial convolution.** $$ 8(8)(8)(3)(3)(16)=73728 $$ **Step 2: 1 × 1 reduction.** $$ 8(8)(4)(1)(1)(16)=4096 $$ **Step 3: reduced-depth 3 × 3 convolution.** $$ 8(8)(8)(3)(3)(4)=18432 $$ **Step 4: add the stages and subtract from the direct cost.** $$ 4096+18432=22528 $$ $$ 73728-22528=51200 $$ **Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values. ## The faculty's 480-channel bottleneck numerical ![Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.](https://deep-learning-study-room.vercel.app/assets/figures/u4_bottleneck_480.svg) Diagram: Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.. Labels: u4 bottleneck 480 | Same output shape, different computation | Input | 14 × 14 × 480 | → | Direct 5 × 5 | 48 filters | → | Output | 14 × 14 × 48 | 112,896,000 MACs | 1 × 1 reduction | 480 → 16 channels | → | Reduced 5 × 5 | 16 → 48 channels | → | Output | 14 × 14 × 48 | 1,505,280 + 3,763,200 = 5,268,480 MACs | The lower path also starts from 14 × 14 × 480. Count both layers. Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers. Tap the diagram to enlarge. **Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.** ### Given Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters. ### To find MACs for the direct path, both bottleneck stages, their total and percentage saved. ### Formula — which one and why $$ M=H_oW_oC_oK^2C_i $$ M is the MAC count. H\_oW\_oC\_o counts output values; K²C\_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck. $$ \mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}} $$ ### Solution — step by step **Step 1: direct output count.** 14 × 14 × 48 = 9,408. **Step 2: products per direct output.** 5 × 5 × 480 = 12,000. **Step 3: direct MACs.** $$ 9408(12000)=112896000 $$ **Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values. $$ 14(14)(16)(480)=1505280 $$ **Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values. $$ 9408(400)=3763200 $$ **Step 6: add both bottleneck costs.** $$ 1505280+3763200=5268480 $$ **Step 7: calculate the saving.** $$ 112896000-5268480=107627520 $$ $$ 100\frac{107627520}{112896000}\approx95.333333\% $$ **Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone. ## Drill 37 Bottleneck cost with new values ## Question **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first. ## Answer — step by step ### Given Input = 16 × 16 × 64. Direct branch: 32 same-padded 3 × 3 filters. Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer. Spatial size remains 16 × 16; exclude bias additions. ### To find Find both MAC counts, absolute saving and percentage saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Meaning and formula A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ ### Step 1: direct 3 × 3 convolution Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 reduction from 64 to 8 channels $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ ### Step 3: 3 × 3 convolution now receives only 8 channels $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: add the two bottleneck stages $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: calculate the saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32. ### Bottleneck computation on an 8 × 8 input **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8 in both stages. Input channels = 16; output filters = 8; main kernel = 3 × 3. Bottleneck uses 4 filters of 1 × 1. ### To find Direct MACs, two-stage bottleneck MACs and savings. ### Formula — which one and why Each output scalar requires one multiply-accumulate per incoming kernel entry. A bottleneck reduces the input depth of the expensive spatial convolution. One MAC is one multiply-accumulate pair. If counting multiplication and addition separately, the operation count is approximately twice the MAC count. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K_hK_wC_{\mathrm{in}} $$ **Yaad rakhne ka cue:** Spatial kernel se pehle channels kam; dono stages ka cost add. ### Solution — step by step **Step 1: count output positions per channel.** $$ 8(8)=64 $$ **Step 2: count direct work per output scalar.** $$ 3(3)(16)=144 $$ **Step 3: count the full direct layer.** $$ 64(8)(144)=73728 $$ **Step 4: count the 1 × 1 reduction.** $$ 64(4)(1)(1)(16)=4096 $$ **Step 5: count the reduced-depth 3 × 3 layer.** $$ 64(8)(3)(3)(4)=18432 $$ **Step 6: sum both bottleneck stages.** $$ 4096+18432=22528 $$ **Step 7: subtract to obtain savings.** $$ 73728-22528=51200 $$ **Final answer:** Direct = 73,728 MACs; bottleneck = 22,528 MACs; savings = 51,200 MACs. ## Concatenation and residual addition Concatenation and addition combine branches differently. Concatenation places channel groups next to each other and increases the channel count. Residual addition adds matching entries, so the two full shapes must be compatible. $$ C_{concat}=\sum_b C_b $$ $$ y_{res}=F(x)+x $$ ## Inception concatenation and residual addition **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths. ### Explanation and working Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations. ![Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.](https://deep-learning-study-room.vercel.app/assets/figures/v_join_add.svg) Diagram: Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.. Labels: Concatenation and addition produce different shapes | Concatenation and addition produce different shapes | Use height and width 14 × 14 throughout. | CONCATENATE four branches | Channels: 16 + 32 + 8 + 8 | Result: 14 × 14 × 64 | Stack all 64 channels. | ADD two matching tensors | 14 × 14 × 64 + 14 × 14 × 64 | Result: 14 × 14 × 64 | Add each matching pair. | A 14 × 14 × 32 shortcut needs a projection before addition to a 14 × 14 × 64 | branch. Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape. Tap the diagram to enlarge. Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64. A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition. ## ResNet adds a shortcut **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x. $$ y=F(x)+x $$ If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch. ![The two paths must have equal output shape before elementwise addition.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge. ### Effect on gradient propagation For the identity-shortcut formulation, the derivative includes a direct identity contribution. $$ \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I $$ This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients. ### Projection when dimensions differ If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride. $$ y=F(x)+W_sx $$ An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition. **Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum. ### Given x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3]. ### To find Find the three components of x+F(x). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. ### Solution — step by step **Step 1: add the first matching pair.** $$ 1+0.1=1.1 $$ **Step 2: add the second matching pair.** $$ 2+(-0.2)=1.8 $$ **Step 3: add the third matching pair.** $$ 3+0.3=3.3 $$ **Step 4: assemble in the original order.** $$ y=[1.1,1.8,3.3] $$ The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum. ## Drill 44 Inception concatenation and parameter count ## Question **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters. ## Answer — step by step ### Given Input = 10 × 10 × 8. Branch A: 1 × 1 convolution, 4 filters. Branch B: 3 × 3 same convolution, 6 filters. Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters. ### To find Find each branch shape and parameters, then the concatenated shape and total parameters. ### Formula — which one and why **Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches. C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions. $$ C_{\mathrm{out}}=\sum_k C_k $$ $$ P_{\mathrm{total}}=\sum_k P_k $$ **Yaad rakhne ka cue:** Channels jodo; height-width same rakho. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values. $$ C_{\mathrm{combined}}=C_A+C_B+C_C $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Assume one bias per filter. Pooling has no learned parameters. **Step 1: Branch A.** Its 4 filters produce shape 10 × 10 × 4. $$ P_A=(1\times1\times8+1)\times4=9\times4=36 $$ **Step 2: Branch B.** Its 6 filters produce shape 10 × 10 × 6. $$ P_B=(3\times3\times8+1)\times6=73\times6=438 $$ **Step 3: Branch C.** Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2. $$ P_C=0+(1\times1\times8+1)\times2=18 $$ **Step 4: Join the shapes and add the parameters.** $$ C_{\mathrm{combined}}=4+6+2=12 $$ $$ P_{\mathrm{total}}=36+438+18=492 $$ **Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches. ## Drill 45 Projection shortcut ## Question **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters. ## Answer — step by step ### Given Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16. Shortcut kernel = 1 × 1; stride 2; padding 0. Use one bias per projection filter. ### To find Choose the number of filters and find projection output shape and parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ ### Step 1: choose 16 filters One filter creates one channel, so 16 filters give the required 16 output channels. ### Step 2: use the specified 1 × 1 kernel and stride 2 $$ N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8 $$ The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1. ### Step 3: count weights and biases $$ \text{weights}=1\times1\times8\times16=128 $$ $$ \text{biases}=16,\qquad \text{total}=128+16=144 $$ **Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid. ## Drill 38 Residual addition ## Question **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block. ## Answer — step by step ### Given x = [2,−3,1]; F(x) = [−1,1,0.5]. ### To find Find x+F(x), apply ReLU, and compare with ReLU(F(x)). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join. $$ H(x)=x+F(x),\qquad a_j=\max(0,H_j) $$ ### Step 1: add each pair $$ H_1=2+(-1)=1 $$ $$ H_2=-3+1=-2 $$ $$ H_3=1+0.5=1.5 $$ The residual output before ReLU is **[1, −2, 1.5]**. ### Step 2: apply ReLU $$ [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5] $$ ### Step 3: compare with the plain branch Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**. Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation. ## Batch normalization and learned scale/shift Batch normalization first uses batch statistics, then applies a learned scale and shift. Epsilon belongs inside the square root. Inference ordinarily uses stored running statistics. $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ ## Batch normalization inside a residual network **Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training. For one feature with m values in a training batch: $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ The batch mean μ\_B is the centre; batch variance σ²\_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch. **Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.** ### Given x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5. ### To find Batch mean, variance, normalized values and scale-and-shift outputs. ### Formula — which one and why Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale. ### Solution — step by step **Step 1: mean.** $$ \mu_B=\frac{2+4}{2}=3 $$ **Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1. **Step 3: squared deviations and variance.** $$ \sigma_B^2=\frac{(-1)^2+1^2}{2}=\frac{1+1}{2}=1 $$ **Step 4: stable denominator.** $$ d=\sqrt{1+0.00001}\approx1.00000499999 $$ **Step 5: normalize separately.** $$ \hat{x}_1=\frac{-1}{d}\approx-0.99999500004 $$ $$ \hat{x}_2=\frac{1}{d}\approx0.99999500004 $$ **Step 6: scale the first normalized value, then shift it.** $$ y_1=2(-0.99999500004)+0.5\approx-1.49999000008 $$ **Step 7: scale the second normalized value, then shift it.** $$ y_2=2(0.99999500004)+0.5\approx2.49999000008 $$ **Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here. **Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo. ## Global average pooling Global average pooling takes one mean per complete spatial channel. The worked example compares the resulting dense classifier with a classifier after flattening. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ ## Global average pooling: one average for each channel ![Average each channel separately: these are the exact values used in the worked example.](https://deep-learning-study-room.vercel.app/assets/figures/u4_gap.svg) Diagram: Average each channel separately: these are the exact values used in the worked example.. Labels: u4 gap | Global average pooling keeps channels separate | Channel A | 1 | 3 | 5 | 7 | → | 4 | 16 ÷ 4 | Channel B | 2 | 4 | 6 | 8 | → | 5 | 20 ÷ 4 | 2 × 2 × 2 input → two channel means → [4, 5] Average each channel separately: these are the exact values used in the worked example. Tap the diagram to enlarge. **Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier. **Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = $\begin{bmatrix}1 & 3\\5 & 7\end{bmatrix}$ and channel B = $\begin{bmatrix}2 & 4\\6 & 8\end{bmatrix}$ . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.** ### Given H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs. ### To find The GAP result and dense-head parameter count. ### Formula — which one and why Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias. $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Solution — step by step **Step 1: add channel A.** 1 + 3 + 5 + 7 = 16. **Step 2: average channel A.** 16 ÷ 4 = 4. **Step 3: add channel B.** 2 + 4 + 6 + 8 = 20. **Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**. **Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights. **Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero. **Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters. **Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao. ## Top-k error percentage Top-k error counts examples whose correct class is absent from the highest-scoring k predictions. Success counts must not be added twice. $$ E_k=\frac{N-C_k}{N}\times100\% $$ ## Top-1 and top-5 error: what an ImageNet result means **Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models. For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures. **Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.** ### Given Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes. ### To find The percentage of mistakes under each rule. ### Formula — which one and why Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage. $$ E_k=\frac{N-C_k}{N}\times100\% $$ N means total evaluated images; C\_k means images whose correct class appears among the top k predictions. ### Solution — step by step **Step 1: top-1 mistakes.** 100 − 78 = 22. **Step 2: top-1 error percentage.** $$ E_1=\frac{22}{100}\times100\%=22\% $$ **Step 3: top-5 mistakes.** 100 − 92 = 8. **Step 4: top-5 error percentage.** $$ E_5=\frac{8}{100}\times100\%=8\% $$ **Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions. **Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success. --- ## MT2 · Calculator sequence without model guesswork Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c19-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/19/mt2-c19-s2 The reported model name is “Casio FX ES 990”; the exact model is not confirmed. Use your calculator’s actual function labels. On common ES/ES Plus models, exp(x) is usually the shifted ln key, while the EXP/ × 10ˣ key enters scientific notation and is NOT the same as exp(x). Check the exact key labeling on your device. For sigmoid at 2.2, type 1÷(1+exp(−2.2)). Keep the negative sign inside the exponential. Expected 0.900249511. For a negative gradient update, type old−rate × (negative gradient); use parentheses around the negative value. For cross-entropy, use −ln(probability), not log 10. For tanh, use the hyperbolic function or its exponential fraction if that function is difficult to find. Retain 6–9 digits in intermediate results or use stored values; round final answers to 4–6 decimal places unless the paper says otherwise. Work line by line on paper even if the calculator can evaluate the whole expression. Check exp(0) = 1, ln(1) = 0, sigmoid(0) = 0.5, tanh(0) = 0 before relying on the chosen key sequence. --- ## MT2 · What to write if the exam page feels blank Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c19-s3/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/19/mt2-c19-s3 Pause briefly, put both feet down and take one slow breath. Then write only the GIVEN values and what is ASKED. Classify the question using the method map. Write its general formula before inserting numbers. One correct first line is a smaller task than “solve the whole question”. For CNN, draw the shape chain. For back prop, draw arrows and label the loss. For an optimizer, draw columns for step, gradient, state, new parameter. If you remain stuck after a short attempt, leave visible working space and move to a question you can start. Return after collecting those marks. The goal is to reduce the next action to something concrete and familiar. Practice it during mocks so it is available under time pressure. --- ## MT2 · Answer lengths and partial credit habits Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c19-s4/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/19/mt2-c19-s4 For 2 marks, give a precise definition plus the key distinction/example. For 5 marks, give the definition, a labeled diagram or formula, working/principle and a relevant implication. For 10 marks, answer every subpart: architecture or setup, main equations, stepwise explanation, required numerical working, and limitations or justification where asked. Do not pad a numerical with unrelated theory. Always label assumptions when inputs, loss, threshold or matrix orientation are missing. Box the requested final result, include shapes or units, and retain working. An unsupported final number makes it harder to award method credit. Never say an optimizer guarantees global convergence or an architecture always outperforms another unless a specific theorem and its conditions actually apply. --- ## MT2 · Methods to revise before the examination Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c19-s5/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/19/mt2-c19-s5 ### Last-day revision: read in this order This is a revision digest after studying the lessons. Full solutions remain beside each topic and under Solved Questions. If a step here is unclear, return to that worked example; the final day is for recovering the method, not memorizing unexplained final numbers. ### 1. Learning and the neuron Machine learning estimates a model from data. Deep learning uses multiple trainable layers to learn successive representations. Inputs describe a sample; targets are the required answers; predictions are the model's answers. Weights and biases are learned parameters. Learning rate, batch size and architecture are chosen hyperparameters. Training changes parameters using training examples. Validation selects settings/stopping time. Testing evaluates the final choice on held-out data. Overfitting means strong training performance with weak generalization; underfitting means the model has not captured enough of the relationship. For one neuron, multiply each input by its own weight, add all products, add the bias once and apply the activation. A layer repeats this for several receiving neurons. Each weight-matrix row belongs to one receiving neuron. Transpose swaps rows and columns; it does not change signs. $$ z=\sum_iw_ix_i+b,\qquad a=f(z) $$ **Memory cue:** multiply → sum → bias → activate. In a full network, finish one complete layer before the next. ### 2. Activation and loss choices Sigmoid maps to (0,1) and is used for binary-class probabilities. Tanh maps to (−1,1), centred on zero. Both can saturate, producing small derivatives. ReLU retains positive values and returns zero on the negative side; its negative-side gradient is zero. Leaky ReLU retains a specified small negative-side slope. Softmax normalizes a complete list of logits into mutually exclusive class probabilities. $$ \sigma(z)=\frac1{1+e^{-z}},\qquad\sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ $$ \tanh\prime(z)=1-\tanh^2(z) $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ For squared errors, SSE sums squares, half SSE divides that sum by 2, and MSE divides it by the number of scalar predictions. Their derivatives differ. For one-hot categorical cross-entropy, only the correct-class probability contributes. For binary cross-entropy, p means probability of class 1, not automatically the true class. $$ L_{\mathrm{half}}=\frac12\sum_i(\hat y_i-y_i)^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i(\hat y_i-y_i)^2 $$ $$ L_{\mathrm{BCE}}=-[y\ln p+(1-y)\ln(1-p)] $$ $$ L_{\mathrm{CCE}}=-\ln p_{\mathrm{correct}} $$ **Memory cue:** loss ka denominator dekho before differentiating. Sigmoid output is not its derivative. ### 3. Backpropagation: the repeatable method Write the given loss and activation first. Compute all forward sums/activations with the original parameters. Differentiate loss with respect to each output, multiply by the output activation derivative, and call the result the output delta. $$ \delta_o=\frac{\partial L}{\partial o}f_o\prime(z_o) $$ For a hidden neuron, add the outgoing weighted deltas first and then multiply by the hidden activation derivative. Each connection gradient is receiving delta × sending activation. The bias gradient is the receiving delta. Use old weights for all gradients before making the simultaneous update. $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i,\qquad\frac{\partial L}{\partial b_j}=\delta_j $$ $$ \theta_{\mathrm{new}}=\theta-\eta\frac{\partial L}{\partial\theta} $$ **Memory cue:** path mein multiply; paths milne par add; update sab gradients ke baad. For sigmoid+BCE or softmax+CCE, the logit delta simplifies to prediction−target for the stated single-example sum convention. ### 4. Optimizers: identify the stored state Full-batch GD uses the entire dataset for one update; SGD uses one sample; mini-batch uses a subset. An epoch is one pass through the data. With a final incomplete batch included, updates per epoch are ceiling(N/B). | Method | State and calculation order | Detail that changes the answer | | --- | --- | --- | | GD | Current gradient → subtract ηg | Recompute at the new parameter unless gradients are explicitly supplied | | Momentum | Retain βv → subtract ηg → add new velocity to parameter | Signed velocity and gradient-average conventions must not be mixed | | Nesterov | Compute lookahead θ+βv → gradient there → new velocity → update actual θ | Do not add new velocity to the lookahead point | | AdaGrad | Add g² to all-past sum G → divide gradient by √G+ε | G never decreases | | RMSProp | Decay old squared state → add recent g² → normalize | Use the specified decay factor | | Adam | Update signed mean m and squared mean v → correct both → normalize | Carry raw states forward; correction powers use iteration t | $$ v_t=\beta v_{t-1}-\eta g_t,\qquad\theta_t=\theta_{t-1}+v_t $$ $$ G_t=G_{t-1}+g_t^2 $$ $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad\hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Here v means signed velocity in momentum and squared-gradient state in Adam; the method defines the symbol. **Memory cue:** AdaGrad sab squares; RMSProp recent squares; Adam direction + squares + correction. ### 5. Regularization and generalization L1 adds λΣ|w| and a nonzero-weight derivative λsign(w), encouraging sparsity. L2 with the half convention adds (λ/2)Σw² and derivative λw. Without that half, the derivative is 2λw. Regularization gradients add to data gradients before updating the weight. Inverted dropout uses keep probability q: retain a sampled activation with scale 1/q, or replace it by zero. Inference uses the full activation without that training-time rescaling. Drop probability p gives q = 1−p. Early stopping tracks validation performance and restores the best checkpoint under the stated patience rule. $$ a'_i=\frac{m_i a_i}{q},\qquad m_i\in\{0,1\} $$ **Memory cue:** dropout probability aur keep probability alag; L2 mein half hai ya nahi, pehle dekho. ### 6. CNN calculation sequence Use the deep-learning cross-correlation convention unless the question explicitly requires kernel flipping. At each output position, multiply matching input-window and kernel entries, sum over spatial entries and all input channels, and add that filter's bias once. A new filter creates a new output channel. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Apply the shape formula separately to height and width. Pooling changes spatial dimensions but preserves channels; ordinary pooling and flattening have zero learned parameters. Maximum pooling chooses the largest window value; average pooling adds its values and divides by the number of entries. A filter's spatial reuse changes computation, not its parameter count. For a complete architecture, make one row per layer: input shape → operation → output shape → weights → biases → total. Flatten uses the size after the final convolution/pool. **Memory cue:** shape count aur parameter count ko kabhi mix mat karo. ### 7. CNN theory and architectures CNNs use local connectivity and weight sharing, reducing parameters compared with unrestricted dense image mappings. Convolution learns features; activations introduce nonlinearity; pooling summarizes local responses; dense or global-pooling/output stages make the final task prediction. AlexNet: stacked convolutions with ReLU, pooling and dense classification; dropout in dense layers and data augmentation help control overfitting. Inception: parallel spatial scales, 1 × 1 reductions and channel concatenation. ResNet: learns a residual F(x) and adds a shortcut x; use projection when shapes differ. Concatenation adds channels; residual addition requires equal shapes and retains their count. Transfer learning reuses a pretrained feature extractor, replaces the task head, trains the head and optionally fine-tunes selected pretrained layers at a suitable smaller learning rate. Explain classification, detection and segmentation as different outputs, with an appropriate concrete application. ### 8. Match the announced Unit 3–4 mix Revise all theory from optimization, regularization, CNN operations and CNN architectures. For 2 marks, answer exactly the requested definition, distinction or small calculation. For 3 marks, show the complete requested numerical method or explain the mechanism clearly. The paper may combine both. Prepare Unit 3 numericals and especially the two Unit 4 numerical opportunities highlighted by the faculty. Their exact question numbers are not specified. RNN, LSTM and GRU remain in the TEE material; they are not part of this focused MT2 revision. ### The final answer-writing routine For every numerical: full question → Given → To find → governing formula and why → separate numbered steps → final answer. Carry extra calculator precision; round at the end. Label each layer, iteration and connection so the examiner can follow the calculation. For a definition question, state the technical meaning and the requested distinction/example. For a longer mechanism question, include a labelled diagram or equations, explain their sequence, then give the relevant benefits, limitations or comparison. Marks printed beside a historical question are its actual marks; individual steps here are not assigned invented official marks. For this 60-minute MT2, a workable allocation is 5 minutes to read and select, 15 minutes for Q1(a,b), 15 minutes for each selected 3-mark question and 10 minutes to review. Adjust if one calculation needs more working. If stuck, write the givens and governing formula first, complete the solvable parts, then return to the unfinished calculation. --- ## MT2 · Learning and the numbers we adjust Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c20-s0/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/20/mt2-c20-s0 | Term | Meaning | Tiny example | | --- | --- | --- | | Input / feature | A number given to the model | Image pixel intensity | | Sample | One example processed by the model | One labeled image | | Dataset | A collection of examples | 1,000 training images | | Target / label | Correct answer supplied for training | Class 2 is correct | | Prediction | Model's current answer | Class probabilities | | Weight | Learned multiplier on a connection | 0.4 multiplying an input | | Bias | Learned value added to a weighted sum | z = wx + b | | Parameter | A value learned during training | Weight or bias | | Hyperparameter | A setting chosen for training | Learning rate or batch size | | Activation | Transformation of a neuron's raw score | ReLU or sigmoid | | Loss | Numerical measure of prediction error | Mean squared error | | Gradient | Local rate of loss change for parameters | Positive g means locally uphill | | Optimizer | Rule using gradients to update parameters | Momentum or Adam | | Epoch | One pass through the training dataset | All 100 samples processed | | Iteration | One parameter update in these exercises | One mini-batch update | | Inference | Applying the trained model | Predicting a new image's class | --- ## MT2 · Images and sequences Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c20-s1/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/20/mt2-c20-s1 | Term | Meaning | What to keep separate | | --- | --- | --- | | Tensor | Array of numbers with one or more axes | Its shape and its values | | Channel | One feature map within an image-like tensor | Channels versus spatial size | | Kernel / filter | Learned weights sliding over a map | Kernel size versus filter count | | Stride | Step between adjacent window positions | Stride versus kernel width | | Padding | Extra border cells around an input | Per-side P versus total 2P | | Feature map | Responses from a filter across positions | Activations versus parameters | | Pooling | Window-based reduction of feature maps | Maximum versus average | | Flatten | Rearrange H × W × C entries as a vector | Rearrangement has no parameters | | Logit | Raw score before an output activation | Score versus probability | | Hidden state | Recurrent features carried through time | Previous state versus current input | | Shared weights | One learned rule reused at many positions/times | Reuse versus extra parameters | --- ## MT2 · Reading a question precisely Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/section/mt2-c20-s2/ Reader: https://deep-learning-study-room.vercel.app/#mt2/read/20/mt2-c20-s2 “Calculate” asks for arithmetic and a result. Write the formula and substitutions before the final number. “Explain” asks how and why a method works. Give its definition, describe the steps, and connect each step to its purpose. “Compare” asks for matched differences. Discuss the same attributes for both methods, such as stored state, update rule, benefit and limitation. “Draw” asks for a labeled structure. Label inputs, layers or time steps, arrows, activations and the output. A diagram and explanation should describe the same computation. “Design” asks for a justified arrangement. State the task, input and output, chosen layers, activation/loss pair and training procedure. State any assumption needed because the question leaves a value unspecified. --- ## MT2 · MT2 Mock A Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/mocks/a/ Reader: https://deep-learning-study-room.vercel.app/#mt2/mocks/A 60 minutes · 10 marks. Q1(a,b): 2 marks each, compulsory; attempt any two of Q2–Q4: 3 marks each. This uses the shared M2 choice pattern already supplied. The latest faculty message guides the Unit 3–4 theory/numerical mix; it does not fix individual question numbers or numerical topics. Q1 theory · two Unit 4 numericals + Unit 3 theory among the 3-mark choices ## Q1a ## Question **Q1a compulsory (2 marks). Explain the key difference between the AdaGrad and RMSProp accumulators. Why can AdaGrad's effective learning rate become very small during prolonged training?** ## Answer — step by step ### 1. AdaGrad stores a running sum Let gₜ be the current gradient and Gₜ the stored sum of squared gradients. AdaGrad updates that sum as follows: $$ G_t=G_{t-1}+g_t^2 $$ Every added square is nonnegative, so G cannot decrease. All past squared gradients remain in the sum. ### 2. RMSProp stores a decaying average Let sₜ be the stored squared-gradient average and ρ the fraction of the previous average retained. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ RMSProp gradually reduces the influence of older squared gradients. Its accumulator can rise or fall, depending on the new squared gradient. ### 3. Explain the effect on step size AdaGrad's effective learning rate is: $$ \eta_{effective}=\frac{\eta}{\sqrt{G_t}+\epsilon} $$ Here η is the base learning rate and ε is a small positive number that stabilizes division. When G becomes large, its square root becomes large, so dividing by it produces a smaller effective learning rate. RMSProp's decaying average avoids this particular permanent-accumulation problem. **Final answer:** AdaGrad uses a nondecreasing sum of all squared gradients; RMSProp uses a decaying average. AdaGrad's growing denominator can make future updates very small. **Yaad rakho:** AdaGrad sab squares jodta rehta hai; RMSProp purane squares ka asar dheere kam karta hai. ## Q1b ## Question **Q1b compulsory (2 marks). Explain local connectivity and parameter sharing in a CNN. How do they reduce the number of learned parameters compared with a fully connected mapping of an image?** ## Answer — step by step ### 1. Local connectivity Each convolutional output depends on a small spatial neighbourhood of the input, called its receptive field, rather than every pixel in the entire image. The filter spans the input channels within that neighbourhood. This limits the connections used to calculate each output. ### 2. Parameter sharing The same filter weights and bias are reused at every spatial position where that filter is applied. A CNN therefore learns one set of parameters for that filter, rather than a separate set for each image location. ### 3. Concrete image example An edge-detecting filter can identify a vertical boundary near the left of an image and then slide to look for a similar boundary near the right. Both locations use the same learned filter. The network does not need to learn an independent edge detector for every position. ### 4. Comparison with a fully connected mapping A fully connected layer generally learns a separate weight for every input-to-output connection. Convolution reduces this requirement in two ways: local connectivity restricts each output's input region, while parameter sharing reuses one filter across locations. More output positions create more activations and computation, but do not create new parameters for that shared filter. **Final answer:** local connectivity uses a limited input neighbourhood; parameter sharing reuses the filter's learned values across spatial positions. Together they can require far fewer parameters than a fully connected image mapping. **Yaad rakho:** Filter ke numbers ek baar seekhte hain; image par baar-baar wahi use hote hain. ## Q2 ## Question **Q2 optional (3 marks). An input of size 7 × 7 × 3 passes through a convolutional layer with two 3 × 3 filters, stride 2 and padding 1 on each side; then a 2 × 2 pooling layer with stride 2 and no padding; then flattening; then a dense layer with three output units. Find the output shape after every layer and the total trainable parameters. Include one bias per convolutional filter and per dense unit.** ## Answer — step by step ### Given - Input: height 7, width 7, channels 3. - Convolution: kernel 3 × 3, filters 2, stride 2, padding 1 per side. - Pooling: window 2 × 2, stride 2, padding 0. - Flattening followed by 3 dense output units. - Include all specified biases. Pooling and flattening have no learned parameters. ### To find The convolution output, pooling output, flatten length, dense output length and total trainable parameter count. ### Formula — which one and why For either spatial axis, use the output-size formula. N is that axis's input size, K the window size, S its stride and P padding per side. The floor means discard any fractional part because a full window must fit. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Each convolution filter spans all input channels; each dense unit connects to every flattened input. Therefore: $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}},\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flattening simply counts all values in the preceding feature maps: $$ n_{\mathrm{flat}}=HWC $$ ### Solution — step by step ### Step 1: convolution height and width $$ N_{\mathrm{conv}}=\left\lfloor\frac{7+2(1)-3}{2}\right\rfloor+1=3+1=4 $$ The numerator is 7 + 2 − 3 = 6. Divide by stride 2 to get 3, then add 1 to get 4 positions. The same calculation applies to height and width. Two filters create two output channels. **Convolution output: 4 × 4 × 2.** ### Step 2: convolution parameters $$ \text{weights}=3\times3\times3\times2=54 $$ $$ \text{biases}=2,\qquad P_{\mathrm{conv}}=54+2=56 $$ ### Step 3: pooling shape and parameters The pooling input size is now 4, not the original 7. $$ N_{\mathrm{pool}}=\left\lfloor\frac{4-2}{2}\right\rfloor+1=1+1=2 $$ Pooling operates separately on each channel, so the two channels remain two. **Pooling output: 2 × 2 × 2. Pooling parameters: 0.** ### Step 4: flattening $$ n_{\mathrm{flat}}=2\times2\times2=8 $$ There are eight values to send to the dense layer. Flattening rearranges them into a list and learns no parameters. **Flatten output: 8 values. Flatten parameters: 0.** ### Step 5: dense output and parameters Each of the three dense units receives eight inputs and has one bias. $$ \text{weights}=8\times3=24,\quad\text{biases}=3,\quad P_{\mathrm{dense}}=27 $$ **Dense output: 3 values.** ### Step 6: total trainable parameters $$ P_{\mathrm{total}}=56+0+0+27=83 $$ **Final answer:** 4 × 4 × 2 → 2 × 2 × 2 → 8 → 3. **Total: 83 trainable parameters.** **Yaad rakho:** Pehle shape; phir weights. Ek layer ka output hi next layer ka input hai. ## Q3 ## Question **Q3 optional (3 marks). Apply the printed kernel K to input I using the deep-learning cross-correlation convention, with stride 1, no padding and zero bias. Do not flip K. Apply ReLU to the resulting map, followed by 2 × 2 max pooling with stride 2 and no padding. Show every convolution entry and find the final output.** $$ I=\begin{bmatrix}1&2&1\\0&1&3\\2&1&0\end{bmatrix},\quad K=\begin{bmatrix}1&-1\\0&1\end{bmatrix} $$ ## Answer — step by step ### Given - Input I is the printed 3 × 3 matrix; kernel K is the printed 2 × 2 matrix. - Convolution stride 1, padding 0, bias 0; use K in its printed orientation. - Activation: ReLU. - Pooling: max, 2 × 2 window, stride 2, padding 0. ### To find All four convolution values, the ReLU output matrix and the final pooled output. ### Formula — which one and why For each input window, multiply entries at matching positions, then add all products and the bias once. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ X is the input, S is stride, u and v select a kernel entry, and b is bias. This formula uses zero-based array indices; the written solution labels positions from 1 for readability. ReLU is required because the question requests activation after convolution. It replaces a negative value by zero. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Max pooling selects the largest value in its current window V. $$ y_{\max}=\max V $$ ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. ### Step 1: determine the convolution shape For each axis: (3 − 2)/1 + 1 = 2. Therefore calculate a 2 × 2 output matrix. ### Step 2: top-left output, position (1,1) The window is $\begin{bmatrix}1 & 2\\0 & 1\end{bmatrix}$ . Multiply matching cells: 1 × 1 = 1; 2 × (−1) = −2; 0 × 0 = 0; 1 × 1 = 1. Add the products and bias: 1 − 2 + 0 + 1 + 0 = **0**. ### Step 3: top-right output, position (1,2) Move the window one column to the right. The window is $\begin{bmatrix}2 & 1\\1 & 3\end{bmatrix}$ . Multiply matching cells: 2 × 1 = 2; 1 × (−1) = −1; 1 × 0 = 0; 3 × 1 = 3. Add: 2 − 1 + 0 + 3 + 0 = **4**. ### Step 4: bottom-left output, position (2,1) The window is $\begin{bmatrix}0 & 1\\2 & 1\end{bmatrix}$ . Multiply matching cells: 0 × 1 = 0; 1 × (−1) = −1; 2 × 0 = 0; 1 × 1 = 1. Add: 0 − 1 + 0 + 1 + 0 = **0**. ### Step 5: bottom-right output, position (2,2) The window is $\begin{bmatrix}1 & 3\\1 & 0\end{bmatrix}$ . Multiply matching cells: 1 × 1 = 1; 3 × (−1) = −3; 1 × 0 = 0; 0 × 1 = 0. Add: 1 − 3 + 0 + 0 + 0 = **−2**. **Convolution map: $\begin{bmatrix}0 & 4\\0 & -2\end{bmatrix}$ .** ### Step 6: apply ReLU separately to every entry - max(0,0) = 0. - max(0,4) = 4. - max(0,0) = 0. - max(0,−2) = 0. **ReLU map: $\begin{bmatrix}0 & 4\\0 & 0\end{bmatrix}$ .** ### Step 7: max pooling The 2 × 2 pooling window covers all four activated entries. Compare 0, 4, 0 and 0. Their maximum is **4**. **Final answer:** convolution $\begin{bmatrix}0 & 4\\0 & -2\end{bmatrix}$ → ReLU $\begin{bmatrix}0 & 4\\0 & 0\end{bmatrix}$ → pooled output ** $\begin{bmatrix}4\end{bmatrix}$ **, a 1 × 1 map. **Yaad rakho:** Multiply-add se convolution; negative hatao ReLU; biggest uthao max pool. ## Q4 ## Question **Q4 optional (3 marks). What is overfitting? Explain how dropout and early stopping help improve generalization. State which dataset should select the stopping point.** ## Answer — step by step ### 1. Definition and identification Overfitting occurs when a model learns training-specific details or noise that do not generalize to new data. It commonly shows low training error together with substantially higher validation error. Generalization means performing well on new examples drawn from the task's data distribution. ### 2. Dropout During training, dropout randomly sets selected activations to zero. The active subnetwork changes across training passes, discouraging reliance on a fixed combination of hidden units. Under inverted dropout, retained activations are divided by the keep probability to preserve their expected scale. During ordinary inference, the masking is disabled and all units are used without that training-time rescaling. Dropout is a regularizer; it can reduce overfitting, but excessive dropout can cause underfitting. ### 3. Early stopping Train on the training set and monitor performance on a separate **validation set**. Save the model whenever validation loss improves. Stop after the chosen number of consecutive non-improving evaluations, called patience, then restore the saved best model. For example, validation losses 0.8, 0.6, 0.5, 0.53 and 0.55 have their minimum at epoch 3. With patience 2, stop after epoch 5 and restore epoch 3. The stopping epoch and saved-model epoch are different. ![Early stopping retains the model with the best validation performance, not necessarily the final trained model.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: Early stopping retains the model with the best validation performance, not necessarily the final trained model.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. Early stopping retains the model with the best validation performance, not necessarily the final trained model. Tap the diagram to enlarge. **Final answer:** dropout restricts dependence on particular training-time activations; early stopping limits training after validation performance stops improving. Choose the stopping point using validation data and reserve the test set for final evaluation. **Yaad rakho:** Training se seekho, validation se best model chuno, test se final result batao. --- ## MT2 · MT2 Mock B Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/mocks/b/ Reader: https://deep-learning-study-room.vercel.app/#mt2/mocks/B 60 minutes · 10 marks. Q1(a,b): 2 marks each, compulsory; attempt any two of Q2–Q4: 3 marks each. This uses the shared M2 choice pattern already supplied. The latest faculty message guides the Unit 3–4 theory/numerical mix; it does not fix individual question numbers or numerical topics. Q1 theory/numerical combination · Unit 3 theory + numerical mixed question, Unit 4 numerical, Unit 4 theory ## Q1a ## Question **Q1a compulsory (2 marks). Explain the purpose of scaling retained activations in inverted dropout. For activations [1,2,3,4], keep probability 0.5 and mask [1,0,1,0], calculate the activations used during this training pass. A mask value of 1 means keep and 0 means drop.** ## Answer — step by step ### Theory: why scale retained values? Randomly dropping units would reduce the average activation. Inverted dropout divides every retained activation by the keep probability. Averaged over the random masks, this preserves the original activation's expected value. A dropped activation is still zero. ### Given - Original activations a = [1,2,3,4]. - Keep probability q = 0.5, meaning a unit has a 50% chance of being kept. - This pass's supplied mask m = [1,0,1,0]. ### To find The four post-dropout activations used in this training pass. ### Formula — which one and why Multiply the original activation aᵢ by its mask mᵢ, then divide by keep probability q. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ ### Solution — step by step ### Step 1: first activation Mask 1 keeps the value. Multiply 1 × 1 = 1; divide 1 by 0.5 to get **2**. ### Step 2: second activation Mask 0 drops the value. Multiply 0 × 2 = 0; divide 0 by 0.5 to get **0**. ### Step 3: third activation Mask 1 keeps the value. Multiply 1 × 3 = 3; divide 3 by 0.5 to get **6**. ### Step 4: fourth activation Mask 0 drops the value. Multiply 0 × 4 = 0; divide 0 by 0.5 to get **0**. **Final answer: [2,0,6,0].** This is the training-pass result. In ordinary inference with inverted dropout, disable the mask and its scaling. **Yaad rakho:** Mask lagao; bache hue values ko keep probability se divide karo. ## Q1b ## Question **Q1b compulsory (2 marks). Explain a ResNet shortcut. Why must the two tensors being added have matching shapes, and what can be done if their shapes differ?** ## Answer — step by step ### 1. Residual addition A residual block combines a learned transformation F(x) with a shortcut carrying the input x: $$ H(x)=F(x)+x $$ F(x) is the output of the learned residual branch. Adding x provides a direct path for information and gradients, helping optimize deep networks. If the branch produces zero, the addition stage returns x. A post-addition activation, if specified by the architecture, is applied afterward. ### 2. Why matching shapes are needed The addition is entry by entry. Each element in F(x) needs a corresponding element in x, so height, width and channel count must match. For example, a 14 × 14 × 32 tensor cannot be directly added to a 14 × 14 × 64 tensor in a standard residual shortcut. ### 3. How a projection helps A learned projection, commonly a 1 × 1 convolution with an appropriate number of filters and stride, maps the shortcut to the required shape. Then add the projected shortcut and the residual branch. This is addition, not channel concatenation. **Final answer:** a shortcut adds input information to the learned residual branch. Use an identity shortcut when shapes match and a suitable projection when they do not. **Yaad rakho:** Same-size entries jodte hain; mismatch ho toh shortcut ka size pehle match karo. ## Q2 ## Question **Q2 optional (3 marks). (a) Explain why Adam uses bias correction for its initially zero moment estimates. (1 mark) (b) Adam starts with θ₀ = 1 and m₀ = v₀ = 0. Use learning rate η = 0.1, β₁ = 0.9 and β₂ = 0.99. The supplied gradients are g₁ = 2 and g₂ = −1. Perform two bias-corrected updates. Show the raw first and second moments, corrected moments and updated parameter after each iteration. Set ε = 0 only for this nonzero-denominator hand calculation. (2 marks)** ## Answer — step by step ### Part (a): why Adam corrects the moments Adam initializes both moving averages to zero. At early iterations, too little gradient history has accumulated, so the raw estimates are biased toward zero. Dividing mₜ by 1 − β₁ᵗ and vₜ by 1 − β₂ᵗ corrects this initialization bias before the parameter update. The power uses the current iteration number t. As t increases, these denominators approach 1. ### Part (b): numerical solution ### Given - Starting parameter θ₀ = 1. - Starting first moment m₀ = 0 and second moment v₀ = 0. - Learning rate η = 0.1. - Decay factors β₁ = 0.9 and β₂ = 0.99. - Supplied gradients, in order: 2, then −1. - ε = 0 for this exercise; both denominators will be nonzero. ### To find For iterations 1 and 2, calculate m, v, their bias-corrected values and the new parameter θ. ### Formula — which one and why Adam keeps two moving averages: m stores signed gradients and v stores squared gradients. Zero initialization makes these averages initially too small, so divide by the correction factors for the current iteration t. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\quad\hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ The hat marks a corrected value. Carry the **raw** m and v into the next moving-average recurrence, not the corrected values. Keep full calculator precision until the final answer. ### Solution — step by step ### Step 1: iteration 1, raw first moment The current gradient is 2. Retain 90% of the old first moment and add 10% of this gradient. $$ m_1=0.9(0)+0.1(2)=0.2 $$ ### Step 2: iteration 1, raw second moment Square the gradient first: 2² = 4. Retain 99% of the old second moment and add 1% of that square. $$ v_1=0.99(0)+0.01(2)^2=0.04 $$ ### Step 3: iteration 1, bias correction Use t = 1. The correction denominators are 1 − 0.9 = 0.1 and 1 − 0.99 = 0.01. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.04}{1-0.99^{1}}=4 $$ ### Step 4: iteration 1, parameter update The square root of the corrected second moment is √4 = 2. The step is 0.1 × 2 ÷ 2 = 0.1. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ ### Step 5: iteration 2, raw first moment Use supplied gradient −1, previous raw m = 0.2 and previous raw v = 0.04. $$ m_2=0.9(0.2)+0.1(-1)=0.08 $$ The two contributions are 0.18 and −0.1; their sum is 0.08. ### Step 6: iteration 2, raw second moment The square of −1 is +1. $$ v_2=0.99(0.04)+0.01(-1)^2=0.0496 $$ The two contributions are 0.0396 and 0.01; their sum is 0.0496. ### Step 7: iteration 2, bias correction Use t = 2. The correction denominators are 1 − 0.9² = 0.19 and 1 − 0.99² = 0.0199. $$ \widehat m_2=\frac{0.08}{1-0.9^{2}}=0.421052632 $$ $$ \widehat v_2=\frac{0.0496}{1-0.99^{2}}=2.492462312 $$ ### Step 8: iteration 2, parameter update Start from the updated parameter 0.9. Take the square root of the corrected second moment, divide the corrected first moment by it, multiply by 0.1 and subtract. $$ \theta_2=0.9-\frac{0.1\times(0.421052632)}{\sqrt{2.492462312}}=0.873330057 $$ The displayed decimals are rounded; evaluate using the unrounded fractions when possible. **Final answer:** θ₁ = **0.9** and θ₂ ≈ **0.873330057**. The second supplied gradient is negative, but the first-moment estimate is still positive because it retains part of the previous positive gradient. **Yaad rakho:** Do raw memories banao → current t se hats nikalo → update; next round mein raw memories le jao. ## Q3 ## Question **Q3 optional (3 marks). A 16 × 16 × 64 feature map is processed by 32 filters of size 3 × 3 using stride 1 and same padding. Compare the direct convolution's MAC count with a bottleneck that first applies eight 1 × 1 filters, then the same 32-filter 3 × 3 convolution. Both stages keep spatial size 16 × 16. Find both costs, the absolute saving and percentage saving. Count one multiply–accumulate as one MAC and omit bias additions.** ## Answer — step by step ### Given - Input map: 16 × 16 with 64 channels. - Direct layer: 32 filters, each 3 × 3, receiving 64 channels. - Bottleneck stage 1: eight 1 × 1 filters, receiving 64 channels. - Bottleneck stage 2: 32 filters, each 3 × 3, now receiving only 8 channels. - Every layer has output height and width 16; ignore bias additions. ### To find Direct MACs, both bottleneck-stage MACs, their total, absolute saving and saving percentage. ### Formula — which one and why Each output entry uses K²Cᵢₙ weight–input pairs. There are HₒᵤₜWₒᵤₜCₒᵤₜ output entries. Multiply these counts: $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ The bottleneck has two layers, so calculate each layer's work and add them. Saving is direct work minus reduced work, expressed as a fraction of direct work. $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ ### Solution — step by step ### Step 1: direct convolution Each output entry uses 3 × 3 × 64 = **576** weight–input pairs. There are 16 × 16 × 32 = **8,192** output entries. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 bottleneck reduction Each reduction output uses 1 × 1 × 64 = **64** weight–input pairs. There are 16 × 16 × 8 = **2,048** reduction outputs. $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ The new map is **16 × 16 × 8**. This changes the next layer's input channels to 8. ### Step 3: following 3 × 3 convolution Each output now uses 3 × 3 × 8 = **72** weight–input pairs, instead of 576. The number of outputs remains 16 × 16 × 32 = **8,192**. $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: total bottleneck cost Include both stages; do not compare the direct layer against only the second stage. $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: absolute saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ ### Step 6: percentage saving Divide the saving by the original direct cost, then multiply by 100. $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct **4,718,592 MACs**; bottleneck **720,896 MACs**; saving **3,997,696 MACs**, approximately **84.72%**. Both routes end in a 16 × 16 × 32 output, though they are different learned transformations. **Yaad rakho:** 1 × 1 se channels kam; expensive 3 × 3 ka kaam kam. Dono stages ka cost jodna hai. ## Q4 ## Question **Q4 optional (3 marks). Explain transfer learning for image classification with limited labelled data. Distinguish feature extraction from fine-tuning and describe a suitable training procedure.** ## Answer — step by step ### 1. Definition Transfer learning reuses knowledge learned on a source task to help solve a target task. For image classification, a pretrained CNN can provide visual features that a new classifier uses for the target classes. This can reduce the amount of target data and training needed, provided the source features are useful for the target task. ### 2. Feature extraction and fine-tuning | Method | Which parameters learn? | Main effect | | --- | --- | --- | | Feature extraction | Keep pretrained backbone weights fixed; train a new classification head | Learn to classify using fixed pretrained features | | Fine-tuning | Update selected pretrained layers as well as the new head | Adapt representations to the target task | Freezing a layer stops its parameter updates; it does not remove its stored weights. Fine-tuning typically exposes more trainable parameters and therefore needs particular care with small datasets. ![Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update. Tap the diagram to enlarge. ### 3. Training procedure 1. Create separate training, validation and test splits for the target problem. 2. Choose a suitable pretrained CNN and apply its required input size and preprocessing. 3. Replace the original classifier with a new head matching the target classes. 4. Freeze the backbone and train the new head on the training set. 5. If validation results justify it, unfreeze selected later layers and continue with an appropriately smaller learning rate. 6. Select the best checkpoint using validation performance; evaluate the final model on the held-out test set. ### 4. Concrete example For a small dataset containing cats and dogs, reuse a pretrained image backbone and replace its original classifier with a binary-classification head. First learn the new head with the backbone frozen. Later, selected high-level layers may be fine-tuned if this improves validation performance. **Final answer:** feature extraction learns a new classifier over fixed features; fine-tuning also adapts selected pretrained features. The correct choice depends on target data quantity, similarity to the source data and validation results. **Yaad rakho:** Freeze mein purane features fixed; fine-tune mein selected purane features bhi seekhte hain. --- ## MT2 · TEE paper rehearsal Canonical: https://deep-learning-study-room.vercel.app/read/dl/mt2/mocks/tee/ Reader: https://deep-learning-study-room.vercel.app/#mt2/mocks/TEE 3 hours · 100 marks. Q1 compulsory + any four from Q2–Q7. Historical paper rehearsal. ## F25 Q1c ## Question **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. ## F25 Q1d ## Question **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. ## F25 Q2b ## Question **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs. ## F25 Q3b ## Question **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. ## F25 Q4a ## Question **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters. ## Answer — step by step ### Given N = 1,000; mini-batch size = 100. J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9. Use signed velocity. ### To find Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Step 1: count full-batch updates.** All 1,000 samples contribute to one update: 1 per epoch. **Step 2: count stochastic updates.** One update per sample: 1,000 per epoch. **Step 3: count mini-batches.** $$ 1000/100=10 $$ Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ. ### Momentum iteration 1 **Step 1: evaluate the gradient at the current parameter.** $$ g_1=1 $$ **Step 2: retain the previous displacement.** $$ 0.9(0)=0 $$ **Step 3: scale the current gradient.** $$ 0.1(1)=0.1 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_1=0-0.1=-0.1 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_1=1+(-0.1)=0.9 $$ ### Momentum iteration 2 **Step 1: evaluate the gradient at the current parameter.** $$ g_2=0.9 $$ **Step 2: retain the previous displacement.** $$ 0.9(-0.1)=-0.09 $$ **Step 3: scale the current gradient.** $$ 0.1(0.9)=0.09 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_2=-0.09-0.09=-0.18 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_2=0.9+(-0.18)=0.72 $$ **Hyperparameters:** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update. ## F25 Q5b ## Question **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD. ## Answer — step by step AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise: $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change. Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864. Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average. ## F25 Q6a ## Question **Question (10 marks).** Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. Given matrices Input A Input A | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Stride 2; P = 0 (no padding). ## Answer — step by step ### Given Input 128 × 128 × 3. Conv K = 3, filters = 16, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 3, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Dense 240 units. Dense 120 units. Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. ### To find Find every layer shape, its parameter count and the network total. Find every convolution output cell. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**128 × 128 × 3**height × width × channels → Convolution**16 filters · 3 × 3 · S = 1 · P = 0** → Output shape**126 × 126 × 16**height × width × channels Weights**432** + Biases**16** = Trainable parameters**448** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **128 × 128 × 3**. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{128+2(0)-3}{1}\right\rfloor+1=126 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **126 × 126 × 16**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times3\times16=432 $$ $$ N_{\mathrm{biases}}=16 $$ $$ P_1=432+16=448 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**126 × 126 × 16**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**63 × 63 × 16**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **126 × 126 × 16**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{126-2}{2}\right\rfloor+1=63 $$ Pooling acts independently on each channel. The output is **63 × 63 × 16**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**63 × 63 × 16**height × width × channels → Convolution**32 filters · 3 × 3 · S = 1 · P = 0** → Output shape**61 × 61 × 32**height × width × channels Weights**4,608** + Biases**32** = Trainable parameters**4,640** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **63 × 63 × 16**. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{63+2(0)-3}{1}\right\rfloor+1=61 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **61 × 61 × 32**. **3. Count trainable values.** Each filter receives all 16 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times16\times32=4{,}608 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_3=4{,}608+32=4{,}640 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**61 × 61 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**30 × 30 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **61 × 61 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{61-2}{2}\right\rfloor+1=30 $$ Pooling acts independently on each channel. The output is **30 × 30 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 5 — flatten Layer 5: what changes and what is learned Input shape**30 × 30 × 32**height × width × channels → Flatten**Rearrange activations** → Output shape**28,800**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **30 × 30 × 32**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=30\times30\times32=28{,}800 $$ Output: a vector with **28,800 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 6 — fully connected layer Layer 6: what changes and what is learned Input shape**28,800**input features → Dense layer**240 neurons** → Output shape**240**output features Weights**6,912,000** + Biases**240** = Trainable parameters**6,912,240** Every output neuron receives every input feature, with one bias per output neuron. Input: **28,800 features**. Output: **240 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=28{,}800\times240=6{,}912{,}000 $$ $$ N_{\mathrm{biases}}=240 $$ $$ P_6=6{,}912{,}000+240=6{,}912{,}240 $$ ### Layer 7 — fully connected layer Layer 7: what changes and what is learned Input shape**240**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**28,800** + Biases**120** = Trainable parameters**28,920** Every output neuron receives every input feature, with one bias per output neuron. Input: **240 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=240\times120=28{,}800 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_7=28{,}800+120=28{,}920 $$ ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**120**input features → Dense layer**5 neurons** → Output shape**5**output features Weights**600** + Biases**5** = Trainable parameters**605** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **5 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times5=600 $$ $$ N_{\mathrm{biases}}=5 $$ $$ P_8=600+5=605 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{128+2 \times 0-3}{1}\right\rfloor+1$ | 126 x 126 x 16 | 432 | 16 | 448 | | 2 pool | $\left\lfloor\frac{126-2}{2}\right\rfloor+1$ | 63 x 63 x 16 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{63+2 \times 0-3}{1}\right\rfloor+1$ | 61 x 61 x 32 | 4,608 | 32 | 4,640 | | 4 pool | $\left\lfloor\frac{61-2}{2}\right\rfloor+1$ | 30 x 30 x 32 | 0 | 0 | 0 | | 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 | | 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 | | 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 | | 8 dense | 120 → 5 | 5 | 600 | 5 | 605 | **Parameter substitutions, in layer order:** Layer 1: (3 x 3 x 3 + 1) x 16 = 448. Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640. Layer 6: (28800 + 1) x 240 = 6,912,240. Layer 7: (240 + 1) x 120 = 28,920. Layer 8: (120 + 1) x 5 = 605. **Total:** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. **Separate convolution calculation:** **Step 1 shape:** $\left\lfloor\frac{5+2 \times 0-3}{2}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −2 | | 4 | 0 | −4 | | 3 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−2) = 1** Row 2**4 + 0 + (−4) = 0** Row 3**3 + 0 + (−1) = 2** **1 + 0 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(1)\times(0)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(2)\times(-2)=0 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(0)+(2)=3 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 2 | 0 | −1 | | 4 | 0 | 0 | | 1 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 + (−1) = 1** Row 2**4 + 0 + 0 = 4** Row 3**1 + 0 + (−1) = 0** **1 + 4 + 0 = 5** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(2)\times(0)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(0)\times(-2)=4 $$ **Window row 3.** $$ R_3=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(1)+(4)+(0)=5 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −1 | | 0 | 0 | 0 | | 1 | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−1) = 2** Row 2**0 + 0 + 0 = 0** Row 3**1 + 0 + 0 = 1** **2 + 0 + 1 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Window row 2.** $$ R_2=(0)\times(2)+(3)\times(0)+(0)\times(-2)=0 $$ **Window row 3.** $$ R_3=(1)\times(1)+(1)\times(0)+(0)\times(-1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(2)+(0)+(1)=3 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 1 | 0 | −1 | | 0 | 0 | −4 | | 0 | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 + (−1) = 0** Row 2**0 + 0 + (−4) = −4** Row 3**0 + 0 + (−3) = −3** **0 + (−4) + (−3) = −7** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0)\times(2)+(1)\times(0)+(2)\times(-2)=-4 $$ **Window row 3.** $$ R_3=(0)\times(1)+(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(0)+(-4)+(-3)=-7 $$ **Final output:** $\begin{bmatrix}3 & 5\\3 & -7\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ## F25 Q7b ## Question **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages. ## Answer — step by step A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification. ![One filter is reused at successive spatial positions](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: One filter is reused at successive spatial positions. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. One filter is reused at successive spatial positions Tap the diagram to enlarge. Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities. Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation. For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions. --- ## TEE · Mean, variance and standard deviation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/1/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/1 The mean is a measure of location; variance and standard deviation quantify dispersion. Distinguish population and sample conventions before calculating these statistics. $$ \mu=\frac{1}{n}\sum_i x_i $$ $$ v=\frac{1}{n}\sum_i(x_i-\mu)^2 $$ $$ s=\sqrt{v} $$ ## Mean, variance and standard deviation from ordinary arithmetic **Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i,\qquad \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2,\qquad \sigma=\sqrt{\sigma^2} $$ These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable. **Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].** ### Given Observations = [3,3,7,7]; N = 4. Use population variance (divide by N). ### To find Find the population mean μ, variance σ² and standard deviation σ. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3. ### Population mean and spread: [1, 2, 3] **Question. For the population [1, 2, 3], find mean, population variance and standard deviation.** ### Given Values = [1, 2, 3]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 1+2+3=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (1-2)=-1 $$ $$ (-1)^2=1 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (3-2)=1 $$ $$ (1)^2=1 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{1+0+1}{3}\approx0.666666667 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.666666667}\approx0.816496581 $$ **Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581. ### Population mean and spread: [2, 2, 2] **Question. For the population [2, 2, 2], find mean, population variance and standard deviation.** ### Given Values = [2, 2, 2]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 2+2+2=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{0+0+0}{3}\approx0.000000000 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.000000000}\approx0.000000000 $$ **Final answer:** μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation. --- ## TEE · Standardization Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/2/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/2 Standardization has a meaning before it has a formula: measure how far an input is from its training mean, then express that distance in standard-deviation units. The worked question below calculates the mean and standard deviation first, so neither number appears without explanation. $$ x_{scaled}=\frac{x-\mu_{train}}{s_{train}} $$ ## Standardization: what the answer 1 actually means **Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units. $$ z=\frac{x-\mu_{\mathrm{train}}}{\sigma_{\mathrm{train}}},\qquad \sigma_{\mathrm{train}}>0 $$ Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1]. **Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.** ### Given Training observations = [3,3,7,7]. New observation = 9. Estimate the population standard deviation from the training values. ### To find Find the training mean and standard deviation; standardize all four training values and the new observation. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. **Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units. x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Step 8: standardize each original training observation.** $$ \frac{3-5}{2}=-1 $$ $$ \frac{7-5}{2}=1 $$ The repeated observations give [-1, −1, 1, 1]. **Step 9: subtract the training mean from the new observation.** $$ 9-5=4 $$ **Step 10: divide by the training standard deviation.** $$ z_{\mathrm{new}}=\frac42=2 $$ **Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy. ### Standardization with x = 3 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.** ### Given x = 3; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=3-5=-2 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{-2}{2}=-1 $$ **Step 3: interpret sign and size.** The value is 1 standard-deviation unit(s) below the training mean. **Final answer:** z = -1. Use the same training statistics when transforming a new test input. ### Standardization with x = 5 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.** ### Given x = 5; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=5-5=0 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{0}{2}=0 $$ **Step 3: interpret sign and size.** The value is 0 standard-deviation unit(s) away from the training mean. **Final answer:** z = 0. Use the same training statistics when transforming a new test input. ### Standardization with x = 9 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.** ### Given x = 9; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=9-5=4 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{4}{2}=2 $$ **Step 3: interpret sign and size.** The value is 2 standard-deviation unit(s) above the training mean. **Final answer:** z = 2. Use the same training statistics when transforming a new test input. --- ## TEE · Power, constant and sum derivatives Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/3/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/3 Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization. $$ \frac{d}{dx}(cx^n)=cnx^{n-1} $$ $$ \frac{d}{dx}c=0 $$ $$ (f+g)\prime=f\prime+g\prime $$ ## Derivatives without a calculus course **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector. $$ \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix} $$ The rules needed for the supplied neural-network numericals are: | Function or operation | Derivative | | --- | --- | | Constant c | 0 | | xⁿ | nxⁿ⁻¹ | | c f(x) | c f′(x) | | f(x) + g(x) | f′(x) + g′(x) | | f(x)g(x) | f′(x)g(x) + f(x)g′(x) | | eˣ | eˣ | | ln x, x > 0 | 1/x | | f(g(x)) | f′(g(x))g′(x) | ### A parameter derivative is not an input derivative For z = wx + b, the variable named in the denominator determines which derivative is required. $$ \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1 $$ At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities. ### Gradient direction For a small parameter displacement Δθ, first-order Taylor expansion gives: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large. ### Differentiate before substituting: 3w^2+5 **Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.** ### Given J(w) = 3w^2+5; w = 2. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(2)=3(2)^2+5=17 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=3(2)w^{2-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(2)=3(2)(2)^{1}=12 $$ **Final answer:** J = 17; derivative = 12. ### Differentiate before substituting: 0.7w^3+0 **Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.** ### Given J(w) = 0.7w^3+0; w = 1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(1)=0.7(1)^3+0=0.7 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=0.7(3)w^{3-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(1)=0.7(3)(1)^{2}=2.1 $$ **Final answer:** J = 0.7; derivative = 2.1. ### Differentiate before substituting: 2w^1+4 **Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.** ### Given J(w) = 2w^1+4; w = -1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(-1)=2(-1)^1+4=2 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=2(1)w^{1-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(-1)=2(1)(-1)^{0}=2 $$ **Final answer:** J = 2; derivative = 2. --- ## TEE · Chain rule Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/4/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/4 The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ ## The chain rule is multiplying linked effects **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions. ### First understand why the rates multiply Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency. The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w. **Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add. ### Chain-rule example 1: two linear links **Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.** ### Given w = 1; u = 3w+1; v = 2u. ### To find The two function values and the rate dv/dw. ### Formula — which one and why v depends on w through u. Both links must be included, so multiply the two local derivatives. du/dw is the slope of 3w+1. dv/du is the slope of 2u. $$ \frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw} $$ **Yaad rakhne ka cue:** Andar ka slope × bahar ka slope. ### Solution — step by step **Step 1: calculate the intermediate value.** $$ u=3(1)+1=4 $$ **Step 2: calculate the final value.** $$ v=2(4)=8 $$ **Step 3: differentiate the first link.** $$ \frac{du}{dw}=3 $$ **Step 4: differentiate the second link.** $$ \frac{dv}{du}=2 $$ **Step 5: multiply along the path.** $$ \frac{dv}{dw}=2(3)=6 $$ **Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different. ### 2. One output: calculate both parameter gradients **Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1. ### Given Input x = 2; target y = 4. Old weight w = 1; old bias b = 0. Linear prediction z = wx+b; loss L = ½(z−y)². Learning rate η = 0.1; update both parameters once. ### To find Find the loss gradient for w and for b, then w\_new and b\_new. Finally evaluate the changed prediction and loss. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ![The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.](https://deep-learning-study-room.vercel.app/assets/figures/chain_linear_revision.svg) Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1). The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient. Tap the diagram to enlarge. **Step 1 — evaluate the original model.** Use the given parameters before applying any update. $$ z=wx+b=1(2)+0=2 $$ $$ e=z-y=2-4=-2 $$ $$ L=\frac12e^2=\frac12(-2)^2=2 $$ Forward pass: three distinct quantities Prediction z**2**1 × 2 + 0 → Error e**−2**Prediction 2 − target 4 → Half squared loss**2**½ × (−2)² The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide. **Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it. $$ \frac{\partial L}{\partial e}=\frac12(2e)=e $$ $$ \frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1 $$ $$ \delta_z=\frac{\partial L}{\partial z}=e(1)=-2 $$ The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection. **Step 3 — differentiate the neuron's affine expression.** When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1. $$ \frac{\partial z}{\partial w}=x=2 $$ $$ \frac{\partial z}{\partial b}=1 $$ **Step 4 — multiply along each parameter's path.** $$ g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4 $$ $$ g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2 $$ Backward pass: a separate derivative for each parameter Shared loss derivative δz**−2**Both parameter paths start here. Weight path: multiply by x = 2**−4**∂L/∂w = (−2) × 2 Bias path: multiply by 1**−2**∂L/∂b = (−2) × 1 Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate. **Step 5 — update both parameters simultaneously.** Gradient descent subtracts the learning rate times the corresponding gradient. $$ w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4 $$ $$ b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2 $$ One simultaneous parameter update Weight**1 → 1.4**Subtract 0.1 × (−4) Bias**0 → 0.2**Subtract 0.1 × (−2) Both updates use gradients from the same original forward pass. Re-evaluate the model afterward. Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure. **Step 6 — perform a new forward pass to evaluate the update.** $$ z_{\mathrm{new}}=1.4(2)+0.2=3 $$ $$ L_{\mathrm{new}}=\frac12(3-4)^2=0.5 $$ **Final answer:** weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3. ### 3. Two outputs: why backward contributions are added A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward. **Question.** A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed. ### Given h = wx+b; outputs o₁ = 2h and o₂ = −h. x = 2; w = 0.5; b = 0.1; targets = [1,0]. Loss = half the sum of the two squared output errors. Output multipliers 2 and −1 stay fixed. ### To find Find total loss and its derivatives with respect to h, w and b. ### Formula — which one and why **Chain rule with more than one path.** The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b. oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h} $$ $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h} $$ **Yaad rakhne ka cue:** Har raaste ka effect pehle multiply, junction par sab add. ### Solution — step by step ![The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.](https://deep-learning-study-room.vercel.app/assets/figures/chain_branch_revision.svg) Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5 The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias. Tap the diagram to enlarge. **Step 1 — forward values.** $$ h=0.5(2)+0.1=1.1 $$ $$ o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1 $$ $$ L_1=\frac12(2.2-1)^2=0.72 $$ $$ L_2=\frac12(-1.1-0)^2=0.605 $$ $$ L=L_1+L_2=0.72+0.605=1.325 $$ **Step 2 — output derivatives.** Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output. $$ \frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2 $$ $$ \frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1 $$ **Step 3 — calculate the contribution of each complete path.** The derivative of 2h with respect to h is 2; the derivative of −h is −1. $$ \frac{\partial L_1}{\partial h}=(1.2)(2)=2.4 $$ $$ \frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1 $$ **Step 4 — add the two contributions at the shared hidden value.** $$ \frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h} $$ $$ \frac{\partial L}{\partial h}=2.4+1.1=3.5 $$ Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path. **Step 5 — continue to the hidden weight and bias.** Since h = wx + b, its derivatives with respect to w and b are x and 1. $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7 $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5 $$ **Final answer:** L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation. ### Chain-rule example 4: a square after a linear link **Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.** ### Given w = 1; z = 3w+1; L = z². ### To find z, L and the derivative of L with respect to w. ### Formula — which one and why The square acts on z, not directly on w. Differentiate the square first and include the change of z with w. dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ $$ \frac{d(z^2)}{dz}=2z $$ **Yaad rakhne ka cue:** Square ka 2z; andar ka 3; dono multiply. ### Solution — step by step **Step 1: evaluate the inner expression.** $$ z=3(1)+1=4 $$ **Step 2: evaluate the loss expression.** $$ L=4^2=16 $$ **Step 3: differentiate the square and use z = 4.** $$ \frac{dL}{dz}=2z=2(4)=8 $$ **Step 4: differentiate the inner expression.** $$ \frac{dz}{dw}=3 $$ **Step 5: multiply, because the changes occur in sequence.** $$ \frac{dL}{dw}=8(3)=24 $$ **Final answer:** z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. --- ## TEE · Dot product and a neuron Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/5/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/5 A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions. $$ z=\sum_i w_i x_i+b $$ $$ a=f(z) $$ ## Model answer: explain an artificial neuron ## Question **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example. ## Answer — step by step An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers. ![Follow each input through its weight, add one bias at the receiving neuron, then apply activation.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Follow each input through its weight, add one bias at the receiving neuron, then apply activation. Tap the diagram to enlarge. ### Formula and symbols $$ z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z) $$ | Part | Meaning | What it does | | --- | --- | --- | | x₁, x₂, … | Input features | Supply the information for this calculation | | w₁, w₂, … | Weights | Control the contribution of each matching input | | b | Bias | Shifts the weighted sum independently of the inputs | | z | Net input | The number before activation | | f | Activation rule | Transforms z, often nonlinearly | | a | Activated output | The number sent onward | ### Worked ReLU neuron example **Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU. ### To find Preactivation z and activated output a. ### Formula — which one and why Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total. z means before activation; a means after activation. $$ z=w_1x_1+w_2x_2+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Product, product, sum, bias, phir activation. ### Solution — step by step **Step 1: multiply the first pair.** $$ 0.5(2)=1 $$ **Step 2: multiply the second pair.** $$ (-0.2)(1)=-0.2 $$ **Step 3: add both contributions.** $$ 1+(-0.2)=0.8 $$ **Step 4: add the bias once.** $$ z=0.8+0.1=0.9 $$ **Step 5: apply ReLU.** $$ a=\max(0,0.9)=0.9 $$ **Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged. The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once. ## What one artificial neuron does **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output. ### What goes into a neuron? An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive. The **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**. **Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output. ### Mathematical model and notation $$ z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b $$ $$ a=f(z) $$ | Symbol | Meaning and role | | --- | --- | | xᵢ | Input component i; the neuron receives d such components | | wᵢ | Trainable weight multiplying that input component | | b | Trainable bias added once to this neuron's total | | z | Preactivation: the result before applying the activation function | | f | Chosen activation function, such as sigmoid or ReLU | | a | Activation: the neuron's output after applying f | The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs. ### Why both a bias and an activation are needed The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin. The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values. ![Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron. Tap the diagram to enlarge. ### Worked numerical **Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output. ### Given Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5]. Bias = 0.1; activation = sigmoid. ### To find Find the preactivation z, then the sigmoid output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. ### From one neuron to a layer With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron: $$ z=Wx+b,\qquad a=f(z) $$ Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together. ### Neuron example 2: ReLU output **Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2]. Weights = [3]; bias = 1. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(3)=6 $$ **Step 2: add the input contributions.** $$ s=(6)=6 $$ **Step 3: add the bias once.** $$ z=6+(1)=7 $$ **Step 4: apply the stated activation.** $$ a=\max(0,7)=7 $$ **Final answer:** z = 7; a = 7. ### Neuron example 3: ReLU output **Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.** ### Given Inputs = [2, 1]. Weights = [-1, 0.5]; bias = 0.2. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(-1)=-2 $$ **Step 2: multiply input 2 by its own weight.** $$ (1)(0.5)=0.5 $$ **Step 3: add the input contributions.** $$ s=(-2)+(0.5)=-1.5 $$ **Step 4: add the bias once.** $$ z=-1.5+(0.2)=-1.3 $$ **Step 5: apply the stated activation.** $$ a=\max(0,-1.3)=0 $$ **Final answer:** z = -1.3; a = 0. ### Neuron example 4: sigmoid output **Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.** ### Given Inputs = [1, 2]. Weights = [0.2, -0.1]; bias = 0. Activation = sigmoid. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (1)(0.2)=0.2 $$ **Step 2: multiply input 2 by its own weight.** $$ (2)(-0.1)=-0.2 $$ **Step 3: add the input contributions.** $$ s=(0.2)+(-0.2)=0 $$ **Step 4: add the bias once.** $$ z=0+(0)=0 $$ **Step 5: apply the stated activation.** $$ a=\frac1{1+e^{0}}=\frac1{1+1}=0.5 $$ **Final answer:** z = 0; a = 0.5. ## Drill 4 Dot product and bias ## Question **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU. ## Answer — step by step ### Given Inputs x = [2, −1, 0.5]. Weights w = [0.3, 0.4, −0.2]; bias b = 0.1. Activation: ReLU. ### To find Find the weighted sum z and ReLU output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero. $$ z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z) $$ Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output. ### Step 1: multiply the matching pairs $$ w_1x_1=0.3\times2=0.6 $$ $$ w_2x_2=0.4\times(-1)=-0.4 $$ $$ w_3x_3=(-0.2)\times0.5=-0.1 $$ ### Step 2: add all products and add bias once $$ z=0.6+(-0.4)+(-0.1)+0.1 $$ $$ z=0.2-0.1+0.1=0.2 $$ ### Step 3: apply the activation $$ a=\max(0,0.2)=0.2 $$ **Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive. --- ## TEE · Matrix multiplication Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/6/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/6 Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias. $$ (AB)_{ij}=\sum_k A_{ik}B_{kj} $$ ## Matrix multiplication: every row has a job ### Row, column, vector and matrix — read these first A **row** runs horizontally, from left to right. A **column** runs vertically, from top to bottom. A **matrix** is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows. A **vector** is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors. ### Transpose: rows become columns The superscript T means **transpose**, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m. $$ [A^T]_{ij}=A_{ji} $$ The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1. **Yaad rakhne ka cue:** transpose = row ko column bana; number aur sign ko haath mat laga. ### Transpose example 1 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}2&1&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 1 row(s) and 3 column(s), so its size is 1 × 3. **Step 2: move original row 1 into new column 1.** Copy [2, 1, 3] down column 1, in the same order. **Step 3: write the transposed matrix.** $$ A^T=\begin{bmatrix}2\\1\\3\end{bmatrix} $$ **Step 4: state its new size.** The result has 3 row(s) and 1 column(s): 3 × 1. **Final answer:** Aᵀ has size 3 × 1. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 1 rows × 3 columns A: 1 rows × 3 columns | | | | | --- | --- | --- | | 2 | 1 | 3 | Read each colored row left to right. Aᵀ: 3 rows × 1 columns Aᵀ: 3 rows × 1 columns | | | --- | | 2 | | 1 | | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 2 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 3 column(s), so its size is 2 × 3. **Step 2: move original row 1 into new column 1.** Copy [1, 2, 3] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4, 5, 6] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix} $$ **Step 5: state its new size.** The result has 3 row(s) and 2 column(s): 3 × 2. **Final answer:** Aᵀ has size 3 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 3 columns A: 2 rows × 3 columns | | | | | --- | --- | --- | | 1 | 2 | 3 | | 4 | 5 | 6 | Read each colored row left to right. Aᵀ: 3 rows × 2 columns Aᵀ: 3 rows × 2 columns | | | | --- | --- | | 1 | 4 | | 2 | 5 | | 3 | 6 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 3 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&-2\\0&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 2 column(s), so its size is 2 × 2. **Step 2: move original row 1 into new column 1.** Copy [1, -2] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [0, 3] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix} $$ **Step 5: state its new size.** The result has 2 row(s) and 2 column(s): 2 × 2. **Final answer:** Aᵀ has size 2 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 2 columns A: 2 rows × 2 columns | | | | --- | --- | | 1 | −2 | | 0 | 3 | Read each colored row left to right. Aᵀ: 2 rows × 2 columns Aᵀ: 2 rows × 2 columns | | | | --- | --- | | 1 | 0 | | −2 | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 4 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}-1\\4\\2\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 3 row(s) and 1 column(s), so its size is 3 × 1. **Step 2: move original row 1 into new column 1.** Copy [-1] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4] down column 2, in the same order. **Step 4: move original row 3 into new column 3.** Copy [2] down column 3, in the same order. **Step 5: write the transposed matrix.** $$ A^T=\begin{bmatrix}-1&4&2\end{bmatrix} $$ **Step 6: state its new size.** The result has 1 row(s) and 3 column(s): 1 × 3. **Final answer:** Aᵀ has size 1 × 3. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 3 rows × 1 columns A: 3 rows × 1 columns | | | --- | | −1 | | 4 | | 2 | Read each colored row left to right. Aᵀ: 1 rows × 3 columns Aᵀ: 1 rows × 3 columns | | | | | --- | --- | --- | | −1 | 4 | 2 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Dot product: multiply matching entries, then add A **dot product** turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed. $$ w^Tx=w_1x_1+w_2x_2+w_3x_3 $$ This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products. ### Dot-product example 1 **Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.** ### Given Weights w = [2, 3]. Matching inputs x = [4, 1]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (2)(4)=8 $$ **Step 2: multiply matching pair 2.** $$ (3)(1)=3 $$ **Step 3: add the products.** $$ w^Tx=(8)+(3)=11 $$ **Final answer:** wᵀx = 11. ### Dot-product example 2 **Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.** ### Given Weights w = [1, -2]. Matching inputs x = [2, 3]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (1)(2)=2 $$ **Step 2: multiply matching pair 2.** $$ (-2)(3)=-6 $$ **Step 3: add the products.** $$ w^Tx=(2)+(-6)=-4 $$ **Final answer:** wᵀx = -4. ### Dot-product example 3 **Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.** ### Given Weights w = [0.5, 1, -1]. Matching inputs x = [4, 2, 3]. Both lists contain 3 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (0.5)(4)=2 $$ **Step 2: multiply matching pair 2.** $$ (1)(2)=2 $$ **Step 3: multiply matching pair 3.** $$ (-1)(3)=-3 $$ **Step 4: add the products.** $$ w^Tx=(2)+(2)+(-3)=1 $$ **Final answer:** wᵀx = 1. ### Matrix multiplication: each row makes one output To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B. $$ C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj} $$ For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once. **Yaad rakhne ka cue:** multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column. **Definition.** A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation. $$ z=Wx+b $$ For d input features and m receiving neurons, the dimensions are: $$ W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1} $$ The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product. ![Each row of the weight matrix calculates one receiving neuron's preactivation.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge. **Question.** Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b. ### Given Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ. ### To find Find both components of z = Wx+b. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. ### Solution — step by step **Step 1: verify compatible dimensions.** W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column. **Step 2: calculate the first receiving row products.** $$ 1(2)=2 $$ $$ (-2)(1)=-2 $$ **Step 3: sum row 1 and add its bias.** $$ z_1=2+(-2)+0.1=0.1 $$ **Step 4: calculate the second receiving row products.** $$ 0.5(2)=1 $$ $$ 3(1)=3 $$ **Step 5: sum row 2 and add its bias.** $$ z_2=1+3-0.2=3.8 $$ $$ z_1=1(2)+(-2)(1)+0.1=0.1 $$ Row 1 produces output component z1 1. Select weight row 1 1. Select weight row 1 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 2 | −2 | First × first; second × second. Sum the products**0** + Bias for this row**0.1** = z1**0.1** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z_2=0.5(2)+3(1)-0.2=3.8 $$ Row 2 produces output component z2 1. Select weight row 2 1. Select weight row 2 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 1 | 3 | First × first; second × second. Sum the products**4** + Bias for this row**−0.2** = z2**3.8** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z=\begin{bmatrix}0.1\\3.8\end{bmatrix} $$ An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W. ### Two output columns **Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.** ### Given $$ A=\begin{bmatrix}1&2\\3&4\end{bmatrix} $$ $$ B=\begin{bmatrix}2&0\\1&3\end{bmatrix} $$ ### To find The 2 × 2 matrix C = AB. ### Formula — which one and why Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication. Cᵢⱼ is the entry in output row i, column j. $$ C_{ij}=\sum_k A_{ik}B_{kj} $$ **Yaad rakhne ka cue:** Output ki position dekho: wahi row, wahi column. ### Solution — step by step **Step 1: check the dimensions.** (2 × 2)(2 × 2) is valid; output size = 2 × 2. **Step 2: row 1 with column 1.** $$ C_{11}=1(2)+2(1)=2+2=4 $$ **Step 3: row 1 with column 2.** $$ C_{12}=1(0)+2(3)=0+6=6 $$ **Step 4: row 2 with column 1.** $$ C_{21}=3(2)+4(1)=6+4=10 $$ **Step 5: row 2 with column 2.** $$ C_{22}=3(0)+4(3)=0+12=12 $$ **Step 6: assemble the output by position.** $$ C=\begin{bmatrix}4&6\\10&12\end{bmatrix} $$ **Final answer:** C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2. ### Two products that must not be confused **Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.** ### Given a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5. ### To find The 2 × 1 entry-by-entry product and the 2 × 2 outer product. ### Formula — which one and why ⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons. ⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row. $$ (a\odot b)_i=a_i b_i $$ $$ (ab^T)_{ij}=a_i b_j $$ **Yaad rakhne ka cue:** ⊙ mein same jagah; outer product mein har pair. ### Solution — step by step **Step 1: multiply matching entries.** $$ a\odot b=\begin{bmatrix}8\\15\end{bmatrix} $$ **Step 2: transpose b.** $$ b^T=\begin{bmatrix}4&5\end{bmatrix} $$ **Step 3: form row 1 of the outer product.** $$ 2(4)=8,\qquad2(5)=10 $$ **Step 4: form row 2 of the outer product.** $$ 3(4)=12,\qquad3(5)=15 $$ **Step 5: write both rows.** $$ ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix} $$ **Final answer:** Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ . ## Drill 5 Matrix rows receive inputs ## Question **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ed1fcbfdf5.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ. Activation: ReLU on each result. ### To find Find Wx+b and the activated output. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formula Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row. $$ z=Wx+b,\qquad a_j=\max(0,z_j) $$ $$ W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix} $$ ### Step 1: calculate receiving neuron 1 from row 1 $$ z_1=(1\times2)+(2\times1)+0.5 $$ $$ z_1=2+2+0.5=4.5 $$ ### Step 2: calculate receiving neuron 2 from row 2 $$ z_2=((-1)\times2)+(3\times1)+(-0.5) $$ $$ z_2=-2+3-0.5=0.5 $$ ### Step 3: apply ReLU to each entry $$ a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5 $$ $$ z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix} $$ The shape is 2 rows by 1 column: one output for each receiving row. --- ## TEE · Layer forward pass Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/7/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/7 A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f(z^{(l)}) $$ ## Forward propagation through multiple neurons **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations. ### Input, hidden and output layers The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task. In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections. ![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg) Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights. A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge. ### General layer equations and dimensions Let a⁽⁰⁾ = x. For layer l: $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f^{(l)}(z^{(l)}) $$ If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j. $$ W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1} $$ For one hidden layer, these computations compose to give: $$ h=f^{(1)}(W^{(1)}x+b^{(1)}) $$ $$ \hat y=f^{(2)}(W^{(2)}h+b^{(2)}) $$ The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer. ### Forward propagation versus training Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture. ### Complete two-layer numerical **Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.** ### Given A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. ### To find Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg) Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge. **Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output. **Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias. $$ z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3 $$ **Step 3: calculate the second hidden preactivation.** $$ z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5 $$ **Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them. $$ a^{(1)}_1=\max(0,0.3)=0.3 $$ $$ a^{(1)}_2=\max(0,0.5)=0.5 $$ $$ a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix} $$ **Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs. $$ z^{(2)}=0.5(0.3)+0.6(0.5)+0.1 $$ $$ z^{(2)}=0.15+0.30+0.10=0.55 $$ **Step 6: apply the output activation.** $$ \hat y=\max(0,0.55)=0.55 $$ **Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass. ## Model answer: forward propagation, backpropagation and updating ## Question **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations. ## Answer — step by step Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs. ![Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge. | Operation | Direction or action | Result | | --- | --- | --- | | Forward propagation | Inputs move through successive layers | Predictions and intermediate activations | | Loss calculation | Compare predictions with targets | A scalar objective for the example or batch | | Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters | | Optimizer update | Use those gradients to change parameters | A new set of weights and biases | ### 1. Forward propagation At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ ### 2. Loss Use the loss specified by the task. For one regression output, half squared loss is one possible choice: $$ L=\frac12(a-y)^2 $$ Here a is the prediction and y is the target. Other stated losses produce different derivatives. ### 3. Backpropagation Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it. ### 4. Update and repeat $$ w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w} $$ Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 13 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 14 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## TEE · Sigmoid and its derivative Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/8/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/8 Sigmoid takes a raw score and changes it to a number between 0 and 1. Its derivative is a different number: it tells backpropagation how sensitive that output is to its raw score. The complete example below calculates both and shows where each is used. $$ \sigma(z)=\frac{1}{1+e^{-z}} $$ $$ \sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ### sigmoid example 1 **Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.** ### Given z = -2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(-2)=2 $$ **Step 2: calculate the exponential.** $$ e^{2}\approx7.389056099 $$ **Step 3: add one in the denominator.** $$ 1+7.389056099\approx8.389056099 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: use the activated output in the slope.** $$ f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585 $$ **Final answer:** a ≈ 0.119202922. Derivative ≈ 0.104993585. ### sigmoid example 2 **Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.** ### Given z = 0; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: add one in the denominator.** $$ 1+1.000000000\approx2.000000000 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: use the activated output in the slope.** $$ f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000 $$ **Final answer:** a ≈ 0.500000000. Derivative ≈ 0.250000000. ### sigmoid example 3 **Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.** ### Given z = 2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(2)=-2 $$ **Step 2: calculate the exponential.** $$ e^{-2}\approx0.135335283 $$ **Step 3: add one in the denominator.** $$ 1+0.135335283\approx1.135335283 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: use the activated output in the slope.** $$ f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585 $$ **Final answer:** a ≈ 0.880797078. Derivative ≈ 0.104993585. --- ## TEE · Tanh and its derivative Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/9/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/9 Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input. $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ \tanh\prime(z)=1-\tanh^2(z) $$ **Question. Calculate tanh(0.5) and its derivative with respect to the input.** ### Given z = 0.5. ### To find Find tanh(z) and its derivative. ### Formula — which one and why **Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a². z = value entering tanh; a = its output. A prime means derivative, or local rate of change. $$ a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}} $$ $$ \frac{da}{dz}=1-a^2 $$ **Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square. ### Solution — step by step **Step 1: identify the input.** The input z is 0.5. In this version of the tanh formula, the exponential uses 2z. $$ 2z=2(0.5)=1,\qquad e^{2z}=e^1\approx2.718281828 $$ **Step 2: calculate the numerator.** Subtract 1 from that exponential. $$ e^{2z}-1=2.718281828-1=1.718281828 $$ **Step 3: calculate the denominator.** Add 1 to the same exponential. $$ e^{2z}+1=2.718281828+1=3.718281828 $$ **Step 4: divide the top by the bottom.** $$ a=\tanh(0.5)=\frac{1.718281828}{3.718281828}\approx0.462117157 $$ **Step 5: calculate the local derivative.** Square the output a and subtract it from 1. Keep the calculator's unrounded result. $$ a^2\approx0.213552267,\qquad1-a^2\approx0.786447733 $$ **Final answer:** tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward. ### tanh example 1 **Question. For preactivation z = -2, calculate the tanh output and state its derivative.** ### Given z = -2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(-2)=-4 $$ **Step 2: calculate the exponential.** $$ e^{-4}\approx0.018315639 $$ **Step 3: calculate numerator and denominator separately.** $$ 0.018315639-1\approx-0.981684361 $$ $$ 0.018315639+1\approx1.018315639 $$ **Step 4: divide.** $$ a=\frac{-0.981684361}{1.018315639}\approx-0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(-2)=1-(-0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ -0.964027580. Derivative ≈ 0.070650825. ### tanh example 2 **Question. For preactivation z = 0, calculate the tanh output and state its derivative.** ### Given z = 0; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: calculate numerator and denominator separately.** $$ 1.000000000-1\approx0.000000000 $$ $$ 1.000000000+1\approx2.000000000 $$ **Step 4: divide.** $$ a=\frac{0.000000000}{2.000000000}\approx0.000000000 $$ **Step 5: square the activated output for its slope.** $$ f\prime(0)=1-(0.000000000)^2\approx1.000000000 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 1.000000000. ### tanh example 3 **Question. For preactivation z = 2, calculate the tanh output and state its derivative.** ### Given z = 2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(2)=4 $$ **Step 2: calculate the exponential.** $$ e^{4}\approx54.598150033 $$ **Step 3: calculate numerator and denominator separately.** $$ 54.598150033-1\approx53.598150033 $$ $$ 54.598150033+1\approx55.598150033 $$ **Step 4: divide.** $$ a=\frac{53.598150033}{55.598150033}\approx0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(2)=1-(0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ 0.964027580. Derivative ≈ 0.070650825. --- ## TEE · ReLU and its derivative Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/10/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/10 ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side. $$ f(z)=\max(0,z) $$ $$ f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### ReLU example 1 **Question. For preactivation z = -2, calculate the ReLU output and state its derivative.** ### Given z = -2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 0.000000000. ### ReLU example 2 **Question. For preactivation z = 0, calculate the ReLU output and state its derivative.** ### Given z = 0; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### ReLU example 3 **Question. For preactivation z = 2, calculate the ReLU output and state its derivative.** ### Given z = 2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. --- ## TEE · Leaky ReLU Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/11/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/11 Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function. $$ f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### Leaky ReLU example 1 **Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = -2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=-0.2 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0.1 $$ **Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000. ### Leaky ReLU example 2 **Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 0; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### Leaky ReLU example 3 **Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. --- ## TEE · Stable softmax Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/12/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/12 Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ $$ c=\max_j z_j $$ ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. ### Softmax for [0, 0, 0] **Question. Find stable softmax probabilities for logits [0, 0, 0].** ### Given Logits = [0, 0, 0]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}0&0&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{0}\approx1.000000000 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{0}\approx1.000000000 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=1.000000000+1.000000000+1.000000000\approx3.000000000 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding. ### Softmax for [1, 2, 3] **Question. Find stable softmax probabilities for logits [1, 2, 3].** ### Given Logits = [1, 2, 3]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}-2&-1&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{-2}\approx0.135335283 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{-1}\approx0.367879441 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=0.135335283+0.367879441+1.000000000\approx1.503214724 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{0.135335283}{1.503214724}\approx0.090030573 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{0.367879441}{1.503214724}\approx0.244728471 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{1.503214724}\approx0.665240956 $$ **Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding. --- ## TEE · Squared and absolute losses Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/13/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/13 SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate. $$ \mathrm{SSE}=\sum_i(o_i-y_i)^2 $$ $$ L_{\mathrm{half}}=\frac{\mathrm{SSE}}2 $$ $$ L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n $$ $$ L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i| $$ ## Loss: how one wrong prediction becomes a number **Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative. $$ e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2 $$ For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective. ### Derivatives and their scaling When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_j}=2e_j,\qquad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_j}=e_j,\qquad \frac{\partial\mathrm{MSE}}{\partial\hat y_j}=\frac{2}{n}e_j $$ The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight. **Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.** ### Given Predictions = [0.2,0.7,0.4]. Targets = [0,1,1]; n = 3 scalar predictions. ### To find Find SSE, half SSE and MSE, then all three prediction-gradient lists. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step **Step 1: calculate the three signed prediction errors.** $$ e=[0.2-0,\ 0.7-1,\ 0.4-1]=[0.2,-0.3,-0.6] $$ **Step 2: square every error.** $$ e\odot e=[0.04,0.09,0.36] $$ **Step 3: add the three squared errors.** $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ **Step 4: apply the requested loss divisors.** $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245,\qquad \mathrm{MSE}=\frac{0.49}{3}\approx0.163333333 $$ **Step 5: multiply every error by 2 for the SSE gradient.** $$ \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2] $$ **Step 6: use each error itself for the half-SSE gradient.** $$ \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6] $$ **Step 7: multiply every error by 2/3 for the MSE gradient.** $$ \nabla_{\hat y}\mathrm{MSE}=\frac23[0.2,-0.3,-0.6]=[0.133333333,-0.2,-0.4] $$ The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate. ## Drill 2 Half sum versus mean ## Question **Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE. ## Answer — step by step ### Given Predictions = [0.2, 0.7, 0.4]. Targets = [0, 1, 1]; number of outputs n = 3. ### To find Find SSE, half SSE, MSE and the prediction gradients requested in the question. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step ### Given values and loss definitions There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i. $$ \mathrm{SSE}=\sum_{i=1}^{n}e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}2 $$ $$ \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}n $$ SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively. ### Step 1: calculate each residual and its square | Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual | | --- | --- | --- | --- | --- | | 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 | | 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 | | 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 | ### Step 2: sum the squared residuals $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel. ### Step 3: apply each requested normalization For half SSE, the denominator is 2: $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245 $$ For MSE, the denominator is the number of pairs, n = 3: $$ \mathrm{MSE}=\frac{0.49}{3}=0.163333333\ldots $$ **Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question. ### Absolute error uses a different operation **Question. Targets are [2, 4] and predictions are [1, 6]. Find mean absolute error.** ### Given Targets = [2,4]; predictions = [1,6]; n = 2. ### To find Find mean absolute error. ### Formula — which one and why **Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes. n = number of predictions. |e| means the nonnegative magnitude of error e. $$ \mathrm{MAE}=\frac1n\sum_i|\hat y_i-y_i| $$ **Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square. ### Solution — step by step The signed errors are 1 − 2 = −1 and 6 − 4 = 2. Absolute value keeps the size of a number without its sign, so their absolute errors are 1 and 2. Add them to obtain 3. There are two prediction–target pairs, so divide by 2. $$ \mathrm{MAE}=\frac{|1-2|+|6-4|}{2}=\frac{1+2}{2}=1.5 $$ **Final answer: MAE = 1.5.** Do not square the errors for MAE. --- ## TEE · Binary cross entropy Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/14/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/14 Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### Binary cross-entropy: target 1, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.8 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.223143551)=0.223143551 $$ **Final answer:** Loss ≈ 0.223143551 nats. ### Binary cross-entropy: target 0, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 0; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ 1-p=1-0.8=0.2 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.2)\approx-1.609437912 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-1.609437912)=1.609437912 $$ **Final answer:** Loss ≈ 1.609437912 nats. ### Binary cross-entropy: target 1, predicted positive-class probability 0.5 **Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.5; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.5 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.693147181)=0.693147181 $$ **Final answer:** Loss ≈ 0.693147181 nats. --- ## TEE · Categorical cross entropy Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/15/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/15 For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ \log_2q=\frac{\ln q}{\ln2} $$ $$ L=-\sum_i y_i\ln p_i=-\ln p_{true} $$ ## Cross-entropy for probability distributions **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities. $$ H(p,q)=-\sum_{i=1}^{K}p_i\log q_i $$ Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms. ### Logarithm base and units Natural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2: $$ \log_2 q=\frac{\ln q}{\ln 2} $$ For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities. ### A target distribution with two nonzero entries **Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.** ### Given A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms. ### To find Find cross-entropy in the requested units. ### Formula — which one and why **Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute. pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ H_2(p,q)=\frac{H(p,q)}{\ln2} $$ **Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai. ### Solution — step by step **Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded. $$ H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)] $$ **Step 2: calculate the logarithms.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: weight each negative logarithm by its target probability.** $$ -0.75\ln(0.5)\approx0.519860385 $$ $$ -0.25\ln(0.5)\approx0.173286795 $$ **Step 4: add the contributions, retaining calculator precision.** $$ H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats} $$ **Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1. $$ H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit} $$ **Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale. ### Cross-entropy, entropy and relative entropy These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p) $$ For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus. **Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.** ### Given For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats. ### To find Find entropy and KL divergence in nats. ### Formula — which one and why **Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p. p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p) $$ **Yaad rakhne ka cue:** KL = cross-entropy minus target ki own entropy. ### Solution — step by step **Step 1: calculate the target entropy.** $$ H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)] $$ $$ \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361 $$ $$ H(p)\approx0.215761554+0.346573590=0.562335145 $$ **Step 2: subtract target entropy from cross-entropy.** $$ D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036 $$ **Final answer:** H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ. ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. --- ## TEE · Output delta for half squared loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/16/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/16 Delta is a short name for the loss derivative with respect to a neuron’s raw input z. For half squared loss, multiply prediction minus target by the activation’s derivative. The result is then multiplied by an incoming activation to obtain a weight gradient. $$ \delta^{out}=(o-y)f\prime(z) $$ ## Loss scaling: derive the gradient instead of guessing a factor ### 1. One error, three possible objectives Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square: ![Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge. $$ \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1 $$ Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below. | Stated loss over n outputs | Derivative with respect to one prediction aⱼ | | --- | --- | | SSE: sum of all squared errors | 2(aⱼ − yⱼ) | | Half SSE: half of that sum | aⱼ − yⱼ | | MSE: that sum divided by n | 2(aⱼ − yⱼ)/n | | Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n | ### 2. Why only the matching output appears Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor. Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages. ### 3. The activation contributes another factor If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore: $$ \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a) $$ For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question. ### 4. A numerical comparison For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042. The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used. ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ## Drill 13 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 14 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## TEE · Sigmoid plus BCE; softmax plus CCE Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/17/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/17 Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect. $$ \delta^{out}=p-y $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### The multiclass version, entry by entry **Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.** ### Given Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0]. One-example categorical cross-entropy. ### To find Find all three logit derivatives. ### Formula — which one and why **Softmax with categorical cross-entropy.** For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target. zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry. $$ \frac{\partial L}{\partial z_i}=p_i-y_i $$ **Yaad rakhne ka cue:** Softmax + CCE together: p minus y. Extra sigmoid slope nahi. ### Solution — step by step A logit is the raw score before softmax. Subtract each target entry from its matching probability entry. $$ \frac{\partial L}{\partial z_1}=0.1-0=0.1 $$ $$ \frac{\partial L}{\partial z_2}=0.8-1=-0.2 $$ $$ \frac{\partial L}{\partial z_3}=0.1-0=0.1 $$ **Final answer: [0.1, −0.2, 0.1].** Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor. ### Sigmoid+BCE logit gradient: p=0.8, y=1 **Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 1; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(-1.25)(0.16)=-0.2 $$ **Final answer:** dL/dz = -0.2. No further sigmoid derivative is needed. ### Sigmoid+BCE logit gradient: p=0.8, y=0 **Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 0; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-0}{0.16}=5 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(5)(0.16)=0.8 $$ **Final answer:** dL/dz = 0.8. No further sigmoid derivative is needed. --- ## TEE · Hidden delta, weight gradient and bias gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/18/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/18 A hidden neuron can influence several outputs. Add all those backward contributions before multiplying by its own activation derivative. For each connection, use the receiving neuron’s delta and the sending neuron’s activation. Keep the old weights unchanged until all gradients have been found. $$ \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out} $$ $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ ## More outputs and vectorized backpropagation **Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path. ### Why the hidden derivative is a sum If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives: $$ \frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2} $$ For any number of connected outputs, sum their contributions and then differentiate the hidden activation: $$ \delta_{h_j}=\left(\sum_k W_{2,kj}\delta_{o_k}\right)f'(z_{h_j}) $$ Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss. ![The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_branch_gradient.svg) Diagram: The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.. Labels: Two output paths both affect the hidden neuron | Two output paths both affect the hidden neuron | Example: hidden activation h = 0.6, using sigmoid in the hidden neuron. | h = 0.6 | Output 1 delta = 0.2 | Connecting weight = 0.4 | Output 2 delta = −0.1 | Connecting weight = 0.5 | Downstream sum = 0.2 × 0.4 + (−0.1) × 0.5 = 0.03 | Hidden delta = 0.03 × 0.6 × (1 − 0.6) = 0.0072 | Add the two downstream paths first. Then multiply by the hidden activation | derivative. The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative. Tap the diagram to enlarge. ### Matrix equations for one training sample Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂: $$ L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2 $$ $$ \delta_2=(\hat y-y)\odot f_2'(z_2) $$ $$ \frac{\partial L}{\partial W_2}=\delta_2a_1^T,\qquad\frac{\partial L}{\partial b_2}=\delta_2 $$ $$ \delta_1=(W_2^T\delta_2)\odot f_1'(z_1) $$ $$ \frac{\partial L}{\partial W_1}=\delta_1x^T,\qquad\frac{\partial L}{\partial b_1}=\delta_1 $$ Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation. | Quantity | Shape | Reason | | --- | --- | --- | | W₁ and its gradient | m × d | m receiving hidden neurons, d inputs | | W₂ and its gradient | k × m | k receiving output neurons, m hidden activations | | δ₂ | k × 1 | One derivative per output preactivation | | W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit | | δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ | ### How the loss changes the output delta For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target: $$ \delta_{\mathrm{output}}=\hat y-y $$ That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required. For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 13 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 14 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 15 A negative ReLU blocks this path ## Question **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2. ## Answer — step by step ### Given x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2. Hidden activation: ReLU; output activation: linear. Target y = 1; loss: half squared error. ### To find Find the gradients of w₁ and w₂ and explain any zero gradients. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output. $$ z_h=w_1x+b_1,\quad h=\max(0,z_h),\quad o=w_2h+b_2 $$ $$ L=\frac12(o-y)^2,\quad \delta_o=o-y $$ $$ \frac{\partial L}{\partial w_2}=\delta_o h,\quad \delta_h=\delta_o w_2\operatorname{ReLU}'(z_h),\quad \frac{\partial L}{\partial w_1}=\delta_h x $$ ### Step 1: hidden weighted sum and activation $$ z_h=(-0.4)\times2+0.1=-0.8+0.1=-0.7 $$ $$ h=\max(0,-0.7)=0 $$ Because the raw hidden score is negative, ReLU's derivative at this point is 0. ### Step 2: output prediction and output delta $$ o=0.5\times0+0.2=0.2 $$ $$ \delta_o=0.2-1=-0.8 $$ ### Step 3: output weight gradient $$ \frac{\partial L}{\partial w_2}=(-0.8)\times0=0 $$ This is zero because the input to that connection, h, is zero. ### Step 4: hidden delta and input weight gradient $$ \delta_h=(-0.8)\times0.5\times0=0 $$ $$ \frac{\partial L}{\partial w_1}=0\times2=0 $$ **Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero. --- ## TEE · Gradient descent Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/19/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/19 A gradient tells us the local direction in which loss rises. Gradient descent subtracts a small multiple of that gradient. Learning rate η controls the size of this correction. After the first update, calculate the next gradient at the new parameter. $$ \theta_{new}=\theta-\eta g $$ $$ g=\frac{\partial J}{\partial\theta} $$ ## Reading an optimizer formula without getting lost An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated. ![Old value, current gradient and new value belong to different moments in the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. Old value, current gradient and new value belong to different moments in the update. Tap the diagram to enlarge. ### Old, current and new At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position. $$ \theta_t=\theta_{t-1}-\eta g_t $$ η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition. ### What a stored state means An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares. ### A weighted average from ordinary arithmetic If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square. $$ 0.9\times4=3.6,\qquad(1-0.9)\times9=0.9 $$ $$ s_{\mathrm{new}}=3.6+0.9=4.5 $$ The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum. ### Squared memory and signed direction For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g². ### Epsilon and hats ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second. ## The ordinary gradient descent update **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate. ### Explanation and working $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent. Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction. ![Calculate the slope first, multiply it by the learning rate, then subtract that correction.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: Calculate the slope first, multiply it by the learning rate, then subtract that correction.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. Calculate the slope first, multiply it by the learning rate, then subtract that correction. Tap the diagram to enlarge. **Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations. ### Given J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1. ### To find Differentiate J and perform two gradient-descent iterations. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ². **Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79. **Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061. **Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**. The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum. **Common error:** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients. ## Drill 18 GD variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ \theta_1=1-0.1(1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ \theta_2=0.9-0.1(0.9)=0.81 $$ **Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 19 GD variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ \theta_1=-0.5-0.1(-1.5)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ \theta_2=-0.35-0.1(-1.35)=-0.215 $$ **Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. --- ## TEE · Batch and mini-batch gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/20/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/20 A batch is a group of training examples used together for one update. An epoch is one pass through the dataset. Averaging gradients and counting updates are different questions, so both receive complete worked examples below. $$ g_{batch}=\frac1B\sum_{i=1}^{B}g_i $$ $$ U_{epoch}=\left\lceil\frac NB\right\rceil $$ ## Model answer: batch, stochastic and mini-batch training ## Question **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration. ## Answer — step by step All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update. ![All methods visit the same six examples. Their grouping determines how often they update.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: All methods visit the same six examples. Their grouping determines how often they update.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge. A **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations. | Basis | Batch GD | Stochastic GD | Mini-batch GD | | --- | --- | --- | --- | | Examples in one gradient | Entire training dataset | One example | A smaller group of B examples | | Updates per epoch | One | N | Ceiling of N/B if the remainder is retained | | Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging | | Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation | ### Example to distinguish the three counts With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch. The basic parameter update is unchanged: $$ \theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g $$ For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples. ## A tiny dataset shows why the methods differ **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order. ### Given Model ŷ = wx; initial w = 1; η = 0.1. Samples in order: (1,2), (2,4). Mean half squared loss for the batch. ### To find Compare one batch update with a sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### One batch update **Step 1: predict sample 1 with the original weight.** $$ \hat y_1=1(1)=1 $$ **Step 2: calculate its error.** $$ e_1=1-2=-1 $$ **Step 3: multiply its error by its input.** $$ g_1=(-1)(1)=-1 $$ **Step 4: predict sample 2 with the same original weight.** $$ \hat y_2=1(2)=2 $$ **Step 5: calculate its error.** $$ e_2=2-4=-2 $$ **Step 6: multiply its error by its input.** $$ g_2=(-2)(2)=-4 $$ **Step 7: average the two gradients.** $$ g=\frac{-1+(-4)}2=-2.5 $$ **Step 8: update once.** $$ w_{\mathrm{new}}=1-0.1(-2.5)=1.25 $$ ![This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge. ### One sequential SGD epoch **Step 1: use the first sample gradient.** $$ g_1=-1 $$ **Step 2: update after the first sample.** $$ w_1=1-0.1(-1)=1.1 $$ **Step 3: predict sample 2 using the changed weight.** $$ \hat y_2=1.1(2)=2.2 $$ **Step 4: find sample 2 error.** $$ e_2=2.2-4=-1.8 $$ **Step 5: find the new sample 2 gradient.** $$ g_2=(-1.8)(2)=-3.6 $$ **Step 6: update after sample 2.** $$ w_2=1.1-0.1(-3.6)=1.46 $$ **Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples. Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim. ## Drill 16 Batch and sequential SGD ## Question **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch. ## Answer — step by step ### Given Model ŷ = wx; initial w = 0; η = 0.1. Samples in order: (x,y) = (1,1), then (2,2). Batch objective: mean half squared loss. ### To find Compare the weight after one batch update and one sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### Meaning and formulas For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample. $$ \hat y=wx,\quad L=\frac12(\hat y-y)^2,\quad g=(\hat y-y)x $$ $$ g_{\mathrm{batch}}=\frac{g_1+g_2}{2},\qquad w_{\mathrm{new}}=w-\eta g $$ **Step 1: Part A: one batch update from w = 0.** For sample (1,1), prediction = 0 × 1 = 0, so: $$ g_1=(0-1)\times1=-1 $$ For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so: $$ g_2=(0-2)\times2=-4 $$ Average these two gradients, then update once. $$ g_{\mathrm{batch}}=\frac{-1+(-4)}2=-2.5 $$ $$ w_{\mathrm{batch}}=0-0.1(-2.5)=0.25 $$ **Step 2: Part B: one SGD epoch, restarting at w = 0.** The first sample has gradient −1, just as above. $$ w_1=0-0.1(-1)=0.1 $$ For the second sample use this new weight, so prediction is: $$ \hat y_2=0.1\times2=0.2 $$ $$ g_2=(0.2-2)\times2=(-1.8)\times2=-3.6 $$ $$ w_2=0.1-0.1(-3.6)=0.1+0.36=0.46 $$ **Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates. ## Drill 17 Incomplete mini batch ## Question **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it. ## Answer — step by step ### Given N = 103 samples; B = 20 samples per mini-batch. E = 3 epochs. Compare both remainder policies: keep or drop the final 3 examples. ### To find Find total updates for full-batch GD, SGD, and both mini-batch policies. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step ### Meaning and formulas An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs. $$ U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NE $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained. ### Step 1: batch GD All 103 samples form one batch per epoch. $$ U_{\mathrm{batch}}=3\times1=3 $$ ### Step 2: SGD Each sample gets its own update. $$ U_{\mathrm{SGD}}=103\times3=309 $$ ### Step 3: mini-batches with the remainder $$ 103=5\times20+3 $$ The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give: $$ U_{\mathrm{keep}}=6\times3=18 $$ ### Step 4: mini-batches without the remainder Only the five complete batches are used each epoch. $$ U_{\mathrm{drop}}=5\times3=15 $$ **Final answer:** 3, 309, 18 and 15 updates respectively. --- ## TEE · Momentum: signed velocity Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/21/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/21 Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ## Model answer: momentum and Nesterov ## Question **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ? ## Answer — step by step Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions. ![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge. ### Momentum formulas This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained. $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ### Nesterov's change Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look}}) $$ The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point. | Basis | Momentum | Nesterov | | --- | --- | --- | | Gradient evaluated at | Current parameter | Lookahead parameter | | Information retained | Previous velocity | Previous velocity | | Intuition | Continue useful movement | Inspect the slope where old momentum points | | Initial zero velocity | No movement contribution yet | First lookahead equals current parameter | Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently. ## Momentum remembers movement **Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions. ### Update equations and convention This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter: $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters. ![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg) Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients. Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration. **Step 1: Iteration 1: gradient, velocity, parameter.** $$ g_1=J'(1)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Iteration 2: use θ₁ and v₁.** $$ g_2=J'(0.9)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ | Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter | | --- | --- | --- | --- | --- | | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.9 | −0.18 | 0.72 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient. ### Properties and alternative notation Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective. Some texts store a positive gradient accumulator u instead: $$ u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t $$ With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions. ## Drill 20 Momentum variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ **Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 21 Momentum variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ v_2=0.9(0.15)-0.1(-1.35)=0.27 $$ $$ \theta_2=-0.35+(0.27)=-0.08 $$ **Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## TEE · Nesterov: look-ahead gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/22/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/22 Nesterov first uses the old movement to choose a temporary look-ahead position. It measures the gradient there, calculates the new movement, then adds that movement to the original parameter. The temporary look-ahead position does not replace the original parameter in the final addition. $$ \theta_{look}=\theta+\beta v $$ $$ v_{new}=\beta v-\eta g(\theta_{look}) $$ $$ \theta_{new}=\theta+v_{new} $$ ## Nesterov looks before correcting **Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update. ### What changes relative to momentum Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured: $$ \theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look},t}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁. ![Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective. **Iteration 1 — step 1: locate the lookahead point.** $$ \theta_{\mathrm{look},1}=1+0.9(0)=1 $$ **Step 2: evaluate the gradient there.** $$ g_1=J'(1)=1 $$ **Step 3: calculate the new velocity and actual parameter.** $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Iteration 2 — step 1: use the previous actual parameter and velocity.** $$ \theta_{\mathrm{look},2}=0.9+0.9(-0.1)=0.81 $$ **Step 2: evaluate the gradient at 0.81, not at 0.9.** $$ g_2=J'(0.81)=0.81 $$ **Step 3: calculate the new velocity.** $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ **Step 4: update the old actual parameter.** $$ \theta_2=0.9+(-0.171)=0.729 $$ | Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ | | --- | --- | --- | --- | --- | --- | | 1 | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice. ## Drill 22 Nesterov variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=1+0.9(0)=1 $$ $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81 $$ $$ g_2=0.81-(0)=0.81 $$ $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ $$ \theta_2=0.9+(-0.171)=0.729 $$ **Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. ## Drill 23 Nesterov variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=-0.5+0.9(0)=-0.5 $$ $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=-0.35+0.9(0.15)=-0.215 $$ $$ g_2=-0.215-(1)=-1.215 $$ $$ v_2=0.9(0.15)-0.1(-1.215)=0.2565 $$ $$ \theta_2=-0.35+(0.2565)=-0.0935 $$ **Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. --- ## TEE · AdaGrad Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/23/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/23 AdaGrad stores a running sum of squared gradients for each parameter. Squaring removes the sign only in this stored total. The unsquared gradient in the numerator still decides the direction of the update. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ ## AdaGrad adds every squared gradient **Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator. ### State, effective learning rate and update For one parameter θ, let gₜ be its current gradient and s₀ = 0: $$ s_t=s_{t-1}+g_t^2 $$ $$ \eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon} $$ $$ \theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t $$ s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign. ![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg) Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update. The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge. ### Complete numerical: supplied gradients **Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter. **Iteration 1 — step 1: update the accumulator.** $$ s_1=0+2^2=4 $$ **Step 2: calculate the effective learning rate.** $$ \eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05 $$ **Step 3: apply the signed gradient.** $$ \theta_1=1-0.05(2)=0.9 $$ **Iteration 2 — step 1: retain s₁ and add the new squared gradient.** $$ s_2=4+1^2=5 $$ **Step 2: calculate the new coefficient.** $$ \eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360 $$ **Step 3: update from θ₁.** $$ \theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640 $$ **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5. ### Advantage, limitation and epsilon convention Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes. ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical. ## Drill 24 AdaGrad variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_1=0+(2)^2=4 $$ $$ \sqrt{G_1}=\sqrt{4}=2 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** Add the new squared gradient to all previous squares. $$ G_2=4+(1)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=0.9-\frac{0.1\times(1)}{\sqrt{5}}=0.85527864 $$ ## Drill 25 AdaGrad variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Add the new squared gradient to all previous squares. $$ G_1=0+(-1)^2=1 $$ $$ \sqrt{G_1}=\sqrt{1}=1 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_2=1+(2)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=-0.4-\frac{0.1\times(2)}{\sqrt{5}}=-0.489442719 $$ --- ## TEE · RMSProp Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/24/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/24 RMSProp remembers a weighted average of squared gradients. With decay 0.9, keep 90% of the old stored value and add 10% of the new squared gradient. Calculate this new stored value before taking its square root and updating the parameter. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ ## A moving average remembers the recent past **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients. $$ m_t=\beta m_{t-1}+(1-\beta)g_t,\qquad0\le\beta<1 $$ For m₀ = 0, expanding the recurrence gives: $$ m_t=(1-\beta)\sum_{k=1}^{t}\beta^{t-k}g_k $$ The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history. ### Signed gradients versus squared gradients An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form. **Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂. ### Given β = 0.9; initial m₀ = 0. Gradients in order: g₁ = 2, g₂ = −1. ### To find Find m₁ and m₂, preserving the sign of each gradient. ### Formula — which one and why **Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state. β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state. $$ m_t=\beta m_{t-1}+(1-\beta)g_t $$ **Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai. ### Solution — step by step **Step 1: find the recent-gradient coefficient.** $$ 1-0.9=0.1 $$ **Step 2: retain the old memory for step 1.** $$ 0.9(0)=0 $$ **Step 3: weight the first gradient.** $$ 0.1(2)=0.2 $$ **Step 4: add to obtain the first memory.** $$ m_1=0+0.2=0.2 $$ **Step 5: retain the first memory for step 2.** $$ 0.9(0.2)=0.18 $$ **Step 6: weight the second gradient.** $$ 0.1(-1)=-0.1 $$ **Step 7: add to obtain the second memory.** $$ m_2=0.18+(-0.1)=0.08 $$ **Final answer:** m₁ = 0.2; m₂ = 0.08. The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable. ## RMSProp remembers recent squared gradients **Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator. ### Recurrence and interpretation Using decay β and zero initial state: $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update. If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever. ![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg) Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient. Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β = 0.9. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step **Iteration 1 — step 1: calculate the squared-gradient average.** $$ s_1=0.9(0)+0.1(2^2)=0.4 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_1}=\sqrt{0.4}\approx0.632455532 $$ $$ \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766 $$ **Step 3: update the parameter.** $$ \theta_1=1-0.316227766\approx0.683772234 $$ **Iteration 2 — step 1: use s₁, not the original zero state.** $$ s_2=0.9(0.4)+0.1(1^2)=0.46 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_2}=\sqrt{0.46}\approx0.678232998 $$ $$ \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956 $$ **Step 3: update from θ₁, retaining calculator precision.** $$ \theta_2\approx0.683772234-0.147441956\approx0.536330278 $$ **Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46. ### Comparison with AdaGrad and Adam AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum. ## Drill 26 RMSProp variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(2)^2=0.4 $$ $$ \sqrt{s_1}=\sqrt{0.4}=0.632455532 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{0.4}}=0.683772234 $$ **Step 2: Iteration 2 with supplied gradient 1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.4)+0.1(1)^2=0.46 $$ $$ \sqrt{s_2}=\sqrt{0.46}=0.678232998 $$ $$ \theta_2=0.683772234-\frac{0.1\times(1)}{\sqrt{0.46}}=0.536330278 $$ ## Drill 27 RMSProp variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(-1)^2=0.1 $$ $$ \sqrt{s_1}=\sqrt{0.1}=0.316227766 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{0.1}}=-0.183772234 $$ **Step 2: Iteration 2 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.1)+0.1(2)^2=0.49 $$ $$ \sqrt{s_2}=\sqrt{0.49}=0.7 $$ $$ \theta_2=-0.183772234-\frac{0.1\times(2)}{\sqrt{0.49}}=-0.46948652 $$ --- ## TEE · Adam: both memories Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/25/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/25 Adam keeps two separate running averages. The first remembers signed gradients and can be negative. The second remembers squared gradients and is nonnegative. These stored averages are not weights, and they are not extra gradients to multiply together. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ ## Optimizer notation: old value, memory, new value ### 1. A parameter and a stored state are different θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries. ![The time label tells you which value to use. Carry the previous memory into the current update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: The time label tells you which value to use. Carry the previous memory into the current update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. The time label tells you which value to use. Carry the previous memory into the current update. Tap the diagram to enlarge. At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values. ### 2. A supplied gradient versus a supplied objective If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead. For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64. ### 3. A squared gradient keeps size and loses sign Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square. ### 4. Square root and epsilon belong to the denominator If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero. ### 5. A table prevents mixing iterations Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1. The worked examples below use this exact order, with every state shown. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 28 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 29 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ --- ## TEE · Adam: correction and update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/26/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/26 Adam starts its two averages at zero. Early averages therefore contain less total weight on observed gradients. Bias correction compensates for that initialization. The power in βᵗ uses the current update number, and both averages must be corrected before the parameter update. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \widehat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ ## Adam bias correction: why the denominator is one minus beta to a power **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate. ### Explanation and working ### 1. What the moving average actually contains Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization. ![The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam_correction.svg) Diagram: The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.. Labels: Correct the early zero-initialized memory | Correct the early zero-initialized memory | Example: β₁ = 0.9, first gradient = 2, initial m = 0. | Raw memory | m₁ = 0.9 × 0 + 0.1 × | 2 = 0.2 | Correction factor | 1 − 0.9¹ = 0.1 | Corrected memory | m̂₁ = 0.2 ÷ 0.1 = 2 | The small raw value is partly caused by the starting zero. Bias correction | removes that effect. The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor. Tap the diagram to enlarge. $$ m_1=(1-\beta)g_1 $$ At the second step, substitute the first expression into the recurrence: $$ m_2=\beta(1-\beta)g_1+(1-\beta)g_2 $$ ### 2. Add the coefficients After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term: $$ \beta(1-\beta)+(1-\beta)=(1-\beta)(1+\beta)=1-\beta^2 $$ After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ The squared-gradient memory uses its own decay β₂ and the same reasoning: $$ \widehat s_t=\frac{s_t}{1-\beta_2^t} $$ ### 3. A constant-gradient example makes the purpose visible Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38. $$ \widehat m_1=\frac{0.2}{1-0.9}=2 $$ $$ \widehat m_2=\frac{0.38}{1-0.9^2}=\frac{0.38}{0.19}=2 $$ Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient. ### 4. Two betas mean two separate powers At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators. ### 5. Correct first, then combine The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows. This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 28 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 29 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ --- ## TEE · L1 penalty and update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/27/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/27 L1 adds a cost based on the absolute sizes of weights. Its derivative is positive for a positive weight and negative for a negative weight. The optimizer adds this penalty gradient to the data gradient before applying the usual update. $$ J=L+\lambda\sum_j|w_j| $$ $$ w_{new}=w-\eta\left[g_{data}+\lambda\operatorname{sign}(w)\right] $$ ## Model answer: regularization and the L1–L2 difference ## Question **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength. ## Answer — step by step Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost. ![Follow the stated penalty convention, including lambda and any half factor, when you form the objective.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: Follow the stated penalty convention, including lambda and any half factor, when you form the objective.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge. ### Objectives used in this course $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question. | Basis | L1 | L2 | | --- | --- | --- | | Quantity penalized | Absolute weight magnitude | Squared weight magnitude | | Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention | | Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes | | Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights | The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total. Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective. ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 31 L1 update on positive and negative weights ## Question **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero. ## Answer — step by step ### Given Old weights = [−0.5,0.3]. Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2. ### To find Find total gradients and the two updated weights. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formula The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j) $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed. **Step 1: Weight 1: old value −0.5.** $$ \lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1 $$ $$ g_{1,\mathrm{total}}=0.2+(-0.1)=0.1 $$ $$ w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52 $$ **Step 2: Weight 2: old value 0.3.** $$ \lambda\operatorname{sign}(0.3)=0.1(1)=0.1 $$ $$ g_{2,\mathrm{total}}=-0.1+0.1=0 $$ $$ w_{2,\mathrm{new}}=0.3-0.2(0)=0.3 $$ **Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero. ## Drill 30 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 47 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. --- ## TEE · L2 penalty and update with a half Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/28/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/28 L2 adds a cost based on squared weights. Differentiating a square produces a factor of 2. If the objective includes a half, that half cancels the 2; if it does not, the 2 remains. Read the objective before choosing the gradient. $$ J=L+\frac\lambda2\sum_jw_j^2 $$ $$ w_{new}=w-\eta(g_{data}+\lambda w) $$ ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 32 L2 update ## Question **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw². ## Answer — step by step ### Given Old weights = [0.5,−0.2]; data gradients = [0.1,0.3]. λ = 0.2; η = 0.1; penalty = (λ/2)Σw². ### To find Find penalty gradients, total gradients and updated weights. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Formula and its reason The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda w_j $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ **Step 1: Weight 1.** $$ \lambda w_1=0.2\times0.5=0.1 $$ $$ g_{1,\mathrm{total}}=0.1+0.1=0.2 $$ $$ w_{1,\mathrm{new}}=0.5-0.1(0.2)=0.48 $$ **Step 2: Weight 2.** $$ \lambda w_2=0.2\times(-0.2)=-0.04 $$ $$ g_{2,\mathrm{total}}=0.3+(-0.04)=0.26 $$ $$ w_{2,\mathrm{new}}=-0.2-0.1(0.26)=-0.226 $$ **Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative. ## Drill 30 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 47 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. --- ## TEE · Inverted dropout Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/29/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/29 Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated. $$ q=1-p $$ $$ a_{train}=\frac{m\odot a}{q} $$ $$ a_{test}=a $$ ## Model answer: dropout with training and inference explained ## Question **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling. ## Answer — step by step Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights. ![The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge. Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop. ### Inverted dropout during training $$ a'=\frac{ma}{q} $$ A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks. For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is: $$ 0.5(6)+0.5(0)=3 $$ ### Inference Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions. Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data. ## Dropout removes activations during training **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training. ### Explanation and working Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count. ![Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout.svg) Diagram: Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.. Labels: Dropout temporarily removes selected activations | Dropout temporarily removes selected activations | Example: keep probability q = 0.5. Inverted dropout during training. | Original | 0.8 | -0.4 | 0.6 | 0.2 | Mask | 1 | 0 | 1 | 0 | After mask ÷ q | 1.6 | 0 | 1.2 | 0 | The kept values double: 0.8 ÷ 0.5 = 1.6 and 0.6 ÷ 0.5 = 1.2. At inference, no | mask is used. Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge. Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions. $$ a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout} $$ $$ a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout} $$ **Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs. ### Given Activations = [0.8,−0.4,0.6,0.2]. Dropout rate = 50%, so keep probability q = 0.5. Chosen mask = [1,0,1,0]. ### To find Find training outputs under inverted dropout. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step **Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key. **Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer. Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition. ## Drill 33 Inverted dropout ## Question **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference. ## Answer — step by step ### Given Activations = [2,−1,0,3]. Keep probability q = 0.75; keep mask = [1,0,1,1]. ### To find Find training outputs and state the inference rule. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step ### Meaning and formula The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved. $$ a'_j=\frac{m_j a_j}{q} $$ The multiplication is entry by entry; there is no dot-product sum. ### Step 1: apply the mask $$ [1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3] $$ ### Step 2: divide each result by 0.75 $$ a'_1=\frac2{0.75}=2.666666667 $$ $$ a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0 $$ $$ a'_4=\frac3{0.75}=4 $$ **Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero. ### During inference Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training. ## Drill 48 Mixed dropout and a momentum update ## Question **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1. ## Answer — step by step ### Given Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5. Linear-output weights = [0.2,0.3]; bias = 0; target = 1. Half squared loss; update only the output weights. Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1. ### To find Find dropped/scaled activations, prediction, gradients, new velocities and new output weights. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Formulas used Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients. $$ a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j $$ The signed-velocity momentum convention is: $$ v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}} $$ ### Step 1: apply inverted dropout $$ a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0 $$ ### Step 2: prediction and error $$ o=0.2(2)+0.3(0)+0=0.4 $$ $$ o-y=0.4-1=-0.6 $$ ### Step 3: current weight gradients $$ g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0 $$ ### Step 4: remembered movement plus new correction $$ v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21 $$ $$ v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09 $$ ### Step 5: add velocities to old weights $$ w_{1,\mathrm{new}}=0.2+0.21=0.41 $$ $$ w_{2,\mathrm{new}}=0.3-0.09=0.21 $$ **Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero. --- ## TEE · Convolution output dimensions Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/30/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/30 This formula counts how many legal filter positions fit along one image dimension. Padding adds space at both ends, the kernel occupies K positions, and stride controls how far the next window moves. The floor keeps only complete valid positions. $$ N_{out}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ ## Reading image shapes before doing CNN arithmetic **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension. ### Explanation and working ### Height, width and channels The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors. ![Count height, width and channels separately. Three channels are stacked at the same spatial positions.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_shape.svg) Diagram: Count height, width and channels separately. Three channels are stacked at the same spatial positions.. Labels: Read height, width and channels separately | Read height, width and channels separately | The text uses 4 × 5 × 3: four rows, five columns and three channels. | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 3 stacked channels | H × W × C | H = 4 rows | W = 5 columns | C = 3 channels | Total entries = 60 | A filter spans the incoming channels. Its spatial width is a different quantity. Count height, width and channels separately. Three channels are stacked at the same spatial positions. Tap the diagram to enlarge. ### One position versus the whole map A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels. ### Why filter depth matches input channels A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels. ### Reading a complete shape calculation The spatial-size formula computes height and width. It does not compute the channel count. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position. ### Shape, parameters and computation are three different counts **Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights. In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count. ## Stride padding and floor **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output. ### Explanation and working Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension. ![The border adds to both sides of the input. The stride changes the next starting position of the filter window.](https://deep-learning-study-room.vercel.app/assets/figures/v_stride_padding.svg) Diagram: The border adds to both sides of the input. The stride changes the next starting position of the filter window.. Labels: Padding adds a border. Stride moves the window. | Padding adds a border. Stride moves the window. | Example: a 3 × 3 input becomes 5 × 5 after one zero border. | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 3 | 0 | 0 | 4 | 5 | 6 | 0 | 0 | 7 | 8 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | Padding P = 1 | 0 | 1 | 2 | 3 | 4 | Stride S = 2 | Starting positions: 0, 2, … | For each spatial dimension: add 2P, subtract K, divide by S, take floor, then add | 1. The border adds to both sides of the input. The stride changes the next starting position of the filter window. Tap the diagram to enlarge. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: $\left\lfloor 2.5\right\rfloor$ = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels. **Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1. ### Output dimension with a fractional intermediate result **Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.** ### Given N = 6; K = 3; S = 2; P = 0; dilation = 1. ### To find The integer output side length. ### Formula — which one and why The formula counts complete kernel placements. Flooring removes an incomplete final placement. N is input side; P is padding on each side; K is kernel side; S is stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1. ### Solution — step by step **Step 1: include padding and subtract kernel size.** $$ 6+2(0)-3=3 $$ **Step 2: divide by the stride.** $$ \frac32=1.5 $$ **Step 3: take the floor.** $$ \lfloor1.5\rfloor=1 $$ **Step 4: count the first placement.** $$ N_{\mathrm{out}}=1+1=2 $$ **Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter. ### Padding on both sides **Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.** ### Given N = 5; K = 3; S = 1; P = 1 on each side. ### To find Padded input and output dimensions. ### Formula — which one and why Padding adds P entries on each of two opposite sides; the output counts complete kernel placements. N = unpadded side; K = kernel side; S = stride; P = per-side padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\lfloor(N+2P-K)/S\rfloor+1 $$ **Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain. ### Solution — step by step **Step 1: count both padding edges.** $$ 2P=2(1)=2 $$ **Step 2: calculate the padded input side.** $$ N+2P=5+2=7 $$ **Step 3: subtract kernel width.** $$ 7-3=4 $$ **Step 4: divide by stride and take the floor.** $$ \lfloor4/1\rfloor=4 $$ **Step 5: include the first placement.** $$ N_{out}=4+1=5 $$ **Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter. “Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case. ## Even-sized kernels: SAME padding is not always symmetric **Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides. For a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention. **Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho. ### Example 1: one entry on every side **Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)** ### Given Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter. ### To find The padded input dimensions and the convolution output dimensions. ### Formula — which one and why The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel. ### Solution — step by step **Step 1: count the two padding borders.** $$ 2P=2(1)=2 $$ **Step 2: find the padded side length.** $$ N_{\mathrm{padded}}=5+2=7 $$ The padded input is 7 × 7 × 1. **Step 3: subtract the kernel size.** $$ 7-2=5 $$ **Step 4: divide by stride, take the floor and count the first placement.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5}{1}\right\rfloor+1=5+1=6 $$ **Step 5: write height, width and channels.** Both spatial dimensions equal 6. There is one filter, so there is one output channel. **Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel. ### Example 2: preserve the size with SAME padding **Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)** ### Given Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found. ### To find Padding above, below, left and right; padded-input shape; final output shape. ### Formula — which one and why For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+P_{\mathrm{before}}+P_{\mathrm{after}}-K}{S}\right\rfloor+1 $$ At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1. $$ P_{\mathrm{total}}=P_{\mathrm{before}}+P_{\mathrm{after}}=K-1 $$ ### Solution — step by step **Step 1: calculate total padding along the height.** $$ P_{\mathrm{total}}=2-1=1 $$ Only one extra row in total is needed. **Step 2: place the row using the stated convention.** Padding above = 0; padding below = 1. **Step 3: apply the same calculation to width.** Only one extra column in total is needed. Padding left = 0; padding right = 1. **Step 4: find the padded-input shape.** $$ H_{\mathrm{padded}}=5+0+1=6 $$ $$ W_{\mathrm{padded}}=5+0+1=6 $$ The padded input is 6 × 6 × 1. **Step 5: count the output positions along each axis.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5+0+1-2}{1}\right\rfloor+1=4+1=5 $$ **Step 6: include the output channel.** One filter creates one output channel. **Final answer:** add **0 above, 1 below, 0 left and 1 right**; padded input = **6 × 6 × 1**; output = **5 × 5 × 1**. The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula. ## Drill 35 Padding stride and channels ## Question **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total. ## Answer — step by step ### Given Input shape = 31 × 31 × 3. Filter size = 5 × 5; number of filters = 8. Stride S = 2; padding P = 1 per side; one bias per filter. ### To find Find output shape, weights, biases and total parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning of each given value The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels. ### Formulas used $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ \text{weights}=K^2C_{\mathrm{in}}C_{\mathrm{out}},\quad \text{biases}=C_{\mathrm{out}} $$ $$ \text{parameters}=\text{weights}+\text{biases} $$ ### Step 1: find the output height and width Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33. $$ N_{\mathrm{out}}=\left\lfloor\frac{31+2(1)-5}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{28}{2}\right\rfloor+1=14+1=15 $$ The height uses the same calculation. **Output shape = 15 × 15 × 8.** ### Step 2: count weights in one filter Each filter covers 5 × 5 positions in each of 3 channels. $$ 5\times5\times3=75\text{ weights per filter} $$ ### Step 3: count all filters and their biases $$ \text{weights}=75\times8=600 $$ $$ \text{biases}=8,\qquad \text{total}=600+8=608 $$ **Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights. --- ## TEE · Convolution value Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/31/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/31 One convolution output is a sum of matched input–kernel products. Move the window and repeat that exact rule for the next output. With several input channels, add contributions from every channel and add the filter bias once. $$ Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o $$ ## One filter over multiple channels: all contributions go into one cell **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels. ### Explanation and working ### 1. Separate spatial size from depth Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3. ![Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.](https://deep-learning-study-room.vercel.app/assets/figures/v_channel_products.svg) Diagram: Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.. Labels: A 1 × 1 filter still mixes all three channels | A 1 × 1 filter still mixes all three channels | Use input [2, 5, 1], weights [0.1, −0.2, 0.5] and one bias 0.3. | Channel 1 | 2 × 0.1 = 0.2 | Channel 2 | 5 × (−0.2) = −1 | Channel 3 | 1 × 0.5 = 0.5 | One filter → one output value | 0.2 + (−1) + 0.5 + 0.3 = 0 | One spatial position can contain several channel values. The one-by-one filter | mixes them. Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example. Tap the diagram to enlarge. $$ z=0.1(2)+(-0.2)(5)+0.5(1)+0.3 $$ $$ z=0.2-1+0.5+0.3=0 $$ The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels. ### 2. A larger filter has one spatial grid per input channel A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias. The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified. ### 3. What changes when there are several filters? Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth. For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters. ### 4. One-by-one does not mean one weight A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed. This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately. ## Faculty example: a complete binary-image convolution **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)** ### Given Image X = $\begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}$ . Filter K = $\begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}$ . These are the values in the Unit 4 lecture example. ### To find The output shape and all nine feature-map values. ### Formula — which one and why The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window. ![Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.](https://deep-learning-study-room.vercel.app/assets/figures/u4_binary_windows.svg) Diagram: Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.. Labels: u4 binary windows | One filter, nine window positions | Input: 5 × 5 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | Shared filter: 3 × 3 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | → | Output: 3 × 3 | 4 | 3 | 4 | 2 | 4 | 3 | 2 | 3 | 4 | Orange window → orange output cell = 4. Move one cell at a time. Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions. Tap the diagram to enlarge. ### Solution — step by step **Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3. **Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**. **Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**. **Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**. **Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**. **Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**. **Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**. **Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**. **Final answer:** Y = $\begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}$ . All nine outputs use the same nine filter weights. They do not introduce nine different filters. **Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain. ## Faculty example: combine three RGB channel sums **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)** ### Given Red patch = $\begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}$ ; red kernel = $\begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}$ . Green patch = $\begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}$ ; green kernel = $\begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}$ . Blue patch = $\begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}$ ; blue kernel = $\begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}$ . Bias = 1. These are three slices of **one** full-depth filter, not three independent output filters. ### To find The contribution of each input channel and the final scalar output at this position. ### Formula — which one and why Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions. $$ z=s_R+s_G+s_B+b $$ The quantities s\_R, s\_G and s\_B are intermediate weighted sums, not separate final output channels. ### Solution — step by step **Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1. **Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**. **Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1). **Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**. **Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1. **Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**. **Step 7: add the channel contributions.** 308 −498 +164 = −26. **Step 8: add one filter bias.** −26 +1 = **−25**. **Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages. **Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo. ## The convolution calculation **Definition.** A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention. ### Explanation and working In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it. **Question.** Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias. ### Given Input X = $\begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}$ . Kernel K = $\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$ . Stride = 1; padding = 0; bias = 0; no kernel flip. ### To find Find the 2 × 2 output map. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_cd39cabe48.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 0 | 0 | −1 | Matrix 2 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 Given matrices for this question Tap the diagram to enlarge. $$ X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix} $$ **Step 1. Top-left window.** 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4. **Step 2. Move right by one.** 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4. **Step 3. Move down and return left.** 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4. **Step 4. Bottom-right window.** 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is $\begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}$ . If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested. ![Four windows produce four output cells](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: Four windows produce four output cells. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. Four windows produce four output cells Tap the diagram to enlarge. ## Drill 34 Convolution with a negative coefficient ## Question **Question.** Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | --- | --- | | 1 | −1 | | 0 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel K are printed with the question. Stride S = 1; padding P = 0; bias = 0. Use the kernel without flipping it. ### To find Calculate all nine output cells. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−2) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(-1)+(1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 3 = 3** **2 + 3 = 5** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(-1)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(3)=5 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 2 = 2** **(−1) + 2 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(-1)+(2)=1 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(-1)+(1)=0 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −3 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−3) = −2** Row 2**0 + 0 = 0** **(−2) + 0 = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(-1)=-2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(-2)+(0)=-2 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + (−2) = 1** Row 2**0 + 1 = 1** **1 + 1 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(1)+(1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | −1 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + (−1) = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(1)+(0)=1 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 2 = 2** **1 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(2)=3 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 3 = 3** **(−1) + 3 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(-1)+(3)=2 $$ **Final output:** $\begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ## Drill 46 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. --- ## TEE · Max, min and average pooling Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/32/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/32 Pooling reduces each window to one value. Max pooling keeps the largest entry; average pooling adds all included entries and divides by their count. Pooling is performed independently in each channel. $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{max}=\max_{(i,j)\in W}x_{ij} $$ $$ y_{avg}=\frac1{|W|}\sum_{(i,j)\in W}x_{ij} $$ ## Pooling needs comparisons or averages **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases. ### Explanation and working Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged. **Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding: ### Given The 4 × 4 input map is printed below. Window = 2 × 2; stride = 2; padding = 0. ### To find Find the pooled output by the operation(s) requested. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step $$ A=\begin{bmatrix}21&11&15&10\\12&9&10&11\\11&13&18&12\\7&4&15&12\end{bmatrix} $$ ### Top-left window **Step 1: select the four entries.** [21, 11, 12, 9] **Step 2: choose the largest entry.** $$ y_{\max}=21 $$ **Step 3: add all entries.** $$ 21+11+12+9=53 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{53}4=13.25 $$ ### Top-right window **Step 1: select the four entries.** [15, 10, 10, 11] **Step 2: choose the largest entry.** $$ y_{\max}=15 $$ **Step 3: add all entries.** $$ 15+10+10+11=46 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{46}4=11.5 $$ ### Bottom-left window **Step 1: select the four entries.** [11, 13, 7, 4] **Step 2: choose the largest entry.** $$ y_{\max}=13 $$ **Step 3: add all entries.** $$ 11+13+7+4=35 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{35}4=8.75 $$ ### Bottom-right window **Step 1: select the four entries.** [18, 12, 15, 12] **Step 2: choose the largest entry.** $$ y_{\max}=18 $$ **Step 3: add all entries.** $$ 18+12+15+12=57 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{57}4=14.25 $$ Outputs are max $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ and average $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . ![Each pooling window reduces several values to one](https://deep-learning-study-room.vercel.app/assets/figures/pooling.svg) Diagram: Each pooling window reduces several values to one. Labels: INPUT 4 × 4 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 | MAX POOL | 21 | 15 | 13 | 18 | AVERAGE POOL | 13.25 | 11.5 | 8.75 | 14.25 | 2 × 2 windows | stride 2 Each pooling window reduces several values to one Tap the diagram to enlarge. Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance. ## Max, min and average pooling on the same lecture grid **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases. **Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)** ### Given X = $\begin{bmatrix}12 & 20 & 30 & 0\\8 & 12 & 2 & 0\\34 & 70 & 37 & 4\\112 & 100 & 25 & 12\end{bmatrix}$ . Window = 2 × 2; stride = 2; padding = 0. ### To find The shared output shape and each of the four cells for each pooling method. ### Formula — which one and why Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap. $$ y_{\max}=\max(x_1,x_2,x_3,x_4) $$ $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{\mathrm{avg}}=\frac{x_1+x_2+x_3+x_4}{4} $$ ![The four colours identify four separate pooling windows. Each window creates one cell in each output map.](https://deep-learning-study-room.vercel.app/assets/figures/u4_pooling_windows.svg) Diagram: The four colours identify four separate pooling windows. Each window creates one cell in each output map.. Labels: u4 pooling windows | Same four windows, three different summaries | Input: 4 × 4 | 12 | 20 | 30 | 0 | 8 | 12 | 2 | 0 | 34 | 70 | 37 | 4 | 112 | 100 | 25 | 12 | Max | 20 | 30 | 112 | 37 | Min | 8 | 0 | 34 | 4 | Average | 13 | 8 | 79 | 19.5 | Bottom-right average: (37 + 4 + 25 + 12) ÷ 4 = 19.5 | Pooling changes values and shape; ordinary pooling learns no parameters. The four colours identify four separate pooling windows. Each window creates one cell in each output map. Tap the diagram to enlarge. ### Solution — step by step **Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2. **Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13. **Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8. **Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79. **Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20. **Step 6: put each result back in its window's position.** Max = $\begin{bmatrix}20 & 30\\112 & 37\end{bmatrix}$ . Min = $\begin{bmatrix}8 & 0\\34 & 4\end{bmatrix}$ . Average = $\begin{bmatrix}13 & 8\\79 & 19.5\end{bmatrix}$ . **Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation. ### Why the choice matters Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour. Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter. **Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide. ## Drill 36 Max and average pooling ## Question **Question.** Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_563e4ee8f7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | A | 1 | 4 | 2 | 3 | 0 | 2 | 5 | 1 | 6 | 3 | 2 | 0 | 1 | 7 | 4 | 8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input A is printed with the question. Window size = 2 × 2; stride = 2; no padding. ### To find Find the complete max-pooling and average-pooling outputs. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 4, 0, 2] **Step 2: add all selected entries.** 1 + 4 + 0 + 2 = 7 **Step 3: divide by the 4 entries in this window.** 7/4 = **1.75**. Maximum = **4** ### Pooling window (1,2) **Step 1: select the entries.** [2, 3, 5, 1] **Step 2: add all selected entries.** 2 + 3 + 5 + 1 = 11 **Step 3: divide by the 4 entries in this window.** 11/4 = **2.75**. Maximum = **5** ### Pooling window (2,1) **Step 1: select the entries.** [6, 3, 1, 7] **Step 2: add all selected entries.** 6 + 3 + 1 + 7 = 17 **Step 3: divide by the 4 entries in this window.** 17/4 = **4.25**. Maximum = **7** ### Pooling window (2,2) **Step 1: select the entries.** [2, 0, 4, 8] **Step 2: add all selected entries.** 2 + 0 + 4 + 8 = 14 **Step 3: divide by the 4 entries in this window.** 14/4 = **3.5**. Maximum = **8** **Average output:** $\begin{bmatrix}1.75 & 2.75\\4.25 & 3.5\end{bmatrix}$ . **Max output:** $\begin{bmatrix}4 & 5\\7 & 8\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. ## Drill 46 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. --- ## TEE · Convolution parameter count Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/33/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/33 A filter has one learned weight for each spatial position in each input channel. Count the weights in one filter, add its one bias, then multiply by the number of filters. Sliding a filter to many positions reuses these weights; it does not create new parameters. $$ P_{conv}=(K_hK_wC_{in}+1)C_{out} $$ ## Output size and parameter count are different **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters. ### Explanation and working **Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias. ### Given Input = 7 × 7 × 3; two 3 × 3 filters. Stride = 2; padding = 1; include biases. ### To find Find the output shape and trainable parameter count. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step **Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2. ![Output size counts produced values; parameter count counts adjustable weights and biases.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Output size counts produced values; parameter count counts adjustable weights and biases.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Output size counts produced values; parameter count counts adjustable weights and biases. Tap the diagram to enlarge. **Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters. ## Model answer: shape, activations and parameters ## Question **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them. ## Answer — step by step Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions. ![Keep output shape, activation count and parameter count as three separate quantities.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Keep output shape, activation count and parameter count as three separate quantities.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Keep output shape, activation count and parameter count as three separate quantities. Tap the diagram to enlarge. ### Spatial output shape For an ordinary convolution with dilation 1: $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters. ### Output activations $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ This is the number of output values for one example, excluding any batch dimension. ### Trainable parameters $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions. ### Small example A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels. ## Faculty CNN table: shapes, activations and 62958 parameters **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)** ### Given Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question. ### To find Output shape and number of output values for each layer, its trainable parameters, and the total parameters. ### Formula — which one and why Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ A counts produced activations. P\_conv and P\_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here. ### Solution — step by step **Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0. **Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272. **Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**. **Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight. **Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600. **Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**. **Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters. **Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations. **Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations. **Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation. **Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**. | Layer | Output shape | Activations | Parameters | | --- | --- | --- | --- | | Input | 32 × 32 × 3 | 3,072 | 0 | | Conv1 | 28 × 28 × 8 | 6,272 | 608 | | Pool1 | 14 × 14 × 8 | 1,568 | 0 | | Conv2 | 10 × 10 × 16 | 1,600 | 3,216 | | Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 | | Dense 120 | 120 | 120 | 48,120 | | Dense 84 | 84 | 84 | 10,164 | | Dense 10 → softmax | 10 | 10 | 850 + 0 | **Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network. **Yaad rakho:** Shape aage carry karo; weights aur activations ko alag columns mein rakho. ## Parameters are different from activation values **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters. ### Explanation and working An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights. ![The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations. Tap the diagram to enlarge. $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation. ### One filter, then all filters **Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.** ### Given Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter. ### To find Weights, biases, total trainable parameters and output channels. ### Formula — which one and why Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights. Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters. $$ P=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply. ### Solution — step by step **Step 1: count weights in one filter.** $$ 3(3)(3)=27 $$ **Step 2: include its one bias.** $$ 27+1=28 $$ **Step 3: count all weights.** $$ 27(16)=432 $$ **Step 4: count all biases.** $$ 1(16)=16 $$ **Step 5: add and state channel count.** $$ P=432+16=448 $$ Each filter produces one channel, so output channels = 16. **Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels. $$ P_{\mathrm{pool}}=0,\quad P_{\mathrm{flatten}}=0,\quad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values. ## Drill 37 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. --- ## TEE · Flatten and dense parameter count Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/34/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/34 Flattening rearranges all activation values into a list. It learns no new numbers. A dense layer then connects every value in that list to every receiving neuron; each receiving neuron also has one bias. $$ N_{flat}=HWC $$ $$ P_{dense}=(N_{in}+1)N_{out} $$ ## Counting parameters in a dense network **Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases. ### Deriving the parameter count For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters: $$ P_{\mathrm{dense}}=mn+n=(m+1)n $$ Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram. ![Count all input-to-neuron connections and then one bias per receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/v_dense_count.svg) Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters. Count all input-to-neuron connections and then one bias per receiving neuron. Tap the diagram to enlarge. ### Complete numerical **Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters. ### Given Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs. Every hidden/output neuron has one bias. ### To find Count weights and biases at each receiving layer, then the total. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step **Step 1: identify the consecutive layer widths.** $$ 10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4 $$ There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters. **Step 2: count weights and biases for each transformation.** | Transformation | Weights | Biases | Layer total | | --- | --- | --- | --- | | Input → hidden 1 | 10 × 5 = 50 | 5 | 55 | | Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 | | Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 | | Hidden 3 → output | 5 × 4 = 20 | 4 | 24 | **Step 3: add the layer totals.** $$ P=55+30+30+24=139 $$ Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139. **Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network. ## One complete CNN pipeline **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron. ### Given Input = 128 × 128 × 3. Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2. Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2. Flatten → dense 240 → dense 120 → dense 5. One bias per learned filter/dense neuron. ### To find Find every output shape and layer parameter count, then the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step **Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448. ![Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.](https://deep-learning-study-room.vercel.app/assets/figures/v_cnn_pipeline.svg) Diagram: Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.. Labels: Follow the complete CNN question, one layer at a time | Follow the complete CNN question, one layer at a time | Start with 128 × 128 × 3. Shape and learned-parameter count are separate. | LAYER | OUTPUT SHAPE / LENGTH | PARAMETERS | 1. Convolution 1 | 126 × 126 × 16 | 448 | 2. Pooling 1 | 63 × 63 × 16 | 0 | 3. Convolution 2 | 61 × 61 × 32 | 4,640 | 4. Pooling 2 | 30 × 30 × 32 | 0 | 5. Flatten | 28,800 | 0 | 6. Dense 240 | 240 | 6,912,240 | 7. Dense 120 | 120 | 28,920 | 8. Classifier | 5 | 605 | Total learned parameters = 6,946,853 | Each output is the next layer’s input. Pooling and flattening add no learned | parameters. Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count. Tap the diagram to enlarge. **Step 2. Pool 1 size.** $\left\lfloor\frac{126 - 2}{2}\right\rfloor+1$ = 63; output 63 × 63 × 16. Parameters = 0. **Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640. **Step 4. Pool 2 size.** $\left\lfloor\frac{61 - 2}{2}\right\rfloor+1$ = $\left\lfloor 29.5\right\rfloor$ + 1 = 30; output 30 × 30 × 32. Parameters = 0. **Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0. **Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240. **Step 7. Dense 120.** (240 + 1) × 120 = 28,920. **Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer. **Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer. ## Drill 37 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. ## Drill 49 Mixed CNN dimensions softmax and loss ## Question **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE. ## Answer — step by step ### Given Input = 8 × 8 × 3. Conv: kernel 3, stride 1, padding 1, 2 filters. Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs. Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]). ### To find Find shapes, parameter counts, probabilities and CCE. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Formulas used Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}} $$ Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.367879441, 1, 0.135335283]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Correct class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**. --- ## TEE · Convolution multiply-accumulate count Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/35/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/35 A parameter count counts learned numbers. A multiply-accumulate count counts how many times input–weight products are used. One filter is reused at every output position, so computation includes output height and width even though its parameter count does not. $$ MAC=H_{out}W_{out}K_hK_wC_{in}C_{out} $$ ## Why one by one convolution saves work **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts. ![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg) Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications. Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge. **Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.** ### Given Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size. ### To find Direct MACs, total two-stage bottleneck MACs and percentage saved. ### Formula — which one and why Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs. ### Solution — step by step **Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400 $$ **Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels. $$ \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448 $$ **Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels. $$ \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200 $$ **Step 4: add the two bottleneck costs.** $$ \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648 $$ **Step 5: calculate the saving.** $$ \mathrm{saving}=120422400-12443648=107978752 $$ $$ \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\% $$ **Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption. ### A second bottleneck with different dimensions **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1. ### To find Direct MACs, bottleneck MACs and saving. ### Formula — which one and why Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers. ### Solution — step by step **Step 1: direct spatial convolution.** $$ 8(8)(8)(3)(3)(16)=73728 $$ **Step 2: 1 × 1 reduction.** $$ 8(8)(4)(1)(1)(16)=4096 $$ **Step 3: reduced-depth 3 × 3 convolution.** $$ 8(8)(8)(3)(3)(4)=18432 $$ **Step 4: add the stages and subtract from the direct cost.** $$ 4096+18432=22528 $$ $$ 73728-22528=51200 $$ **Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values. ## The faculty's 480-channel bottleneck numerical ![Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.](https://deep-learning-study-room.vercel.app/assets/figures/u4_bottleneck_480.svg) Diagram: Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.. Labels: u4 bottleneck 480 | Same output shape, different computation | Input | 14 × 14 × 480 | → | Direct 5 × 5 | 48 filters | → | Output | 14 × 14 × 48 | 112,896,000 MACs | 1 × 1 reduction | 480 → 16 channels | → | Reduced 5 × 5 | 16 → 48 channels | → | Output | 14 × 14 × 48 | 1,505,280 + 3,763,200 = 5,268,480 MACs | The lower path also starts from 14 × 14 × 480. Count both layers. Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers. Tap the diagram to enlarge. **Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.** ### Given Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters. ### To find MACs for the direct path, both bottleneck stages, their total and percentage saved. ### Formula — which one and why $$ M=H_oW_oC_oK^2C_i $$ M is the MAC count. H\_oW\_oC\_o counts output values; K²C\_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck. $$ \mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}} $$ ### Solution — step by step **Step 1: direct output count.** 14 × 14 × 48 = 9,408. **Step 2: products per direct output.** 5 × 5 × 480 = 12,000. **Step 3: direct MACs.** $$ 9408(12000)=112896000 $$ **Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values. $$ 14(14)(16)(480)=1505280 $$ **Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values. $$ 9408(400)=3763200 $$ **Step 6: add both bottleneck costs.** $$ 1505280+3763200=5268480 $$ **Step 7: calculate the saving.** $$ 112896000-5268480=107627520 $$ $$ 100\frac{107627520}{112896000}\approx95.333333\% $$ **Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone. ## Drill 39 Bottleneck cost with new values ## Question **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first. ## Answer — step by step ### Given Input = 16 × 16 × 64. Direct branch: 32 same-padded 3 × 3 filters. Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer. Spatial size remains 16 × 16; exclude bias additions. ### To find Find both MAC counts, absolute saving and percentage saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Meaning and formula A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ ### Step 1: direct 3 × 3 convolution Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 reduction from 64 to 8 channels $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ ### Step 3: 3 × 3 convolution now receives only 8 channels $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: add the two bottleneck stages $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: calculate the saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32. ### Bottleneck computation on an 8 × 8 input **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8 in both stages. Input channels = 16; output filters = 8; main kernel = 3 × 3. Bottleneck uses 4 filters of 1 × 1. ### To find Direct MACs, two-stage bottleneck MACs and savings. ### Formula — which one and why Each output scalar requires one multiply-accumulate per incoming kernel entry. A bottleneck reduces the input depth of the expensive spatial convolution. One MAC is one multiply-accumulate pair. If counting multiplication and addition separately, the operation count is approximately twice the MAC count. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K_hK_wC_{\mathrm{in}} $$ **Yaad rakhne ka cue:** Spatial kernel se pehle channels kam; dono stages ka cost add. ### Solution — step by step **Step 1: count output positions per channel.** $$ 8(8)=64 $$ **Step 2: count direct work per output scalar.** $$ 3(3)(16)=144 $$ **Step 3: count the full direct layer.** $$ 64(8)(144)=73728 $$ **Step 4: count the 1 × 1 reduction.** $$ 64(4)(1)(1)(16)=4096 $$ **Step 5: count the reduced-depth 3 × 3 layer.** $$ 64(8)(3)(3)(4)=18432 $$ **Step 6: sum both bottleneck stages.** $$ 4096+18432=22528 $$ **Step 7: subtract to obtain savings.** $$ 73728-22528=51200 $$ **Final answer:** Direct = 73,728 MACs; bottleneck = 22,528 MACs; savings = 51,200 MACs. --- ## TEE · Concatenation and residual addition Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/36/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/36 Concatenation and addition combine branches differently. Concatenation places channel groups next to each other and increases the channel count. Residual addition adds matching entries, so the two full shapes must be compatible. $$ C_{concat}=\sum_b C_b $$ $$ y_{res}=F(x)+x $$ ## Inception concatenation and residual addition **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths. ### Explanation and working Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations. ![Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.](https://deep-learning-study-room.vercel.app/assets/figures/v_join_add.svg) Diagram: Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.. Labels: Concatenation and addition produce different shapes | Concatenation and addition produce different shapes | Use height and width 14 × 14 throughout. | CONCATENATE four branches | Channels: 16 + 32 + 8 + 8 | Result: 14 × 14 × 64 | Stack all 64 channels. | ADD two matching tensors | 14 × 14 × 64 + 14 × 14 × 64 | Result: 14 × 14 × 64 | Add each matching pair. | A 14 × 14 × 32 shortcut needs a projection before addition to a 14 × 14 × 64 | branch. Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape. Tap the diagram to enlarge. Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64. A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition. ## ResNet adds a shortcut **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x. $$ y=F(x)+x $$ If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch. ![The two paths must have equal output shape before elementwise addition.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge. ### Effect on gradient propagation For the identity-shortcut formulation, the derivative includes a direct identity contribution. $$ \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I $$ This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients. ### Projection when dimensions differ If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride. $$ y=F(x)+W_sx $$ An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition. **Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum. ### Given x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3]. ### To find Find the three components of x+F(x). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. ### Solution — step by step **Step 1: add the first matching pair.** $$ 1+0.1=1.1 $$ **Step 2: add the second matching pair.** $$ 2+(-0.2)=1.8 $$ **Step 3: add the third matching pair.** $$ 3+0.3=3.3 $$ **Step 4: assemble in the original order.** $$ y=[1.1,1.8,3.3] $$ The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum. ## Drill 51 Inception concatenation and parameter count ## Question **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters. ## Answer — step by step ### Given Input = 10 × 10 × 8. Branch A: 1 × 1 convolution, 4 filters. Branch B: 3 × 3 same convolution, 6 filters. Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters. ### To find Find each branch shape and parameters, then the concatenated shape and total parameters. ### Formula — which one and why **Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches. C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions. $$ C_{\mathrm{out}}=\sum_k C_k $$ $$ P_{\mathrm{total}}=\sum_k P_k $$ **Yaad rakhne ka cue:** Channels jodo; height-width same rakho. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values. $$ C_{\mathrm{combined}}=C_A+C_B+C_C $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Assume one bias per filter. Pooling has no learned parameters. **Step 1: Branch A.** Its 4 filters produce shape 10 × 10 × 4. $$ P_A=(1\times1\times8+1)\times4=9\times4=36 $$ **Step 2: Branch B.** Its 6 filters produce shape 10 × 10 × 6. $$ P_B=(3\times3\times8+1)\times6=73\times6=438 $$ **Step 3: Branch C.** Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2. $$ P_C=0+(1\times1\times8+1)\times2=18 $$ **Step 4: Join the shapes and add the parameters.** $$ C_{\mathrm{combined}}=4+6+2=12 $$ $$ P_{\mathrm{total}}=36+438+18=492 $$ **Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches. ## Drill 52 Projection shortcut ## Question **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters. ## Answer — step by step ### Given Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16. Shortcut kernel = 1 × 1; stride 2; padding 0. Use one bias per projection filter. ### To find Choose the number of filters and find projection output shape and parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ ### Step 1: choose 16 filters One filter creates one channel, so 16 filters give the required 16 output channels. ### Step 2: use the specified 1 × 1 kernel and stride 2 $$ N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8 $$ The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1. ### Step 3: count weights and biases $$ \text{weights}=1\times1\times8\times16=128 $$ $$ \text{biases}=16,\qquad \text{total}=128+16=144 $$ **Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid. ## Drill 40 Residual addition ## Question **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block. ## Answer — step by step ### Given x = [2,−3,1]; F(x) = [−1,1,0.5]. ### To find Find x+F(x), apply ReLU, and compare with ReLU(F(x)). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join. $$ H(x)=x+F(x),\qquad a_j=\max(0,H_j) $$ ### Step 1: add each pair $$ H_1=2+(-1)=1 $$ $$ H_2=-3+1=-2 $$ $$ H_3=1+0.5=1.5 $$ The residual output before ReLU is **[1, −2, 1.5]**. ### Step 2: apply ReLU $$ [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5] $$ ### Step 3: compare with the plain branch Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**. Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation. --- ## TEE · Vanilla RNN Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/37/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/37 A recurrent state uses two sources: the current input and the previous hidden state. Transform each with its own weight, add the contributions and bias, and apply tanh. The activated state—not its raw sum—is carried into the next time step. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h) $$ $$ o_t=W_{hy}h_t+b_y $$ ## The vanilla RNN equations from scratch **Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction. ### Hidden-state and output equations For a tanh recurrent layer: $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h $$ $$ h_t=\tanh(a_t) $$ aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit. For classification at time t, an output layer can produce logits and probabilities: $$ z_t=W_{hy}h_t+b_y,\qquad p_t=\operatorname{softmax}(z_t) $$ A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation. ![The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state. Tap the diagram to enlarge. ### Complete numerical: two recurrent time steps **Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation. ### Given h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0. x₁ = 1; x₂ = 2; activation = tanh. ### To find Find h₁ and h₂ in order. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step **Step 1: Time step 1 — calculate the preactivation.** $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ **Apply the recurrent activation.** $$ h_1=\tanh(0.5)\approx0.462117157 $$ This activated value, not a₁ = 0.5, is passed to the next time step. **Step 2: Time step 2 — combine the new input and previous state.** $$ a_2=0.5(2)+0.2h_1+0 $$ $$ a_2\approx1+0.2(0.462117157)\approx1.092423431 $$ **Apply tanh to the complete new preactivation.** $$ h_2=\tanh(a_2)\approx0.797760954 $$ | Time step | Current input | Previous state | Preactivation | New state | | --- | --- | --- | --- | --- | | 1 | 1 | 0 | 0.5 | 0.462117157 | | 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 | **Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update. ## A complete vector RNN calculation **Question from R26 Q5a.** xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons. ### Given Input x = [0,1]ᵀ; previous h = [0,1]ᵀ. Wxh = $\begin{bmatrix}0.6 & 0.4\\0.3 & 0.7\end{bmatrix}$ . Whh = $\begin{bmatrix}0.5 & 0.3\\0.2 & 0.4\end{bmatrix}$ ; bias = 0. Apply tanh separately to the two sums. ### To find Find the new two-component hidden state; output-layer parameters are not supplied. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_deac39f14c.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Wxh | 0.6 | 0.4 | 0.3 | 0.7 | Whh | 0.5 | 0.3 | 0.2 | 0.4 Given matrices for this question Tap the diagram to enlarge. **Step 1 new input:** Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7]. **Step 2 old memory:** Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4]. **Step 3 add:** a = [0.4+0.3, 0.7+0.4] = [0.7,1.1]. **Step 4 activate each element:** h = [tanh(0.7), tanh(1.1)] = **[0.604368,0.800499]**. An output-layer prediction cannot be numerically calculated without its weights and bias. ## Drill 41 Two RNN steps ## Question **Question.** h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂. ## Answer — step by step ### Given Initial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0. Inputs: x₁ = 1, x₂ = 2; activation: tanh. ### To find Calculate h₁, then use it to calculate h₂. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ### Meaning and formulas At time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\qquad h_t=\tanh(a_t) $$ The raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward. $$ \tanh(a)=\frac{e^{2a}-1}{e^{2a}+1} $$ ### Step 1: calculate the first raw sum $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ ### Step 2: activate it to obtain the first state $$ h_1=\tanh(0.5)=\frac{e^1-1}{e^1+1} $$ $$ h_1=\frac{2.718281828-1}{2.718281828+1}=0.462117157 $$ ### Step 3: use that state in the second raw sum $$ a_2=0.5(2)+0.2(0.462117157)+0 $$ $$ a_2=1+0.092423431=1.092423431 $$ ### Step 4: activate the second sum $$ h_2=\tanh(1.092423431)=0.797760954 $$ **Final answer:** h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5. --- ## TEE · Perceptron update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/38/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/38 The perceptron uses a threshold to choose a class and an error-based learning rule to change weights. Its threshold convention must be stated, especially at zero. This discrete learning rule is different from sigmoid backpropagation. $$ \widehat y=\operatorname{step}(w^Tx+b) $$ $$ w_{new}=w+\eta(y-\widehat y)x $$ $$ b_{new}=b+\eta(y-\widehat y) $$ ## Perceptrons and the threshold A perceptron is a simple neuron used for classification with a threshold decision. If a question says output 1 for z ≥ 0 and 0 otherwise, then z = 0 gives 1. If it says output 1 for z > 0 and 0 otherwise, then z = 0 gives 0. That single equality sign can change an entire training table. ![The weighted sum is a real number. The threshold turns that number into a hard decision.](https://deep-learning-study-room.vercel.app/assets/figures/v_threshold.svg) Diagram: The weighted sum is a real number. The threshold turns that number into a hard decision.. Labels: A perceptron makes a threshold decision | A perceptron makes a threshold decision | Example threshold: return 1 when the weighted sum is greater than 0. | Inputs | x₁ and x₂ | Weighted sum | z = w₁x₁ + w₂x₂ + b | Threshold | z > 0: 1; otherwise: | 0 | Unlike sigmoid, this rule makes a hard 0-or-1 decision. The weighted sum is a real number. The threshold turns that number into a hard decision. Tap the diagram to enlarge. **Question.** Implement binary AND with w₁ = w₂ = 0.6, b = −0.8 and output 1 when z ≥ 0. ### Given w₁ = w₂ = 0.6; bias b = −0.8. Binary inputs; output 1 when z ≥ 0. ### To find Calculate the output for 00,01,10,11 and show that the result is AND. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step ### Input (0, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0+0=0 $$ **Step 4: add the bias once.** $$ z=0+(-0.8)=-0.8 $$ **Step 5: apply the stated threshold.** z = -0.8; output = 0 because output is 1 exactly when z ≥ 0. ### Input (0, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0+0.6=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0.6+0=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0.6+0.6=1.2 $$ **Step 4: add the bias once.** $$ z=1.2+(-0.8)=0.4 $$ **Step 5: apply the stated threshold.** z = 0.4; output = 1 because output is 1 exactly when z ≥ 0. **Final answer:** outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND. ### Summary of the four cases | Inputs | Full net calculation | Net z | Output | AND target | | --- | --- | --- | --- | --- | | 0, 0 | 0.6(0) + 0.6(0) − 0.8 | −0.8 | 0 | 0 | | 0, 1 | 0.6(0) + 0.6(1) − 0.8 | −0.2 | 0 | 0 | | 1, 0 | 0.6(1) + 0.6(0) − 0.8 | −0.2 | 0 | 0 | | 1, 1 | 0.6(1) + 0.6(1) − 0.8 | 0.4 | 1 | 1 | The neuron fires only when both inputs are 1. No weight update was requested, so this is a forward evaluation, not a training problem. For binary targets, a common learning rule is w ← w + η(t − a)x and b ← b + η(t − a). For bipolar targets −1 and +1, the class notes use a different rule: on a mismatch, w ← w + η tx and b ← b + ηt; otherwise keep them. The notes give output 0 at zero threshold, +1 above it and −1 below it. Never combine the activation from one convention with an unexplained update from another. **Training table routine:** copy current weights → calculate z → apply the stated threshold → compare output with target → compute every weight change → update bias → carry new values to the next row. Reset nothing between rows or epochs unless instructed. ### Binary perceptron: target 1 **Question. A binary perceptron has x = [1,2], weights [-0.2, 0.1], bias 0 and target 1. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.** ### Given x = [1,2]; w = [-0.2, 0.1]; b = 0; target = 1; η = 0.1. ### To find Net, threshold output, correction and new weights/bias. ### Formula — which one and why This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias. t is target; o is threshold output; Δ means the amount added to the current parameter. $$ z=\sum_iw_ix_i+b $$ $$ \Delta w_i=\eta(t-o)x_i $$ $$ \Delta b=\eta(t-o) $$ **Yaad rakhne ka cue:** Target minus output; correct prediction ho toh correction zero. ### Solution — step by step **Step 1: calculate the net input.** $$ z=(-0.2)(1)+(0.1)(2)+0=0 $$ **Step 2: apply the threshold.** Net = 0; output = 1 under the stated ≥ 0 rule. **Step 3: calculate the correction.** $$ t-o=1-1=0 $$ **Step 4: update weight 1.** $$ w_1^{\mathrm{new}}=-0.2+0.1(0)(1)=-0.2 $$ **Step 5: update weight 2.** $$ w_2^{\mathrm{new}}=0.1+0.1(0)(2)=0.1 $$ **Step 6: update the bias.** $$ b_{\mathrm{new}}=0+0.1(0)=0 $$ **Final answer:** New weights = [-0.2,0.1]; new bias = 0. ### Binary perceptron: target 0 **Question. A binary perceptron has x = [1,2], weights [0.2, 0.1], bias 0 and target 0. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.** ### Given x = [1,2]; w = [0.2, 0.1]; b = 0; target = 0; η = 0.1. ### To find Net, threshold output, correction and new weights/bias. ### Formula — which one and why This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias. t is target; o is threshold output; Δ means the amount added to the current parameter. $$ z=\sum_iw_ix_i+b $$ $$ \Delta w_i=\eta(t-o)x_i $$ $$ \Delta b=\eta(t-o) $$ **Yaad rakhne ka cue:** Target minus output; correct prediction ho toh correction zero. ### Solution — step by step **Step 1: calculate the net input.** $$ z=(0.2)(1)+(0.1)(2)+0=0.4 $$ **Step 2: apply the threshold.** Net = 0.4; output = 1 under the stated ≥ 0 rule. **Step 3: calculate the correction.** $$ t-o=0-1=-1 $$ **Step 4: update weight 1.** $$ w_1^{\mathrm{new}}=0.2+0.1(-1)(1)=0.1 $$ **Step 5: update weight 2.** $$ w_2^{\mathrm{new}}=0.1+0.1(-1)(2)=-0.1 $$ **Step 6: update the bias.** $$ b_{\mathrm{new}}=0+0.1(-1)=-0.1 $$ **Final answer:** New weights = [0.1,-0.1]; new bias = -0.1. ## Drill 11 One perceptron learning update ## Question **Question.** Binary perceptron, w = [0,0], b = 0, x = [1,1], target 1, η = 0.5. Use output 1 if z>0 else 0. Perform one update. ## Answer — step by step ### Given w = [0,0]; b = 0; x = [1,1]; target t = 1. Learning rate η = 0.5. Output 1 only when z > 0; otherwise output 0. ### To find Perform one update and report new weights and bias. ### Formula — which one and why **Perceptron learning rule.** Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters. t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question. $$ z=\sum_iw_ix_i+b $$ $$ w_i^{\mathrm{new}}=w_i+\eta(t-o)x_i $$ $$ b^{\mathrm{new}}=b+\eta(t-o) $$ **Yaad rakhne ka cue:** Prediction nikalo; target−prediction se correction nikalo. ### Solution — step by step ### Meaning and formula A perceptron first computes a weighted sum z and then applies the exact threshold rule in the question. This question uses **strictly greater than zero**, so z = 0 produces output 0. $$ z=w_1x_1+w_2x_2+b,\qquad o=\begin{cases}1&z>0\\0&z\le0\end{cases} $$ The perceptron learning rule uses target minus output, written e below. This is a discrete perceptron rule, not the sigmoid backpropagation derivative. $$ e=t-o,\qquad w_{j,\mathrm{new}}=w_j+\eta e x_j,\qquad b_{\mathrm{new}}=b+\eta e $$ ### Step 1: obtain the old output $$ z=0\times1+0\times1+0=0,\qquad o=0 $$ ### Step 2: compare it with the target $$ e=1-0=1 $$ ### Step 3: update each weight separately $$ w_{1,\mathrm{new}}=0+0.5\times1\times1=0.5 $$ $$ w_{2,\mathrm{new}}=0+0.5\times1\times1=0.5 $$ $$ b_{\mathrm{new}}=0+0.5\times1=0.5 $$ ### Step 4: interpret the updated prediction $$ z_{\mathrm{new}}=0.5\times1+0.5\times1+0.5=1.5 $$ Since 1.5 > 0, the new output is 1. **Final parameters: w = [0.5, 0.5], b = 0.5.** --- ## TEE · BPTT hidden delta Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/39/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/39 At an RNN hidden state, a loss gradient can arrive from the current output and from the next time step. Add these two contributions, then multiply by the local activation derivative. The displayed output term p−y assumes softmax with categorical cross-entropy. $$ \delta_t=\left[W_{hy}^T(p_t-y_t)+W_{hh}^T\delta_{t+1}\right]\odot(1-h_t^2) $$ ## How an RNN is trained **Definition.** Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters. ### Explanation and working Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT. ![Unroll forward through time, then accumulate backward contributions for the shared parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_bptt.svg) Diagram: Unroll forward through time, then accumulate backward contributions for the shared parameters.. Labels: One shared weight receives contributions from many times | One shared weight receives contributions from many times | Unrolling draws repeated uses of one cell, not independent copies of its | weights. | Time 1 | Same weights W | Time 2 | Same weights W | Time 3 | Same weights W | Gradients flow backward through the state dependencies | Add shared-weight contributions | Total gradient = contribution 1 + contribution 2 + | contribution 3 | After summing the required contributions, update each shared parameter once. Unroll forward through time, then accumulate backward contributions for the shared parameters. Tap the diagram to enlarge. For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero. $$ \delta_t=\left(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1}\right)\odot(1-h_t^2) $$ $$ \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T},\quad \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^{T},\quad \frac{\partial L}{\partial b_h}=\sum_t\delta_t $$ The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters. ### One BPTT delta at h = 0.5 **Question. In a scalar tanh RNN, h\_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.** ### Given Activated h\_t = 0.5; q\_t = 0.2; δ\_(t+1) = 0.3; recurrent weight u = 0.4. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.4(0.3)=0.12 $$ **Step 2: add the current output contribution.** $$ 0.2+0.12=0.32 $$ **Step 3: calculate the current tanh slope.** $$ 1-(0.5)^2=0.75 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.32)(0.75)=0.24 $$ **Final answer:** δ\_t = 0.24. ### One BPTT delta at h = -0.2 **Question. In a scalar tanh RNN, h\_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.** ### Given Activated h\_t = -0.2; q\_t = 0.1; δ\_(t+1) = 0; recurrent weight u = 0.5. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.5(0)=0 $$ **Step 2: add the current output contribution.** $$ 0.1+0=0.1 $$ **Step 3: calculate the current tanh slope.** $$ 1-(-0.2)^2=0.96 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.1)(0.96)=0.096 $$ **Final answer:** δ\_t = 0.096. **Question. In a scalar tanh RNN, the current hidden state is hₜ = 0.6. The current output path contributes 0.1 to the derivative with respect to hₜ. The next hidden preactivation has delta 0.4 and recurrent weight 0.5. Find the current hidden preactivation delta.** ### Given Current hₜ = 0.6; local output-path derivative = 0.1. Next-step delta = 0.4; recurrent weight = 0.5; current activation = tanh. ### To find Find the current hidden preactivation delta. ### Formula — which one and why **Current and future recurrent paths.** Add the local-output contribution and the weighted next-step delta, then multiply by the current tanh slope. hₜ = current activated state; qₜ = current output-path derivative; u = recurrent weight; δₜ₊₁ = next-step preactivation delta. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Current path plus future path, phir current activation slope. ### Solution — step by step The next-step contribution must travel through the recurrent weight. Multiply that weight by the supplied next delta. $$ \text{future contribution}=0.5(0.4)=0.2 $$ Add the current output-path contribution. These are two routes to the same hidden state, so addition is required. $$ \frac{\partial L}{\partial h_t}=0.1+0.2=0.3 $$ The current activation is tanh. Its derivative uses the current activated value hₜ. $$ 1-h_t^2=1-(0.6)^2=1-0.36=0.64 $$ Multiply the combined gradient by this local derivative. $$ \delta_t=0.3(0.64)=0.192 $$ **Final answer: δₜ = 0.192.** This is the derivative with respect to the current hidden raw score. At the last step, there is no future-state contribution unless the question connects another computation there. ### One BPTT delta at h = 0.5 **Question. In a scalar tanh RNN, h\_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.** ### Given Activated h\_t = 0.5; q\_t = 0.2; δ\_(t+1) = 0.3; recurrent weight u = 0.4. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.4(0.3)=0.12 $$ **Step 2: add the current output contribution.** $$ 0.2+0.12=0.32 $$ **Step 3: calculate the current tanh slope.** $$ 1-(0.5)^2=0.75 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.32)(0.75)=0.24 $$ **Final answer:** δ\_t = 0.24. ### One BPTT delta at h = -0.2 **Question. In a scalar tanh RNN, h\_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.** ### Given Activated h\_t = -0.2; q\_t = 0.1; δ\_(t+1) = 0; recurrent weight u = 0.5. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.5(0)=0 $$ **Step 2: add the current output contribution.** $$ 0.1+0=0.1 $$ **Step 3: calculate the current tanh slope.** $$ 1-(-0.2)^2=0.96 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.1)(0.96)=0.096 $$ **Final answer:** δ\_t = 0.096. --- ## TEE · BPTT recurrent-weight gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/40/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/40 The same recurrent weight is reused at several times. Each use contributes to its gradient, so sum all required time-step contributions before updating it. For vectors, delta is a column and the previous state is transposed into a row: their outer product has the same shape as the weight matrix. $$ g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right) $$ $$ \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^T $$ $$ \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^T $$ ## Vanishing and exploding gradients **Definition.** Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment. ### Explanation and working In BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity. ![Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.](https://deep-learning-study-room.vercel.app/assets/figures/v_vanishing.svg) Diagram: Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.. Labels: Repeated multiplication changes gradient size | Repeated multiplication changes gradient size | These scalar examples illustrate one part of the BPTT mechanism. | Repeated factors below 1 | 0.5 × 0.5 × … × 0.5 | Ten factors: 0.0009765625 | Earlier learning signal is tiny. | Repeated large factors | 2 × 2 × … × 2 | Ten factors: 1,024 | Updates can become unstable. | Gradient clipping limits explosion. It does not rebuild a gradient that has | already vanished. Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates. Tap the diagram to enlarge. **Numerical intuition:** 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs. Useful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory. ### Why the effect grows with sequence length For a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians: $$ \frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(1-h_t^2)W_{hh} $$ $$ \frac{\partial h_T}{\partial h_t}=\prod_{k=T}^{t+1}\frac{\partial h_k}{\partial h_{k-1}} $$ The product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone. ### Gradient norm clipping: a complete numerical Norm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient. $$ g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right) $$ If g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation. **Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.** ### Given Gradient g = [6,8]ᵀ; norm threshold τ = 5. Old weights w = [1,1]ᵀ; η = 0.1. ### To find Find the clipped gradient and updated weights. ### Formula — which one and why **Gradient norm clipping.** If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction. g = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate. $$ \lVert g\rVert_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min\left(1,\frac\tau{\lVert g\rVert_2}\right) $$ $$ w_{\mathrm{new}}=w-\eta g_{\mathrm{clip}} $$ **Yaad rakhne ka cue:** Bada gradient? Sab components ko ek hi factor se chhota karo. ### Solution — step by step **Step 1: calculate the original norm.** $$ \lVert g\rVert_2=\sqrt{6^2+8^2}=\sqrt{36+64}=10 $$ **Step 2: find the scale.** Since 10 > 5, scale the vector by 5/10 = 0.5. $$ g_{\mathrm{clipped}}=0.5\begin{bmatrix}6\\8\end{bmatrix}=\begin{bmatrix}3\\4\end{bmatrix} $$ **Step 3: verify the clipped norm.** $$ \sqrt{3^2+4^2}=\sqrt{9+16}=5 $$ **Step 4: update each parameter with the clipped gradient.** $$ w_{1,\mathrm{new}}=1-0.1(3)=0.7 $$ $$ w_{2,\mathrm{new}}=1-0.1(4)=0.6 $$ **Final answer:** clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture. ### Gradient norm clipping: [3, 4] **Question. Clip gradient g = [3, 4] to maximum L2 norm 10. Use global norm clipping.** ### Given g = [3, 4]; threshold τ = 10. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 3^2+4^2=25 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{25}=5 $$ **Step 3: choose the shared scale.** $$ c=\min(1,10/5)=1 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=3(1)=3 $$ $$ g_2^{\mathrm{clip}}=4(1)=4 $$ **Final answer:** Clipped gradient = [3, 4]. ### Gradient norm clipping: [0, 12] **Question. Clip gradient g = [0, 12] to maximum L2 norm 3. Use global norm clipping.** ### Given g = [0, 12]; threshold τ = 3. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 0^2+12^2=144 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{144}=12 $$ **Step 3: choose the shared scale.** $$ c=\min(1,3/12)=0.25 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=0(0.25)=0 $$ $$ g_2^{\mathrm{clip}}=12(0.25)=3 $$ **Final answer:** Clipped gradient = [0.0, 3.0]. ## Drill 45 BPTT shared weights ## Question **Question.** Linear recurrence hₜ = uxₜ+vhₜ₋₁+b, output h₂, target 1, half squared loss. u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1, x₂ = 2, η = 0.1. Find both states and one update. ## Answer — step by step ### Given Linear recurrence hₜ = uxₜ+vhₜ₋₁+b. u = 0.2; v = 0.5; b = 0; h₀ = 0. x₁ = 1; x₂ = 2; target for h₂ = 1. Half squared loss; learning rate η = 0.1. ### To find Find both states, all three shared-parameter gradients and one simultaneous update. ### Formula — which one and why **Backpropagation through time.** Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once. For h\_t=ux\_t+vh\_(t−1)+b with loss only at the last state: δ\_t = loss derivative at that state. The linear recurrence has activation slope 1. $$ \delta_T=h_T-y $$ $$ \delta_t=v\delta_{t+1} $$ $$ g_u=\sum_t\delta_tx_t $$ $$ g_v=\sum_t\delta_th_{t-1} $$ $$ g_b=\sum_t\delta_t $$ **Yaad rakhne ka cue:** Time ke har use ka gradient jodo; shared weight ko ek baar update karo. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas The same u, v and b appear at both times. u multiplies the input, v multiplies the previous state, and b is a shared bias. The loss is only at the final state h₂. Because the recurrence is linear, its local activation derivative is 1. $$ h_t=ux_t+vh_{t-1}+b,\qquad L=\frac12(h_2-y)^2 $$ $$ \delta_2=h_2-y,\qquad \delta_1=v\delta_2 $$ Each shared parameter receives a contribution from both of its uses. $$ \frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2 $$ $$ \frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1,\qquad \frac{\partial L}{\partial b}=\delta_1+\delta_2 $$ ### Step 1: forward through time 1 $$ h_1=0.2(1)+0.5(0)+0=0.2 $$ ### Step 2: forward through time 2 $$ h_2=0.2(2)+0.5(0.2)+0=0.4+0.1=0.5 $$ $$ L=\frac12(0.5-1)^2=0.125 $$ ### Step 3: carry the gradient backwards $$ \delta_2=0.5-1=-0.5 $$ $$ \delta_1=0.5(-0.5)=-0.25 $$ ### Step 4: add contributions for each shared parameter $$ \frac{\partial L}{\partial u}=(-0.25)(1)+(-0.5)(2)=-0.25-1=-1.25 $$ $$ \frac{\partial L}{\partial v}=(-0.25)(0)+(-0.5)(0.2)=0-0.1=-0.1 $$ $$ \frac{\partial L}{\partial b}=-0.25-0.5=-0.75 $$ ### Step 5: update once after summing $$ u_{\mathrm{new}}=0.2-0.1(-1.25)=0.325 $$ $$ v_{\mathrm{new}}=0.5-0.1(-0.1)=0.51 $$ $$ b_{\mathrm{new}}=0-0.1(-0.75)=0.075 $$ **Final answer:** states 0.2 and 0.5; new parameters u = 0.325, v = 0.51, b = 0.075. --- ## TEE · LSTM gates and candidate Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/41/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/41 An LSTM gate is a multiplier between zero and one. The forget gate chooses the fraction of old cell content retained; the input gate chooses the fraction of candidate content admitted. Each gate is calculated from its own weighted sum and sigmoid. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f) $$ $$ i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ ## LSTM a memory path with three gates **Definition.** Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell. ### Explanation and working Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference. ![Follow the cell state from left to right then identify forget input and output gates](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: Follow the cell state from left to right then identify forget input and output gates. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. Follow the cell state from left to right then identify forget input and output gates Tap the diagram to enlarge. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c) $$ $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\qquad h_t=o_t\odot\tanh(C_t) $$ The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state. ## Reading an LSTM equation one operation at a time ### 1. Two memories on the page do not mean the same value The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other. ![Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_split.svg) Diagram: Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.. Labels: Calculate cell memory before the hidden output | Calculate cell memory before the hidden output | Example: old cell 0.8, forget gate 0.25, input gate 0.5, candidate −0.4. | Retain old content | 0.25 × 0.8 = 0.2 | Admit new content | 0.5 × (−0.4) = −0.2 | New cell | 0.2 + (−0.2) = 0 | Hidden output: 0.7 × tanh(0) = 0 | The output gate is 0.7 here. Apply it to tanh of the NEW cell, after the cell | update. Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output. Tap the diagram to enlarge. ### 2. Gate values are multipliers For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention. An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive. ### 3. Add old and new contributions With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel. $$ c_t=f_tc_{t-1}+i_t\widetilde c_t $$ This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update. ### 4. Calculate the hidden state afterward Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0. $$ h_t=o_t\tanh(c_t) $$ If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c. ### 5. Supplied gates and calculated gates are different numerical types When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition. ## LSTM numerical when gates must first be calculated **Question.** At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state. ### Given x = 1; old hidden state = 0; old cell = 0.2. Forget/input/output preactivations are all 0. Candidate preactivation = 0.5. ### To find Activate all gates and candidate, then find C and h. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Distinguish preactivations from gate outputs The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate. $$ \sigma(a)=\frac1{1+e^{-a}} $$ $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ ![A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_gates_first.svg) Diagram: A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.. Labels: When the question supplies gate raw scores | When the question supplies gate raw scores | A gate raw score is not yet a gate value. | Gate scores | f, i, o scores | = 0 | Apply sigmoid | Each gate | becomes 0.5 | Candidate | score | 0.5 → tanh → | 0.462117 | Then update | Compute cell, | then hidden | output | Sigmoid(0) = 0.5. Do not replace a zero raw score with a closed gate. A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update. Tap the diagram to enlarge. **Step 1: activate the three gates.** $$ f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5 $$ **Step 2: activate the candidate preactivation.** $$ \widetilde C_t=\tanh(0.5)\approx0.462117157 $$ **Step 3: calculate the old and new cell contributions separately.** $$ f_tC_{t-1}=0.5(0.2)=0.1 $$ $$ i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579 $$ **Step 4: calculate the updated cell.** $$ C_t=0.1+0.5\tanh(0.5)\approx0.331058579 $$ **Step 5: calculate the hidden output from the updated cell.** $$ h_t=0.5\tanh(C_t)\approx0.159735818 $$ **Final answer:** fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely. ### LSTM gates from raw score 0 **Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 0. Old cell C\_old = 1; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(0)=\frac1{1+e^{-0}}\approx0.500000000 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(0)\approx0.000000000 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.462117157 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579 $$ **Final answer:** f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579. ### LSTM gates from raw score 1 **Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 1. Old cell C\_old = -0.5; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(1)=\frac1{1+e^{-1}}\approx0.731058579 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(1)\approx0.761594156 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.188942847 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289 $$ **Final answer:** f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289. ## Drill 43 Calculate gates before the cell ## Question **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h. ## Answer — step by step ### Given Raw gate scores: zf = zi = zo = 0. Raw candidate score = 0.5; old cell C = 0.2. ### To find Activate gates/candidate, then calculate the new cell and hidden state. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C) $$ $$ \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C) $$ ### Step 1: turn each zero score into a gate $$ \sigma(0)=\frac1{1+e^0}=\frac12=0.5 $$ Thus f = i = o = 0.5. A zero score is not a zero gate. ### Step 2: calculate candidate content $$ \widetilde C=\tanh(0.5)=0.462117157 $$ ### Step 3: combine retained memory and new content $$ fC_{\mathrm{previous}}=0.5\times0.2=0.1 $$ $$ i\widetilde C=0.5\times0.462117157=0.231058579 $$ $$ C=0.1+0.231058579=0.331058579 $$ ### Step 4: compute the visible hidden state $$ \tanh(C)=\tanh(0.331058579)\approx0.319471636 $$ $$ h=0.5\times0.319471636=0.159735818 $$ **Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818. --- ## TEE · LSTM candidate and output gate Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/42/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/42 The candidate is proposed new cell content, so it may be negative. The output gate is a positive sigmoid multiplier that controls how much of the activated cell is exposed. Candidate content and gate values therefore have different roles. $$ \widetilde C_t=\tanh(W_C[h_{t-1},x_t]+b_C) $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o) $$ ## LSTM numerical when gates must first be calculated **Question.** At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state. ### Given x = 1; old hidden state = 0; old cell = 0.2. Forget/input/output preactivations are all 0. Candidate preactivation = 0.5. ### To find Activate all gates and candidate, then find C and h. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Distinguish preactivations from gate outputs The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate. $$ \sigma(a)=\frac1{1+e^{-a}} $$ $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ ![A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_gates_first.svg) Diagram: A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.. Labels: When the question supplies gate raw scores | When the question supplies gate raw scores | A gate raw score is not yet a gate value. | Gate scores | f, i, o scores | = 0 | Apply sigmoid | Each gate | becomes 0.5 | Candidate | score | 0.5 → tanh → | 0.462117 | Then update | Compute cell, | then hidden | output | Sigmoid(0) = 0.5. Do not replace a zero raw score with a closed gate. A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update. Tap the diagram to enlarge. **Step 1: activate the three gates.** $$ f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5 $$ **Step 2: activate the candidate preactivation.** $$ \widetilde C_t=\tanh(0.5)\approx0.462117157 $$ **Step 3: calculate the old and new cell contributions separately.** $$ f_tC_{t-1}=0.5(0.2)=0.1 $$ $$ i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579 $$ **Step 4: calculate the updated cell.** $$ C_t=0.1+0.5\tanh(0.5)\approx0.331058579 $$ **Step 5: calculate the hidden output from the updated cell.** $$ h_t=0.5\tanh(C_t)\approx0.159735818 $$ **Final answer:** fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely. ### LSTM gates from raw score 0 **Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 0. Old cell C\_old = 1; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(0)=\frac1{1+e^{-0}}\approx0.500000000 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(0)\approx0.000000000 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.462117157 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579 $$ **Final answer:** f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579. ### LSTM gates from raw score 1 **Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 1. Old cell C\_old = -0.5; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(1)=\frac1{1+e^{-1}}\approx0.731058579 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(1)\approx0.761594156 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.188942847 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289 $$ **Final answer:** f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289. ## Drill 43 Calculate gates before the cell ## Question **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h. ## Answer — step by step ### Given Raw gate scores: zf = zi = zo = 0. Raw candidate score = 0.5; old cell C = 0.2. ### To find Activate gates/candidate, then calculate the new cell and hidden state. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C) $$ $$ \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C) $$ ### Step 1: turn each zero score into a gate $$ \sigma(0)=\frac1{1+e^0}=\frac12=0.5 $$ Thus f = i = o = 0.5. A zero score is not a zero gate. ### Step 2: calculate candidate content $$ \widetilde C=\tanh(0.5)=0.462117157 $$ ### Step 3: combine retained memory and new content $$ fC_{\mathrm{previous}}=0.5\times0.2=0.1 $$ $$ i\widetilde C=0.5\times0.462117157=0.231058579 $$ $$ C=0.1+0.231058579=0.331058579 $$ ### Step 4: compute the visible hidden state $$ \tanh(C)=\tanh(0.331058579)\approx0.319471636 $$ $$ h=0.5\times0.319471636=0.159735818 $$ **Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818. ## Drill 50 Three step LSTM retention and replacement ## Question **Question.** C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step. ## Answer — step by step ### Given Initial C₀ = 0; output gate o = 0.7 at all three steps. Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6). ### To find Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas At each time step use the previous **cell state C**, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step. $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ **Step 1: Time 1: admit new content.** $$ f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4 $$ $$ C_1=0+0.4=0.4 $$ $$ h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274 $$ **Step 2: Time 2: retain memory and admit no new content.** $$ f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0 $$ $$ C_2=0.36+0=0.36 $$ $$ h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824 $$ **Step 3: Time 3: retain a small part and add negative content.** $$ f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3 $$ $$ C_3=0.072-0.3=-0.228 $$ $$ h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772 $$ **Final cell states:** 0.4, 0.36, −0.228. **Final hidden states:** approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given. --- ## TEE · LSTM cell and hidden state Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/43/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/43 First calculate the retained old contribution. Separately calculate the admitted new contribution. Add them to obtain the cell state. Only afterward apply tanh and the output gate to obtain the hidden state. $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ h_t=o_t\odot\tanh(C_t) $$ ## Reading an LSTM equation one operation at a time ### 1. Two memories on the page do not mean the same value The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other. ![Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_split.svg) Diagram: Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.. Labels: Calculate cell memory before the hidden output | Calculate cell memory before the hidden output | Example: old cell 0.8, forget gate 0.25, input gate 0.5, candidate −0.4. | Retain old content | 0.25 × 0.8 = 0.2 | Admit new content | 0.5 × (−0.4) = −0.2 | New cell | 0.2 + (−0.2) = 0 | Hidden output: 0.7 × tanh(0) = 0 | The output gate is 0.7 here. Apply it to tanh of the NEW cell, after the cell | update. Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output. Tap the diagram to enlarge. ### 2. Gate values are multipliers For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention. An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive. ### 3. Add old and new contributions With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel. $$ c_t=f_tc_{t-1}+i_t\widetilde c_t $$ This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update. ### 4. Calculate the hidden state afterward Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0. $$ h_t=o_t\tanh(c_t) $$ If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c. ### 5. Supplied gates and calculated gates are different numerical types When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition. ## Drill 42 LSTM with a negative candidate ## Question **Question.** f = 0.8, i = 0.2, o = 0.6, candidate = −0.5, Cprevious = 0.3. Find C, h. ## Answer — step by step ### Given Activated gates: f = 0.8, i = 0.2, o = 0.6. Candidate = −0.5; old cell C = 0.3. ### To find Find the new cell and hidden state. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas An LSTM keeps a cell memory C and a visible hidden state h. The forget gate f controls old memory, the input gate i controls new candidate content, and the output gate o controls the visible state. These gate values are already supplied. $$ C=fC_{\mathrm{previous}}+i\widetilde C,\qquad h=o\tanh(C) $$ The candidate, written C with a tilde, may be negative. A negative candidate subtracts content when admitted. ### Step 1: retained old memory $$ fC_{\mathrm{previous}}=0.8\times0.3=0.24 $$ ### Step 2: admitted new content $$ i\widetilde C=0.2\times(-0.5)=-0.1 $$ ### Step 3: add them to obtain the new cell memory $$ C=0.24+(-0.1)=0.14 $$ ### Step 4: apply tanh to the cell memory $$ \tanh(0.14)=\frac{e^{0.28}-1}{e^{0.28}+1}=0.139092448 $$ ### Step 5: apply the output gate $$ h=0.6\times0.139092448=0.083455469 $$ **Final answer:** cell memory 0.14; hidden state approximately 0.083455469. C and h are different quantities. ## Drill 50 Three step LSTM retention and replacement ## Question **Question.** C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step. ## Answer — step by step ### Given Initial C₀ = 0; output gate o = 0.7 at all three steps. Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6). ### To find Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas At each time step use the previous **cell state C**, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step. $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ **Step 1: Time 1: admit new content.** $$ f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4 $$ $$ C_1=0+0.4=0.4 $$ $$ h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274 $$ **Step 2: Time 2: retain memory and admit no new content.** $$ f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0 $$ $$ C_2=0.36+0=0.36 $$ $$ h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824 $$ **Step 3: Time 3: retain a small part and add negative content.** $$ f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3 $$ $$ C_3=0.072-0.3=-0.228 $$ $$ h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772 $$ **Final cell states:** 0.4, 0.36, −0.228. **Final hidden states:** approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given. ## Drill 43 Calculate gates before the cell ## Question **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h. ## Answer — step by step ### Given Raw gate scores: zf = zi = zo = 0. Raw candidate score = 0.5; old cell C = 0.2. ### To find Activate gates/candidate, then calculate the new cell and hidden state. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C) $$ $$ \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C) $$ ### Step 1: turn each zero score into a gate $$ \sigma(0)=\frac1{1+e^0}=\frac12=0.5 $$ Thus f = i = o = 0.5. A zero score is not a zero gate. ### Step 2: calculate candidate content $$ \widetilde C=\tanh(0.5)=0.462117157 $$ ### Step 3: combine retained memory and new content $$ fC_{\mathrm{previous}}=0.5\times0.2=0.1 $$ $$ i\widetilde C=0.5\times0.462117157=0.231058579 $$ $$ C=0.1+0.231058579=0.331058579 $$ ### Step 4: compute the visible hidden state $$ \tanh(C)=\tanh(0.331058579)\approx0.319471636 $$ $$ h=0.5\times0.319471636=0.159735818 $$ **Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818. --- ## TEE · GRU gates, candidate and update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/44/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/44 A GRU has a reset gate and an update gate. Reset changes how earlier state enters the candidate calculation; update controls the final old/new mixture. The numerical below states its convention explicitly: z is the fraction of the new candidate admitted. $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z) $$ $$ r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ ## GRU two gates and one state **Definition.** A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state. ### Gate equations and the convention used here The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state: $$ r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z) $$ The candidate equation used in these notes applies reset before the recurrent candidate transformation: $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h) $$ These notes define z as the fraction of **new candidate** admitted: $$ h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture. ![First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention. Tap the diagram to enlarge. ### Complete numerical with supplied gate values **Question.** A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ. ### Given Old hidden state = 0.4; r = 0.5; z = 0.25. x = 1; W = 0.2; U = 0.5; b = 0. Here z weights the candidate in the final mixture. ### To find Find candidate and new hidden state. ### Formula — which one and why **GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate. r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state. $$ \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b) $$ $$ h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h $$ **Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control. ### Solution — step by step **Step 1: apply the reset gate to the previous hidden state.** $$ r_th_{t-1}=0.5(0.4)=0.2 $$ The gates are already supplied as outputs, so no further sigmoid calculation is needed. **Step 2: calculate the candidate preactivation.** $$ a_{\mathrm{candidate}}=W_hx_t+U_h(r_th_{t-1})+b_h $$ $$ a_{\mathrm{candidate}}=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh to obtain the candidate.** $$ \widetilde h_t=\tanh(0.3)\approx0.291312612 $$ **Step 4: calculate the two mixture contributions.** $$ (1-z_t)h_{t-1}=(1-0.25)(0.4)=0.3 $$ $$ z_t\widetilde h_t=0.25\tanh(0.3)\approx0.072828153 $$ **Step 5: add the contributions.** $$ h_t=0.3+0.25\tanh(0.3)\approx0.372828153 $$ **Final answer:** candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture. ### If the question uses the other update-gate convention Some texts define z as the fraction of the **old state retained**: $$ h_t=z_th_{t-1}+(1-z_t)\widetilde h_t $$ Under that explicitly different convention, the same numerical value z = 0.25 gives: $$ h_t=0.25(0.4)+0.75\tanh(0.3)\approx0.318484459 $$ This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values. ### Relationship to LSTM A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task. ### Retain-old GRU with gate 0 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = 0.4; z = 0; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: retain the old-state fraction.** $$ 0(0.4)=0 $$ **Step 5: calculate the candidate fraction.** $$ (1-0)(0.291312612)\approx0.291312612 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612 $$ **Final answer:** Candidate ≈ 0.291312612; new state ≈ 0.291312612. ### Retain-old GRU with gate 0.8 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = -0.2; z = 0.8; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(-0.2)=-0.1 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(-0.1)+0=0.15 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.15)\approx0.148885034 $$ **Step 4: retain the old-state fraction.** $$ 0.8(-0.2)=-0.16 $$ **Step 5: calculate the candidate fraction.** $$ (1-0.8)(0.148885034)\approx0.029777007 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993 $$ **Final answer:** Candidate ≈ 0.148885034; new state ≈ -0.130222993. ## Drill 44 GRU reset then mix ## Question **Question.** hprevious = 0.4, r = 0.5, z = 0.25, x = 1, W = 0.2, U = 0.5, b = 0. Use candidate = tanh(Wx+U(r × hprevious)) and h = (1−z) hprevious+z candidate. ## Answer — step by step ### Given Old hidden state = 0.4; reset r = 0.5; update z = 0.25. x = 1; W = 0.2; U = 0.5; b = 0. Use h = (1−z)h\_old + z×candidate. ### To find Find reset contribution, candidate and new hidden state. ### Formula — which one and why **GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate. r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state. $$ \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b) $$ $$ h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h $$ **Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control. ### Solution — step by step ### Meaning and given convention The reset gate r scales the old state while making the candidate. The update gate z controls the amount of candidate used in the final mixture **under this question's convention**. Some books reverse the roles of z and 1 − z, so copy the supplied formula. $$ \widetilde h=\tanh\left(Wx+U(rh_{\mathrm{previous}})+b\right) $$ $$ h=(1-z)h_{\mathrm{previous}}+z\widetilde h $$ ### Step 1: reset the old state for candidate construction $$ rh_{\mathrm{previous}}=0.5\times0.4=0.2 $$ ### Step 2: form the candidate's raw score $$ Wx=0.2\times1=0.2 $$ $$ U(rh_{\mathrm{previous}})=0.5\times0.2=0.1 $$ $$ a=0.2+0.1+0=0.3 $$ ### Step 3: apply tanh $$ \widetilde h=\tanh(0.3)=0.291312612 $$ ### Step 4: mix the original old state with the candidate $$ 1-z=1-0.25=0.75 $$ $$ h=0.75(0.4)+0.25(0.291312612) $$ $$ h=0.3+0.072828153=0.372828153 $$ **Final answer: h ≈ 0.372828153.** The final old-state term uses 0.4, not the reset value 0.2. --- ## TEE · GRU state under the retain-old convention Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/45/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/45 Some GRU questions use the opposite naming convention: z is the fraction of the OLD state retained. The formula below uses that convention. Do not combine its formula with an explanation that says z retains the new candidate. $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h) $$ $$ h_t=z_t\odot h_{t-1}+(1-z_t)\odot\widetilde h_t $$ **Question. Use the retain-old GRU convention. Given hprevious = 0.4, reset gate r = 0.5, update gate z = 0.25, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and the next hidden state.** ### Given h\_old = 0.4; r = 0.5; z = 0.25; x = 1. W = 0.2; U = 0.5; b = 0; z retains the old state. ### To find Find the candidate and new hidden state. ### Formula — which one and why **GRU with a retain-old update gate.** Form the candidate using reset memory; then z retains the old state and 1−z admits the candidate. Follow this question-specific convention. r = reset gate; z = retain-old gate; h\_old = previous state; h̃ = candidate. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Question mein z kisko multiply karta hai wahi meaning; yahan old ko retain. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ r h_{\mathrm{previous}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate's raw input.** Its new-input contribution is 0.2 × 1; its reset-memory contribution is 0.5 × 0.2. $$ a=0.2(1)+0.5(0.2)+0=0.2+0.1=0.3 $$ **Step 3: apply tanh to this raw input.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: calculate how much old state is retained.** In this question, z = 0.25 multiplies the original old state 0.4. $$ z h_{\mathrm{previous}}=0.25(0.4)=0.1 $$ **Step 5: calculate how much candidate is admitted.** The other fraction is 1 − z = 0.75. $$ 1-z=1-0.25=0.75 $$ $$ (1-z)\widetilde h=0.75(0.291312612)\approx0.218484459 $$ **Step 6: add the two final contributions.** $$ h=0.1+0.218484459=0.318484459 $$ **Final answer: h ≈ 0.318484459.** The candidate was 0.291312612; the final hidden state is a mixture of that candidate and the original old state. Under the admit-new convention with the same numeric z, the result would instead be 0.372828153. ### Retain-old GRU with gate 0 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = 0.4; z = 0; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: retain the old-state fraction.** $$ 0(0.4)=0 $$ **Step 5: calculate the candidate fraction.** $$ (1-0)(0.291312612)\approx0.291312612 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612 $$ **Final answer:** Candidate ≈ 0.291312612; new state ≈ 0.291312612. ### Retain-old GRU with gate 0.8 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = -0.2; z = 0.8; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(-0.2)=-0.1 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(-0.1)+0=0.15 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.15)\approx0.148885034 $$ **Step 4: retain the old-state fraction.** $$ 0.8(-0.2)=-0.16 $$ **Step 5: calculate the candidate fraction.** $$ (1-0.8)(0.148885034)\approx0.029777007 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993 $$ **Final answer:** Candidate ≈ 0.148885034; new state ≈ -0.130222993. --- ## TEE · Batch normalization and learned scale/shift Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/46/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/46 Batch normalization first uses batch statistics, then applies a learned scale and shift. Epsilon belongs inside the square root. Inference ordinarily uses stored running statistics. $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ ## Batch normalization inside a residual network **Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training. For one feature with m values in a training batch: $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ The batch mean μ\_B is the centre; batch variance σ²\_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch. **Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.** ### Given x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5. ### To find Batch mean, variance, normalized values and scale-and-shift outputs. ### Formula — which one and why Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale. ### Solution — step by step **Step 1: mean.** $$ \mu_B=\frac{2+4}{2}=3 $$ **Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1. **Step 3: squared deviations and variance.** $$ \sigma_B^2=\frac{(-1)^2+1^2}{2}=\frac{1+1}{2}=1 $$ **Step 4: stable denominator.** $$ d=\sqrt{1+0.00001}\approx1.00000499999 $$ **Step 5: normalize separately.** $$ \hat{x}_1=\frac{-1}{d}\approx-0.99999500004 $$ $$ \hat{x}_2=\frac{1}{d}\approx0.99999500004 $$ **Step 6: scale the first normalized value, then shift it.** $$ y_1=2(-0.99999500004)+0.5\approx-1.49999000008 $$ **Step 7: scale the second normalized value, then shift it.** $$ y_2=2(0.99999500004)+0.5\approx2.49999000008 $$ **Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here. **Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo. --- ## TEE · Global average pooling Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/47/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/47 Global average pooling takes one mean per complete spatial channel. The worked example compares the resulting dense classifier with a classifier after flattening. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ ## Global average pooling: one average for each channel ![Average each channel separately: these are the exact values used in the worked example.](https://deep-learning-study-room.vercel.app/assets/figures/u4_gap.svg) Diagram: Average each channel separately: these are the exact values used in the worked example.. Labels: u4 gap | Global average pooling keeps channels separate | Channel A | 1 | 3 | 5 | 7 | → | 4 | 16 ÷ 4 | Channel B | 2 | 4 | 6 | 8 | → | 5 | 20 ÷ 4 | 2 × 2 × 2 input → two channel means → [4, 5] Average each channel separately: these are the exact values used in the worked example. Tap the diagram to enlarge. **Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier. **Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = $\begin{bmatrix}1 & 3\\5 & 7\end{bmatrix}$ and channel B = $\begin{bmatrix}2 & 4\\6 & 8\end{bmatrix}$ . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.** ### Given H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs. ### To find The GAP result and dense-head parameter count. ### Formula — which one and why Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias. $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Solution — step by step **Step 1: add channel A.** 1 + 3 + 5 + 7 = 16. **Step 2: average channel A.** 16 ÷ 4 = 4. **Step 3: add channel B.** 2 + 4 + 6 + 8 = 20. **Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**. **Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights. **Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero. **Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters. **Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao. --- ## TEE · Top-k error percentage Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/formulas/48/ Reader: https://deep-learning-study-room.vercel.app/#tee/formulas/48 Top-k error counts examples whose correct class is absent from the highest-scoring k predictions. Success counts must not be added twice. $$ E_k=\frac{N-C_k}{N}\times100\% $$ ## Top-1 and top-5 error: what an ImageNet result means **Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models. For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures. **Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.** ### Given Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes. ### To find The percentage of mistakes under each rule. ### Formula — which one and why Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage. $$ E_k=\frac{N-C_k}{N}\times100\% $$ N means total evaluated images; C\_k means images whose correct class appears among the top k predictions. ### Solution — step by step **Step 1: top-1 mistakes.** 100 − 78 = 22. **Step 2: top-1 error percentage.** $$ E_1=\frac{22}{100}\times100\%=22\% $$ **Step 3: top-5 mistakes.** 100 − 92 = 8. **Step 4: top-5 error percentage.** $$ E_5=\frac{8}{100}\times100\%=8\% $$ **Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions. **Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success. --- ## TEE · Coverage and examination structure Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c1-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/1/tee-c1-s0 The TEE route covers all five units of course 702AI0C036: introduction and neural-network foundations, optimization and regularization, CNNs and their architectures, and recurrent models including BPTT, LSTM and GRU. Machine-learning prerequisites are explained within the course. Basic algebra and calculus are assumed, with a concise revision chapter available when required. The supplied course policy specifies a three-hour, 100-mark TEE: Q1 is compulsory, followed by any four of Q2–Q7; each main question carries 20 marks. TEE marks are scaled to 50. Read the instructions printed on the actual examination paper. --- ## TEE · Recommended reading order Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c1-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/1/tee-c1-s1 1. **Neural-network foundations:** learning tasks, data partitions, neuron models, perceptrons, activations, losses and backpropagation. 2. **Optimization:** gradient descent variants, momentum, Nesterov, AdaGrad, RMSProp and Adam, including their states and update conventions. 3. **Regularization and generalization:** overfitting, L1/L2 objectives, dropout, early stopping and remedies selected from training behavior. 4. **CNNs:** convolution and pooling operations, output dimensions, trainable parameter counts and complete network calculations. 5. **CNN architectures:** AlexNet, Inception, ResNet, bottlenecks and transfer learning. 6. **Recurrent models:** state equations, BPTT, vanishing/exploding gradients, LSTM and GRU. 7. **Examination preparation:** use the formula reference, complete solved questions and full TEE paper rehearsal. Use the Mathematics revision chapter for notation, matrix operations, differentiation, probability and logarithms. It is a reference rather than the compulsory starting point. --- ## TEE · Theory answers and numerical solutions Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c1-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/1/tee-c1-s2 For theory, write a precise definition, equations or a labelled architecture, the working mechanism, and the relevant properties or limitations. For comparison questions, compare the same aspects on both sides. A table should support the explanation rather than replace it with disconnected keywords. For numericals, state the given values and conventions, show the governing formula, calculate each intermediate state, and identify the final result. Keep unrounded intermediate values when using the calculator. In backpropagation, calculate all gradients using the same forward-pass parameters before updating them. --- ## TEE · Practice resources Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c1-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/1/tee-c1-s3 The reading flow places worked numericals and relevant examination questions beside their concepts. Solved Questions also provides separate lists of practice numericals, past examination questions and theory answers. Every question has its answer directly below it. The supplied historical papers are filtered against the current syllabus. The TEE mock uses the December 2025 paper as a three-hour rehearsal with full solutions. This is historical practice; it is not a prediction of the forthcoming paper. ### How the supplied papers ask questions The course policy allocates **10 marks to Class Test 2**. The shared confirmed pattern is Q1(a,b), 2 marks each compulsory, followed by any two of Q2–Q4, 3 marks each, in one hour. The faculty message relayed on 12 October identifies **Unit 3 and Unit 4 theory and numericals**, highlighting two Unit 4 numerical questions. It describes alternative theory/numerical combinations without fixing their exact question numbers. Use the MT2 route for that focused preparation; the TEE route covers all five units. TEE has **100 marks in three hours**. Q1 is compulsory, then answer any four of Q2–Q7. Each main question carries 20 marks. The supplied 2023–2026 end papers use four 5-mark parts in Q1 and two 10-mark parts in later main questions. The older 2022/January 2023 papers use five 4-mark parts in Q1. This is an observed historical change, not a prediction of the next paper's subparts. | Asked question family | Examples in the supplied papers | What the answer must contain | | --- | --- | --- | | Neuron / activation calculation | MT1A Q2–Q3 (3 marks each); R26 Q6b (10 marks, mixed calculation and explanation) | Matching products, bias once, activation, requested derivative/probabilities | | Forward pass and backpropagation | MT1A Q4 (3); F25 Q2a, R26 Q2a, R24 Q2a (10 each) | Network data and loss convention, forward values, chain-rule factors, requested gradients and updates | | Perceptron training | F25 Q3a, R26 Q3a, R24 Q4a (10 each) | Threshold and target convention, every sample in order, update only as required, final weights | | Optimizer and update-count questions | F25 Q4a and R26 Q4a (10 each) | Differentiate when needed, initialize state, show each iteration; recompute or use supplied gradients correctly | | Regularization / overfitting | R26 Q1c (5); R24 Q3b and R23 Q7b (10 each) | Definition, mechanism, equations where relevant, training/inference distinction, suitable remedy | | CNN map / shape / parameter calculations | F25 Q6a, R26 Q6a, R24 Q4b (10 each) | Every map entry if requested, each layer shape, filter weights and biases, pooling zero parameters, dense count | | Architectures and comparison | F25 Q2b, F23 Q5a, R23 Q4a (10 each) | Labelled structure, working mechanism, reason for design, advantages and limitations | | Recurrent state / LSTM / GRU | R26 Q5a, F25 Q5a, R24 Q6a (10 each) | State/gate definitions, correct activation, carried state, requested calculations and comparison | These are examples of question types, not an official marking rubric or an assurance that topics repeat. The TEE route retains 108 mapped question parts across the two MT1 sets and six end-semester papers. Two historical parts are restricted to their current-syllabus content. A supplied model answer does not override an explicit loss, dimension or convention in the question. --- ## TEE · All recovered TEE papers: unit, topic and marks map Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-pyq-analysis/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/1/tee-pyq-analysis ### How to use this question map Start with the newest final and re-exam, then use older papers to practise variations. The table links each original lettered part to its complete worked answer. **Repeated questions are history, not a prediction.** A syllabus topic stays required even when no recovered paper asks it. **Marks:** 100 marks are attempted: compulsory Q1 (20), then any four other 20-mark questions. Most recovered papers offer140 marks; NLP December2023 offers120 because the complete original ends at Q6. A mixed question retains one mark allocation; its units are not counted as separate questions. Blank individual allocations remain unknown. ### 2026-02-10 · re-exam · R26 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1 · ML versus DL task choices | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s0/) | | **Q1b [5]** | Unit 2 · Activation functions | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s1/) | | **Q1c [5]** | Unit 3 · L1/L2 and specified-rate dropout | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s2/) | | **Q1d [5]** | Unit 5 · RNN sentiment classification | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s3/) | | **Q2a [10]** | Unit 2 · Two-output sigmoid backpropagation | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s4/) · Loss omitted in original; answer states assumed half summed squared loss. | | **Q2b [10]** | Unit 4 · AlexNet, Inception and architecture recommendation | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s5/) | | **Q3a [10]** | Unit 2 · Bipolar perceptron epoch | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s6/) | | **Q3b [10]** | Unit 4 · Five effects of CNN filter count | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s7/) | | **Q4a [10]** | Unit 3 · Batch/SGD, learning rate and cubic-objective updates | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s8/) | | **Q4b [10]** | Unit 5 · LSTM and GRU; legacy bidirectional-RNN subpart | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s9/) · Current-syllabus extract; original part also contains Bidirectional-RNN advantages (iv) | | **Q5a [10]** | Unit 5 · RNN hidden-state matrix calculation | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s10/) | | **Q5b [10]** | Unit 3 · RMSProp mechanism and SGD comparison | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s11/) | | **Q6a [10]** | Unit 4 · CNN sizes and counts with binary-output ambiguity | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s12/) · Binary head unspecified: one sigmoid or two softmax units give different parameter totals; both are stated. | | **Q6b [10]** | Unit 2 · Sigmoid value/derivative and softmax | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s13/) | | **Q7a [10]** | Unit 2 · Feedforward roles and sigmoid forward pass | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s14/) | | **Q7b [10]** | Unit 4 · Transfer learning and its application | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s15/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1: 1 parts, 5 known marks; Unit 2: 5 parts, 45 known marks; Unit 3: 3 parts, 25 known marks; Unit 5: 3 parts, 25 known marks; Unit 4: 4 parts, 40 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. **Current-syllabus qualification:** Q4b: mixed current and legacy tasks; the retained-only marks cannot be determined from the printed allocation. The original whole-part marks remain visible so the paper is faithfully represented. ### 2025-12-03 · final · F25 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1 · ML versus DL and three task choices | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s0/) | | **Q1b [5]** | Unit 2 · Loss choice and categorical cross-entropy | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s1/) | | **Q1c [5]** | Unit 3 · L1 sparsity, overfitting and dropout | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s2/) | | **Q1d [5]** | Unit 5 · RNN context and variable-length translation | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s3/) | | **Q2a [10]** | Unit 2 · Sigmoid network: forward pass and backpropagation | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s4/) · Original omits inputs and loss. Answer explicitly assumes x=[1,1] and half squared loss; the result is conditional on those assumptions. | | **Q2b [10]** | Unit 4 · ResNet; Inception versus AlexNet | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s5/) | | **Q3a [10]** | Unit 2 · Biological neuron and perceptron epoch | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s6/) | | **Q3b [10]** | Unit 4 · Transfer learning for face attendance | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s7/) | | **Q4a [10]** | Unit 3 · Batch counts, momentum iterations and hyperparameters | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s8/) | | **Q4b [10]** | Unit 5 · Vanishing gradients and LSTM forget contribution | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s9/) | | **Q5a [10]** | Unit 5 · LSTM gates, cell state and hidden state | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s10/) | | **Q5b [10]** | Unit 3 · AdaGrad mechanism and comparison with SGD | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s11/) | | **Q6a [10]** | Unit 4 · CNN sizes and parameters; stride-2 convolution | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s12/) | | **Q6b [10]** | Unit 2 · Activation functions and their roles | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s13/) | | **Q7a [10]** | Unit 2 · Feedforward network roles and ReLU forward pass | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s14/) · Receiving-neuron rows give 0.55; the supplied key uses a transposed convention (0.61). Both interpretations are explained. | | **Q7b [10]** | Unit 4 · CNN feature hierarchy and applications | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s15/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1: 1 parts, 5 known marks; Unit 2: 5 parts, 45 known marks; Unit 3: 3 parts, 25 known marks; Unit 5: 3 parts, 25 known marks; Unit 4: 4 parts, 40 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2025-02-22 · re-exam · R25 140 marks offered; 100 attempted. Combined-year archive filename contains one six-page exam, not two independent papers. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1, 4, 5 · ML versus DL and architecture choice | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s0/) | | **Q1b [5]** | Unit 3 · Training and validation errors | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s1/) | | **Q1c [5]** | Unit 4 · CNN hierarchy and stride-two convolution | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s2/) · Padding is not specified; valid/no-padding is declared. CNN cross-correlation convention is used without kernel reversal. | | **Q1d [5]** | Unit 5 · BPTT and recurrent strengths | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s3/) | | **Q2a [10]** | Unit 2 · Activation, ReLU and numeric outputs | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s4/) | | **Q2b [10]** | Unit 5 · Vanishing/exploding gradients and GRU | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s5/) | | **Q3a [10]** | Unit 2, 3 · 4–3–2 forward pass and dropout | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s6/) · Activation/loss/dropout scaling missing from original; result is conditional on explicit standard classroom assumptions. | | **Q3b [10]** | Unit 4 · Four CNN design explanations | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s7/) | | **Q4a [10]** | Unit 4 · 64×64 CNN shape and parameter table | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s8/) | | **Q4b [10]** | Unit 2 · Bipolar AND perceptron | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s9/) · Initialization, order and threshold convention explicitly supplied as assumptions; multiple equivalent separating weights exist. | | **Q5a [10]** | Unit 4 · ResNet shortcut and Inception1×1 | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s10/) | | **Q5b [10]** | Unit 5 · ReLU RNN state with declared orientation | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s11/) · Square matrices and printed horizontal vectors leave orientation ambiguous; both possible readings are explained, main answer follows row-vector form. | | **Q6a [10]** | Unit 2 · 3–2–1 forward pass and update w7,w8 | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s12/) · Original does not specify loss scaling; half-SSE is declared and alternatives distinguished. | | **Q6b [10]** | Unit 3 · AdaGrad stagnation versus RMSProp | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s13/) | | **Q7a [10]** | Unit 5 · Long-text sentiment architecture | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s14/) | | **Q7b [5]** | Unit 4 · Transfer learning | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s15/) | | **Q7c [5]** | Unit 2, 3 · Learning rate and two losses | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s16/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1, 4, 5: 1 parts, 5 known marks; Unit 3: 2 parts, 15 known marks; Unit 4: 5 parts, 40 known marks; Unit 5: 4 parts, 35 known marks; Unit 2: 3 parts, 30 known marks; Unit 2, 3: 2 parts, 15 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2024-02-13 · re-exam · R24 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1 · Drivers of deep learning adoption | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s0/) | | **Q1b [5]** | Unit 2 · 10-5-5-5-4 dense network parameter count | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s1/) | | **Q1c [5]** | Unit 3 · Overfitting diagnosis and remedies | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s2/) | | **Q1d [5]** | Unit 4 · Max and average pooling | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s3/) | | **Q2a [10]** | Unit 2 · Four-hidden-neuron sigmoid MSE and update | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s4/) | | **Q2b [10]** | Unit 3 · Momentum/Nesterov; high momentum; excluded convex rates | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s5/) · Current-syllabus extract; original part also contains Asymptotic convergence-rate comparison on convex problems (part of i) | | **Q3a [10]** | Unit 2 · Activation comparisons and softmax | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s6/) | | **Q3b [10]** | Unit 3 · Regularization importance and techniques | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s7/) | | **Q4a [10]** | Unit 2 · Perceptron training for two epochs | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s8/) | | **Q4b [10]** | Unit 4 · CNN sizes, weights and biases | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s9/) | | **Q5a [10]** | Unit 4 · Replacing architecture to improve gradient flow | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s10/) | | **Q5b [10]** | Unit 1, 5 · X-ray DL justification; RNN pros and cons | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s11/) | | **Q6a [10]** | Unit 5 · GRU update and reset gates | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s12/) | | **Q6b [10]** | Unit 4 · Pretrained vegetable classifier | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s13/) | | **Q7a [10]** | Unit 5 · LSTM architecture | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s14/) | | **Q7b [10]** | Unit 5 · RNN input/output sequence architectures | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s15/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1: 1 parts, 5 known marks; Unit 2: 4 parts, 35 known marks; Unit 3: 3 parts, 25 known marks; Unit 4: 4 parts, 35 known marks; Unit 1, 5: 1 parts, 10 known marks; Unit 5: 3 parts, 30 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. **Current-syllabus qualification:** Q2b: mixed current and legacy tasks; the retained-only marks cannot be determined from the printed allocation. The original whole-part marks remain visible so the paper is faithfully represented. ### 2023-12-02 · final · F23 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1 · Deep learning features and limitations | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s0/) | | **Q1b [5]** | Unit 2, 3 · ANN hyperparameters and significance | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s1/) | | **Q1c [5]** | Unit 5 · Unrolled RNN and BPTT | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s2/) | | **Q1d [5]** | Unit 4 · Convolution map | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s3/) | | **Q2a [10]** | Unit 2 · Binary-input step-activation network | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s4/) | | **Q2b [10]** | Unit 2, 1 · Activation necessity; ML/DL facial-recognition choice | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s5/) | | **Q3a [10]** | Unit 2 · Sigmoid output-weight backpropagation | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s6/) | | **Q3b [10]** | Unit 3 · Adam advantages versus SGD | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s7/) | | **Q4a [10]** | Unit 4 · AlexNet-style layer dimensions and parameters | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s8/) | | **Q4b [10]** | Unit 5 · LSTM autocomplete architecture and token probabilities | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s9/) | | **Q5a [10]** | Unit 4 · Inception branches and 1x1 convolution | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s10/) | | **Q5b [10]** | Unit 2 · Linear separability; XOR and XNOR construction | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s11/) | | **Q6a [10]** | Unit 5 · LSTM gates; source mixes reset/update terminology | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s12/) · Printed LSTM diagram uses update-gate wording and asks reset equations. Answer separates LSTM and GRU rather than inventing an LSTM reset gate. | | **Q6b [10]** | Unit 4 · ResNet gradient flow and block outputs | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s13/) | | **Q7a [10]** | Unit 5 · Paragraph sentiment model | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s14/) | | **Q7b [10]** | Unit 3 · Bias-variance tradeoff and intervention table | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s15/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1: 1 parts, 5 known marks; Unit 2, 3: 1 parts, 5 known marks; Unit 5: 4 parts, 35 known marks; Unit 4: 4 parts, 35 known marks; Unit 2: 3 parts, 30 known marks; Unit 2, 1: 1 parts, 10 known marks; Unit 3: 2 parts, 20 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2023-01-28 · re-exam · R23 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [4]** | Unit 5 · Music-generation RNN architecture | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s0/) | | **Q1b [4]** | Unit 4 · Stride-2 average pooling and purpose | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s1/) | | **Q1c [4]** | Unit 1 · DL advantages and limitations | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s2/) | | **Q1d [4]** | Unit 3 · Mini-batch size and parallelism | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s3/) | | **Q1e [4]** | Unit 2 · AND perceptron design | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s4/) | | **Q2a [10]** | Unit 2, 3, 4 · CNN parameters, hyperparameters and Adam | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s5/) | | **Q2b [10]** | Unit 2, 3 · ReLU, rating regression and overfitting | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s6/) | | **Q3a [10]** | Unit 5 · RNN language model and per-token probabilities | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s7/) | | **Q3b [10]** | Unit 2 · ReLU/sigmoid forward pass and four output-weight updates | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s8/) | | **Q4a [10]** | Unit 4 · Small-data face-attendance transfer learning | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s9/) | | **Q4b [10]** | Unit 4 · AlexNet parameter count | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s10/) | | **Q5a [10]** | Unit 5 · Vanishing-gradient RNN remedies | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s11/) | | **Q5b [10]** | Unit 2 · Fraud classifier design and parameter count | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s12/) | | **Q6a [10]** | Unit 3 · Three optimization-failure scenarios | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s13/) | | **Q6b [10]** | Unit 4 · Convolution cost and unspecified bottleneck width | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s14/) · The 1x1 reduction width is not specified. Answer gives a symbolic count and an explicitly assumed numerical example. | | **Q7a [10]** | Unit 5 · LSTM sentence memory and gates | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s15/) | | **Q7b [10]** | Unit 3 · Overfitting diagnosis and remedies | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s16/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 5: 4 parts, 34 known marks; Unit 4: 4 parts, 34 known marks; Unit 1: 1 parts, 4 known marks; Unit 3: 3 parts, 24 known marks; Unit 2: 3 parts, 24 known marks; Unit 2, 3, 4: 1 parts, 10 known marks; Unit 2, 3: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2022-11-19 · final · F22 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [4]** | Unit 4 · Stride-2 convolution and detected feature | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s0/) | | **Q1b [4]** | Unit 2 · Signed gradient-descent update | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s1/) | | **Q1c [4]** | Unit 1 · ML and DL comparison | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s2/) | | **Q1d [4]** | Unit 2 · Sigmoid versus tanh | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s3/) | | **Q1e [4]** | Unit 5 · Image-captioning RNN architecture | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s4/) | | **Q2a [10]** | Unit 5 · GRU advantages and subject memory in a sentence | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s5/) | | **Q2b [10]** | Unit 2 · Two-sample sigmoid forward pass and mean-MSE backprop | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s6/) | | **Q3a [10]** | Unit 2, 3 · Bias/variance, four-class activation/loss and diagnosis | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s7/) | | **Q3b [10]** | Unit 4 · LeNet-style parameter count | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s8/) | | **Q4a [10]** | Unit 4 · Inception bottleneck computation count | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s9/) | | **Q4b [10]** | Unit 2, 3 · 6-5-5-4 network; AdaGrad parameters and hyperparameters | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s10/) | | **Q5a [10]** | Unit 5 · RNN model and BPTT updates | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s11/) | | **Q5b [10]** | Unit 3 · Optimization challenges and momentum | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s12/) | | **Q6a [10]** | Unit 4 · CNN vanishing-gradient remedies | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s13/) | | **Q6b [10]** | Unit 2, 3 · Clothing classifier design and six diagnostic tasks | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s14/) | | **Q7a [10]** | Unit 5 · LSTM long-sequence language model | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s15/) | | **Q7b [10]** | Unit 2 · Perceptron non-linear separability and solution | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s16/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 4: 4 parts, 34 known marks; Unit 2: 4 parts, 28 known marks; Unit 1: 1 parts, 4 known marks; Unit 5: 4 parts, 34 known marks; Unit 2, 3: 3 parts, 30 known marks; Unit 3: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### Priority without skipping the syllabus The latest papers show the form and depth of questions, while the official current syllabus sets the learning scope. Authored drills, explanatory numbers and optional extensions are not counted as past questions. Source assumptions and ambiguous data are stated in their answers. --- ## TEE · Before maths: what are we trying to learn? Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s0 ### Machine ko ek number do. Imagine a tiny machine that estimates a bill. You tell it how many notebooks you bought. This information going INTO the machine is called the INPUT. Abhi bas naam samjho: input = andar diya hua number. ![Start at the three notebooks on the left. Their count is the input; the machine turns that quantity into a bill estimate.](https://deep-learning-study-room.vercel.app/assets/figures/notebooks_input.png) ### Visible text and connections in this diagram INPUT MODEL OUTPUT 3 notebooks Multiply by 10 Estimated bill 3 × 10 Total Rs 30 3 notebooks × Rs 10 per notebook = Rs 30 Input = quantity. Weight = price per notebook. Output = estimated bill. 7 8 9 ÷ 4 5 6 × 1 2 3 − 0 . = + Green arrow: three notebooks to calculator Green arrow: calculator to receipt Diagram: Start at the three notebooks on the left. Their count is the input; the machine turns that quantity into a bill estimate. Start at the three notebooks on the left. Their count is the input; the machine turns that quantity into a bill estimate. Tap the diagram to enlarge. **Calculation:** Input = 3 notebooks ### Machine se ek answer niklega. Suppose the machine says the bill is 30 rupees. This answer coming OUT is called the OUTPUT. Input aur output alag jobs hain: 3 was the quantity; 30 is the predicted bill. ![Input and output name two different jobs: quantity enters and the estimated bill leaves.](https://deep-learning-study-room.vercel.app/assets/figures/v_output.svg) Diagram: Input and output name two different jobs: quantity enters and the estimated bill leaves.. Labels: Follow the number through the machine | Follow the number through the machine | The units change: notebook quantity goes in; estimated rupees come out. | INPUT | 3 notebooks | MODEL | Multiply by 10 | OUTPUT | Rs 30 | 3 is the quantity. 30 is the bill prediction. Input and output name two different jobs: quantity enters and the estimated bill leaves. Tap the diagram to enlarge. **Calculation:** Input: 3 notebooks → Output: 30 rupees ### Multiply kyun karte hain? If each notebook costs 10 rupees, three notebooks cost 10 + 10 + 10 = 30. Multiplication is just the shorter calculation: 3 × 10. In this simple model the multiplying number is called a WEIGHT. ![Point to each Rs 10 notebook contribution, then add the three equal amounts.](https://deep-learning-study-room.vercel.app/assets/figures/v_multiply.svg) Diagram: Point to each Rs 10 notebook contribution, then add the three equal amounts.. Labels: Multiplication collects equal groups | Multiplication collects equal groups | Three notebooks. Each contributes the same Rs 10. | Rs 10 | + | Rs 10 | + | Rs 10 | = Rs 30 | 10 + 10 + 10 = 30. The shorter way to write it is 3 × 10 = 30. Point to each Rs 10 notebook contribution, then add the three equal amounts. Tap the diagram to enlarge. **Calculation:** 3 × 10 = 30 ### Weight badla, answer badla. Keep the input 3. Change only the weight from 10 to 8. The output becomes 3 × 8 = 24. A weight controls how strongly an input affects the answer. Neural-network weights can also be negative or fractions. ![Only the multiplier changes. Compare both outputs while keeping the three notebooks fixed.](https://deep-learning-study-room.vercel.app/assets/figures/v_weight.svg) Diagram: Only the multiplier changes. Compare both outputs while keeping the three notebooks fixed.. Labels: Same input. Change only the weight. | Same input. Change only the weight. | The multiplier controls the prediction. | Weight = 10 | 3 notebooks × 10 | Prediction = Rs 30 | Weight = 8 | 3 notebooks × 8 | Prediction = Rs 24 | The quantity stays 3 in both calculations. Only the multiplier changes. Compare both outputs while keeping the three notebooks fixed. Tap the diagram to enlarge. **Calculation:** Old: 3 × 10 = 30. New: 3 × 8 = 24. ### Correct answer ko target kehte hain. The real bill is 30, but the machine predicted 24. The correct answer supplied for a training example is its TARGET. Our prediction is 6 too small because 30 − 24 = 6. ![The prediction is the machine’s result. The target is the real bill used to judge that result.](https://deep-learning-study-room.vercel.app/assets/figures/v_target.svg) Diagram: The prediction is the machine’s result. The target is the real bill used to judge that result.. Labels: Prediction and target have different jobs | Prediction and target have different jobs | The target is the known correct answer for this training example. | Prediction | Machine says Rs 24 | Target | Actual bill is Rs 30 | Compare | 30 − 24 = 6 rupees too low | Prediction may change during learning. The correct target stays 30. The prediction is the machine’s result. The target is the real bill used to judge that result. Tap the diagram to enlarge. **Calculation:** Target = 30. Prediction = 24. Difference = 6. ### Learning ka matlab weight improve karna. For this tiny example, changing the weight from 8 to 10 changes the prediction from 24 to the correct 30. Learning means using examples to adjust weights and reduce prediction error. Later, gradient descent will calculate HOW to adjust them. ![The weight changes from 8 to 10. The same input now produces a prediction closer to the fixed target.](https://deep-learning-study-room.vercel.app/assets/figures/v_learning.svg) Diagram: The weight changes from 8 to 10. The same input now produces a prediction closer to the fixed target.. Labels: Learning changes the adjustable number | Learning changes the adjustable number | Here we show a useful change; later, gradient descent calculates the | correction. | Before | 3 × 8 = 24 | Adjust weight | 8 becomes 10 | After | 3 × 10 = 30 | Input 3 and target 30 stay fixed. The weight is what changes. The weight changes from 8 to 10. The same input now produces a prediction closer to the fixed target. Tap the diagram to enlarge. **Calculation:** 3 × 8 = 24 → adjust weight → 3 × 10 = 30 ### Bias ek extra add hone wala number hai. Now suppose there is a fixed packing fee of 2 rupees. First calculate the notebook cost: 3 × 10 = 30. Then add 2. That extra adjustable number is called a BIAS. In a neural network, it lets the result shift independently of the input. ![The packing fee is added after the notebook cost. It is not multiplied by the notebook quantity.](https://deep-learning-study-room.vercel.app/assets/figures/v_bias.svg) Diagram: The packing fee is added after the notebook cost. It is not multiplied by the notebook quantity.. Labels: A bias adds a separate amount | A bias adds a separate amount | A fixed packing fee is a useful picture of an added offset. | Notebook cost | 3 × 10 = 30 | Add packing | 30 + 2 | New output | Rs 32 | Multiply first. Add the bias once. 3 × 10 + 2 = 32. The packing fee is added after the notebook cost. It is not multiplied by the notebook quantity. Tap the diagram to enlarge. **Calculation:** 3 × 10 + 2 = 30 + 2 = 32 ### Letters numbers ke naam hain. Instead of writing input, weight and bias every time, books use short names. x means input; w means weight; b means bias. So x × w + b says exactly the calculation you already did. A letter is a named box holding a number. ![Read each letter as a named box holding a number, then follow the same arithmetic.](https://deep-learning-study-room.vercel.app/assets/figures/v_symbols.svg) Diagram: Read each letter as a named box holding a number, then follow the same arithmetic.. Labels: Letters are labels for numbers | Letters are labels for numbers | Read the meaning first, then put the number in its place. | x = input | x holds 3 | w = weight | w holds 10 | b = bias | b holds 2 | x × w + b = 3 × 10 + 2 = 32 | A letter does not add a new operation. It names a value. Read each letter as a named box holding a number, then follow the same arithmetic. Tap the diagram to enlarge. **Calculation:** x = 3, w = 10, b = 2 → x × w + b = 32 ### Deep learning mein aage kya hoga? A neural network connects many small calculating units called neurons. They multiply inputs by weights, add a bias and apply an activation rule. Several layers can learn more complex relationships. First we will learn the small calculation; phir layers jodenge. ![Each circle is a small calculation. Outputs from one layer become the next layer’s inputs.](https://deep-learning-study-room.vercel.app/assets/figures/v_layers.svg) Diagram: Each circle is a small calculation. Outputs from one layer become the next layer’s inputs.. Labels: A deep network connects several calculation stages | A deep network connects several calculation stages | Numbers from one layer become inputs to the next layer. | 1 | 2 | 1 | 2 | 3 | 1 | 2 | 1 | Inputs | Hidden layer 1 | Hidden layer 2 | Output | Predictions are calculated forward | Circle numbers identify neurons, not computed values. Each connection has a | weight; each receiving neuron has a bias. Each circle is a small calculation. Outputs from one layer become the next layer’s inputs. Tap the diagram to enlarge. **Calculation:** Input → calculations in layers → prediction **Question.** A toy model has input 4, weight 5 and bias 2. Find its output using input × weight + bias. ### Given A toy model has input 4, weight 5 and bias 2. ### To find Find the preactivation and the requested neuron output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. ### Solution — step by step **Step 1:** multiply input by weight: 4 × 5 = 20. **Step 2:** add the bias: 20 + 2 = 22. Final output = 22. The bias is added once, after multiplication. --- ## TEE · Negative numbers are direction Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s1 ### 1. The minus sign can describe a number Zero is the reference point. Positive numbers such as 1, 2 and 3 lie to its right on a number line. Negative numbers such as −1, −2 and −3 lie to its left. The number −3 is three units below zero; it is not an instruction to subtract until another number is supplied. ![Read left as smaller and right as larger. Follow one signed movement at a time.](https://deep-learning-study-room.vercel.app/assets/figures/v_negative.svg) Diagram: Read left as smaller and right as larger. Follow one signed movement at a time.. Labels: Negative numbers are positions and directions | Negative numbers are positions and directions | Left means smaller. Right means larger. | -3 | -2 | -1 | 0 | 1 | 2 | 3 | 0 + (−2) = −2 | −2 + 2 = 0 | Subtracting a negative reverses its direction: 5 − (−2) = 5 + 2 = 7. Read left as smaller and right as larger. Follow one signed movement at a time. Tap the diagram to enlarge. In 5 − 2, the minus sign is an operation: subtract 2 from 5. In 5 + (−2), the sign inside the brackets belongs to the number negative 2. Both expressions finish at 3. Brackets keep the signed number together. ### 2. Adding a negative moves left Start at 5. Adding positive 2 moves two places right, to 7. Adding negative 2 moves two places left, to 3. $$ \begin{aligned}5+2&=7\\5+(-2)&=3\end{aligned} $$ If both numbers are negative, keep moving left. Start at −3 and add −2: the answer is −5. $$ (-3)+(-2)=-5 $$ ### 3. Subtracting reverses that movement Subtracting positive 2 moves left. Subtracting negative 2 reverses a leftward movement, so it moves right. A money example is removing a debt of 2: your balance increases by 2. $$ 5-(-2)=5+2=7 $$ The shortcut “two negatives make a positive” is not a rule for every calculation. Negative 3 plus negative 2 is still −5. First identify the operation: addition, subtraction, multiplication or division. ### 4. Multiplication has its own sign rule For multiplication and division, equal signs give a positive result and different signs give a negative result. Calculate the ordinary size, then attach the correct sign. | Signs being multiplied | Complete calculation | Result sign | | --- | --- | --- | | Positive × positive | 3 × 2 = 6 | Positive | | Negative × positive | (−3) × 2 = −6 | Negative | | Positive × negative | 3 × (−2) = −6 | Negative | | Negative × negative | (−3) × (−2) = 6 | Positive | Division uses the same sign pattern: (−6) ÷ 2 = −3 and (−6) ÷ (−2) = 3. The divisor must be nonzero. ### 5. Fully worked arithmetic **Question.** Calculate −3 + 2, −3 − 2 and −3 − (−2). **Answer — step by step.** ### Given Expressions: −3+2; −3−2; −3−(−2). ### To find Evaluate the three signed-number expressions. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step **Step 1: evaluate the first expression.** start at −3 and move two places right. The answer is −1. $$ -3+2=-1 $$ **Step 2: evaluate the second expression.** start at −3 and move two places left. The answer is −5. $$ -3-2=-5 $$ **Step 3: evaluate the third expression.** replace subtract-negative-2 with add-positive-2. The answer is −1. $$ \begin{aligned}-3-(-2)&=-3+2\\&=-1\end{aligned} $$ Later, a weight update subtracts a correction that can itself be negative. The third expression explains exactly why that update can increase a weight. --- ## TEE · Fractions decimals and percentages Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s2 ### 1. A fraction tells us to divide The upper number is the **numerator**. The lower number is the **denominator**. In one-quarter, split one whole into four equal parts and take one part. The fraction bar means division. $$ \frac14=1\div4=0.25 $$ ![One shaded part out of four is the same amount as 0.25 or 25 percent.](https://deep-learning-study-room.vercel.app/assets/figures/v_fractions.svg) Diagram: One shaded part out of four is the same amount as 0.25 or 25 percent.. Labels: One whole, split into four equal parts | One whole, split into four equal parts | Shade one of the four parts. | 1 out of 4 | 25% | 1/4 = 1 ÷ 4 = 0.25 = 25%. These are three ways to name the same amount. One shaded part out of four is the same amount as 0.25 or 25 percent. Tap the diagram to enlarge. A decimal and a fraction can name the same amount. One-half is 0.5, one-tenth is 0.1, and one-hundredth is 0.01. The denominator cannot be zero: division by zero is undefined. $$ \begin{aligned}\frac12&=0.5\\\frac1{10}&=0.1\\\frac1{100}&=0.01\end{aligned} $$ ### 2. Why small decimals become smaller when multiplied Multiplying by 0.1 means taking one-tenth. Multiplying by 0.2 means taking two-tenths. For 0.2 × 0.3, express both numbers as fractions. Multiply the numerators and multiply the denominators. $$ \begin{aligned}0.2\times0.3&=\frac2{10}\times\frac3{10}\\&=\frac6{100}\\&=0.06\end{aligned} $$ This also explains a calculation used later: 0.1 × (−0.6) is one-tenth of negative 0.6, which is −0.06. The negative sign is kept because the factors have different signs. ### 3. Dividing by one-half asks how many halves fit Two halves fit inside 1. Four halves fit inside 2. Therefore dividing 2 by one-half gives 4, even though division by a number greater than 1 would make 2 smaller. $$ \begin{aligned}2\div0.5&=2\div\frac12\\&=2\times2\\&=4\end{aligned} $$ The **reciprocal** flips a nonzero fraction: the reciprocal of 1/2 is 2/1. Dividing by a fraction means multiplying by its reciprocal. For decimals, multiplying both the numerator and denominator by the same nonzero number preserves their ratio. $$ \begin{aligned}\frac{0.8}{0.5}&=\frac{0.8\times10}{0.5\times10}\\&=\frac85\\&=1.6\end{aligned} $$ ### 4. Percent means out of one hundred To turn a percentage into a decimal, divide by 100. To turn a decimal into a percentage, multiply by 100. $$ \begin{aligned}20\%&=\frac{20}{100}=0.2\\50\%&=\frac{50}{100}=0.5\end{aligned} $$ If 20% is removed, 80% remains. As decimals: 1 − 0.2 = 0.8. The number 1 represents the whole amount, or 100%. This same arithmetic will be used for dropout later. ### 5. A mean is a fair-share amount For 2, 4 and 9, first add all three numbers: 2 + 4 + 9 = 15. Divide by the number of values, which is 3. The mean is 5. Do not divide by 9 just because it is the largest value. $$ \text{mean}=\frac{\text{sum}}{\text{count}} $$ Here the sum is 15 and the count is 3. Apply those two quantities to the formula. $$ \begin{aligned}\text{mean}&=\frac{2+4+9}{3}\\&=\frac{15}{3}\\&=5\end{aligned} $$ A **weighted mean** gives different shares of importance to the values. With shares 0.9 and 0.1, the first value receives 90% of the importance and the second receives 10%. The shares add to 1. **Question.** Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively. **Answer — step by step.** ### Given Find the weighted mean of 10 and 20 using weights 0.9 and 0.1 respectively. ### To find Find the weighted mean. ### Formula — which one and why **Weighted sum.** Multiply each value by its own coefficient and then add. For a weighted mean, divide by the sum of the weights unless that sum is already 1. xᵢ = value i; aᵢ = its weight. Each input keeps its own matching coefficient. $$ \bar x_w=\frac{\sum_i a_ix_i}{\sum_i a_i} $$ **Yaad rakhne ka cue:** Har number apne weight ke saath multiply; phir sab add. ### Solution — step by step **Step 1:** multiply the first value by its weight: 0.9 × 10 = 9. **Step 2:** multiply the second value by its weight: 0.1 × 20 = 2. **Step 3:** add the contributions: 9 + 2 = 11. The weighted mean is **11**. It is closer to 10 because 10 received most of the importance. $$ \begin{aligned}0.9(10)+0.1(20)&=9+2\\&=11\end{aligned} $$ If importance weights do not sum to 1, divide their weighted sum by the sum of the weights to obtain the weighted mean. A neuron's general weighted sum does not require weights to sum to 1 and is not automatically a mean. --- ## TEE · Brackets before everything else Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s3 ### 1. Brackets tell us what must stay together An expression is a written calculation. Brackets group part of that calculation. In 1 ÷ (1 + 3), the whole quantity 1 + 3 is the divisor, so calculate it before dividing. ![Follow the bracketed calculation from left to right. The bottom line shows why removing brackets changes the answer.](https://deep-learning-study-room.vercel.app/assets/figures/v_brackets.svg) Diagram: Follow the bracketed calculation from left to right. The bottom line shows why removing brackets changes the answer.. Labels: Calculate inside the bracket first | Calculate inside the bracket first | Compare the two different expressions. | With brackets | 1 ÷ (1 + 3) | Finish inside | 1 ÷ 4 | Result | 0.25 | Without brackets: 1 ÷ 1 + 3 = 1 + 3 = 4. The expression has changed. Follow the bracketed calculation from left to right. The bottom line shows why removing brackets changes the answer. Tap the diagram to enlarge. $$ \frac1{1+3}=\frac14=0.25 $$ Removing the brackets changes the calculation. In 1 ÷ 1 + 3, division comes before addition: 1 ÷ 1 = 1, then 1 + 3 = 4. $$ 1\div1+3=1+3=4 $$ ### 2. Use this order for the expressions in this course First calculate inside brackets. Then evaluate powers and roots. Then do multiplication and division, reading left to right. Finally do addition and subtraction, again reading left to right. Multiplication does not have priority over division; they share a level. For 8 ÷ 2 × 3, calculate 8 ÷ 2 = 4 and then 4 × 3 = 12. Do not change it into 8 ÷ (2 × 3). ### 3. A full example with brackets and a power **Question.** Evaluate (2 + 3)² ÷ 5. **Answer — step by step.** ### Given Evaluate (2 + 3)² ÷ 5. ### To find Evaluate each expression and state its value. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step **Step 1:** finish the brackets: 2 + 3 = 5. The expression becomes 5² ÷ 5. **Step 2:** square the bracket result: 5² = 25. The expression becomes 25 ÷ 5. **Step 3:** divide: 25 ÷ 5 = **5**. $$ \begin{aligned}\frac{(2+3)^2}{5}&=\frac{5^2}{5}\\&=\frac{25}{5}\\&=5\end{aligned} $$ ### 4. A full example without grouping the addition **Question.** Evaluate 2 + 3² × 4. **Answer — step by step.** ### Given Evaluate 2 + 3² × 4. ### To find Evaluate each expression and state its value. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step **Step 1:** calculate the power: 3² = 9. Keep the other numbers where they are: 2 + 9 × 4. **Step 2:** multiply: 9 × 4 = 36. The expression becomes 2 + 36. **Step 3:** add: 2 + 36 = **38**. $$ \begin{aligned}2+3^2\times4&=2+9\times4\\&=2+36\\&=38\end{aligned} $$ ### 5. A fraction bar groups the whole denominator In a printed fraction, everything below the bar belongs to the denominator. When entering that fraction in a single calculator line, put brackets around the complete denominator. A printed fraction with 1 above and 1 + 3 below must be entered as 1 ÷ (1 + 3). --- ## TEE · Powers roots and very small numbers Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s4 ### 1. A power counts repeated multiplication In 2³, 2 is the **base** and 3 is the **exponent**. For a positive whole-number exponent, multiply that many copies of the base. It does not mean 2 × 3. $$ 2^3=2\times2\times2=8 $$ The exponent 2 is called a **square**: 3² = 3 × 3 = 9. Three rows of three cells make nine cells. ![Count the three rows of three cells. Squaring builds nine; the square root returns three.](https://deep-learning-study-room.vercel.app/assets/figures/v_powers.svg) Diagram: Count the three rows of three cells. Squaring builds nine; the square root returns three.. Labels: A square and its square root | A square and its square root | Three rows of three cells make nine cells. | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 3² = 3 × 3 = 9 | √9 = 3 | The root returns the side length. | (−2)² = (−2) × (−2) = 4. Keep the minus sign inside the brackets. Count the three rows of three cells. Squaring builds nine; the square root returns three. Tap the diagram to enlarge. ### 2. A negative number must stay inside its brackets Squaring negative 2 means multiplying negative 2 by negative 2. The equal signs of the factors give a positive result. $$ (-2)^2=(-2)(-2)=4 $$ Without brackets, −2² means first calculate 2², then apply the minus sign: −(4) = −4. Enter brackets when the entire negative number is the base. For decimals, use the same rule. Negative 0.3 squared is positive 0.09 because 3/10 multiplied by 3/10 is 9/100. $$ \begin{aligned}(-0.3)^2&=(-0.3)(-0.3)\\&=0.09\end{aligned} $$ ### 3. A square root asks which nonnegative number was squared Because 3 × 3 = 9, the principal square root of 9 is 3. Although (−3)² also equals 9, the symbol √9 by itself means the nonnegative root, 3. $$ \begin{aligned}\sqrt9&=3\\\sqrt{0.09}&=0.3\end{aligned} $$ The **absolute value** of a number is its distance from zero: |−4| = 4 and |4| = 4. Squaring and taking a square root returns this nonnegative size. $$ \sqrt{(-4)^2}=\sqrt{16}=4 $$ ### 4. Zero and negative exponents follow a pattern Start with powers of 10. Each time the exponent decreases by 1, divide the previous value by 10: 1000 → 100 → 10 → 1 → 0.1 → 0.01. This explains both the zero power and the negative powers. | Power | Calculation | Value | | --- | --- | --- | | 10³ | 10 × 10 × 10 | 1000 | | 10² | 1000 ÷ 10 | 100 | | 10¹ | 100 ÷ 10 | 10 | | 10⁰ | 10 ÷ 10 | 1 | | 10⁻¹ | 1 ÷ 10 | 0.1 | | 10⁻² | 0.1 ÷ 10 | 0.01 | | 10⁻³ | 0.01 ÷ 10 | 0.001 | For a nonzero base, power zero is 1. A negative exponent means the reciprocal of the corresponding positive power. It does not mean a negative answer. $$ \begin{aligned}10^{-3}&=\frac1{10^3}\\&=\frac1{1000}\\&=0.001\end{aligned} $$ ### 5. Tiny scientific notation is still an ordinary number The notation 1 × 10⁻⁸ equals 0.00000001. Calculators may display it as 1E−8. In optimizer formulas, ε, pronounced “epsilon”, often names a small positive constant added to a denominator. Read its supplied value before calculating. **Question.** Calculate 1 − 0.9². ### Given Expression: 1−0.9². ### To find Evaluate the power first, then the subtraction. ### Formula — which one and why **A square before subtraction.** The exponent applies to 0.9. Squaring is repeated multiplication; perform it before subtracting the result from 1. q denotes the number being squared. Parentheses would be required to square the whole difference. $$ q^2=q\times q $$ $$ 1-q^2=1-(q\times q) $$ **Yaad rakhne ka cue:** Jis number pe power hai, sirf uska square; subtraction baad mein. ### Solution — step by step **Step 1: square only the 0.9.** $$ 0.9^2=0.9(0.9)=0.81 $$ **Step 2: subtract the result from 1.** $$ 1-0.81=0.19 $$ **Final answer:** 0.19. The square acts on 0.9, not on the entire difference. $$ \begin{aligned}1-0.9^2&=1-0.81\\&=0.19\end{aligned} $$ --- ## TEE · Variables functions and symbols Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s5 These notes assume familiarity with algebra and basic calculus. The revision here covers the notation used in neural-network equations and the conventions that affect numerical answers. | Notation | Technical meaning | | --- | --- | | x, y, ŷ | Input feature vector, target, model prediction | | W, b, θ | Weight matrix, bias vector, complete trainable parameter vector | | z, a | Preactivation (affine input) and activation (layer output) | | L, J | Per-example loss and aggregate training objective | | ∂L/∂w | Partial derivative with respect to one parameter | | ∇θJ | Gradient: vector of partial derivatives with respect to θ | | η, λ, ε | Learning rate, regularization coefficient, numerical stabilizer | | Wᵀ, u⊙v | Transpose and elementwise multiplication | | θₜ, W⁽ˡ⁾ | Update index t and layer index l; neither denotes a power | ### Conventions used in numerical solutions Vectors are columns unless a question specifies otherwise. A weight matrix has one row per receiving neuron and one column per incoming feature. Write matrix dimensions before multiplying. A bias is added once per receiving neuron or output feature map. Keep at least six significant digits during intermediate calculations and round the final result as requested. The loss reduction matters: a sum, mean and half-sum produce different gradient factors. The location of ε also matters: a denominator √s + ε differs from √(s + ε). The sign of a gradient belongs to the derivative, not to the loss value. For example, w = 0.4, η = 0.1 and gradient −2 give the update below. $$ w_{\mathrm{new}}=0.4-0.1(-2)=0.6 $$ In later calculations, state the optimizer convention, initial state and iteration number before substituting values. This prevents mixing momentum velocity with Adam's second-moment accumulator. --- ## TEE · Exponentials and natural logarithms Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s6 **Definitions.** The exponential function exp(x) = eˣ uses Euler's constant e. The natural logarithm ln is its inverse on positive real values. These functions occur in sigmoid, tanh, softmax and cross-entropy. $$ e^{-x}=\frac1{e^x},\qquad\ln(e^x)=x $$ $$ e^{\ln x}=x\ (x>0),\qquad\ln(ab)=\ln a+\ln b\ (a,b>0) $$ For positive constants and valid domains, exponent and log rules simplify expressions before a calculator is needed. For example, e^(−ln 2) = 1/2. Use natural ln for the loss formulas, not the calculator's base-10 log. ### Derivatives used in neural networks $$ \frac{d}{dz}e^{-z}=-e^{-z},\qquad\frac{d}{dp}[-\ln p]=-\frac1p $$ The minus sign in the first expression comes from differentiating the inner function −z. Applying the chain rule correctly here gives the sigmoid derivative derived in the activation section. ### Brief integration and trigonometry reminder Integration reverses differentiation up to an additive constant. The basic relevant identities are: $$ \int x^n\,dx=\frac{x^{n+1}}{n+1}+C\quad(n\ne-1) $$ $$ \int\frac1x\,dx=\ln|x|+C,\qquad\int e^x\,dx=e^x+C $$ For angles in radians, d(sin x)/dx = cos x and d(cos x)/dx = −sin x. The identity sin²x + cos²x = 1 concerns ordinary trigonometric functions. Hyperbolic tanh is a different function; using tan instead of tanh gives an incorrect recurrent-network answer. $$ \tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad\frac{d}{dz}\tanh z=1-\tanh^2z $$ The supplied MT2 calculation families mainly require derivatives, matrix products, logarithms and elementary statistics. Extended integration or trigonometric exercises are not needed for those families. ### Calculator values for activation calculations At z = 2, sigmoid = 1/(1 + e⁻²) ≈ 0.880797. At z = 1, tanh ≈ 0.761594. Keep unrounded values when a later gradient uses these activations. --- ## TEE · Calculator routine Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s7 You reported a Casio ES-series calculator. The exact model string is not confirmed. The functions below are mathematical entries; use the keys labelled on your model. On fx-991 ES/fx-991 ES PLUS models, eˣ is normally the SHIFT function of ln. Do not confuse eˣ with the × 10ˣ scientific-notation key. Use the function labels on your own calculator for these entries. ![Calculate the exponential, build the full denominator, then take its reciprocal.](https://deep-learning-study-room.vercel.app/assets/figures/v_calculator.svg) Diagram: Calculate the exponential, build the full denominator, then take its reciprocal.. Labels: Build the calculator expression in pieces | Build the calculator expression in pieces | Example: sigmoid at a raw score of 2. | Exponential | e⁻² ≈ 0.135335 | Denominator | 1 + 0.135335 | Reciprocal | 1 ÷ 1.135335 ≈ | 0.880797 | Enter 1 ÷ (1 + e^(−2)). The entire denominator needs brackets. Calculate the exponential, build the full denominator, then take its reciprocal. Tap the diagram to enlarge. 1. Use ordinary computation mode. Enter complete brackets around a denominator or negative exponent. 2. For sigmoid at z = 2, enter 1 ÷ (1 + e^(−2)). Result: 0.880797078. Keep the full calculator value; write about six decimals during intermediate work. 3. For tanh at z = 1, use the hyperbolic tanh function if available. A universal alternative is (e² − 1) ÷ (e² + 1), giving 0.761594156. Ordinary tan is a different function; do not use it. 4. For cross-entropy with true probability 0.8, enter −ln(0.8), giving 0.223143551. 5. For softmax [2.3, 4.5, 3.2], subtract 4.5 first, then calculate e⁻²·², 1, e⁻¹·³. Add them, then divide each exponential by the same total. 6. If a result displays as a fraction, use your model’s fraction/decimal conversion key, often S⇔D. Do not reset the calculator unnecessarily. **Five-minute warm-up:** compute sigmoid(0) = 0.5; sigmoid(−2) ≈ 0.119203; tanh(0) = 0; −ln(0.5) ≈ 0.693147; √0.25 = 0.5. A different result signals an entry mistake, not a reason to memorize a wrong value. --- ## TEE · A complete arithmetic calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s8 Evaluate 0.4 − 0.1 × (−2). The multiplication has priority over the subtraction. ![The negative product stays in brackets until the final subtraction is simplified.](https://deep-learning-study-room.vercel.app/assets/figures/v_arithmetic.svg) Diagram: The negative product stays in brackets until the final subtraction is simplified.. Labels: Carry a negative result as one bracketed value | Carry a negative result as one bracketed value | Example: 0.4 − 0.1 × (−2). | Multiply first | 0.1 × (−2) = −0.2 | Replace product | 0.4 − (−0.2) | Subtract negative | 0.4 + 0.2 = 0.6 | The two minus signs have separate jobs before they simplify. The negative product stays in brackets until the final subtraction is simplified. Tap the diagram to enlarge. **Step 1:** multiply 0.1 by −2, giving −0.2. **Step 2:** substitute that entire result, including its sign: 0.4 − (−0.2). **Step 3:** subtracting a negative means adding its positive value: 0.4 + 0.2 = 0.6. The answer is **0.6**. This calculation will later appear inside a weight update; the arithmetic is exactly the same. --- ## TEE · Loss: how one wrong prediction becomes a number Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c2-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/2/tee-c2-s9 **Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative. $$ e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2 $$ For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective. ### Derivatives and their scaling When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_j}=2e_j,\qquad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_j}=e_j,\qquad \frac{\partial\mathrm{MSE}}{\partial\hat y_j}=\frac{2}{n}e_j $$ The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight. **Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.** ### Given Predictions = [0.2,0.7,0.4]. Targets = [0,1,1]; n = 3 scalar predictions. ### To find Find SSE, half SSE and MSE, then all three prediction-gradient lists. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step **Step 1: calculate the three signed prediction errors.** $$ e=[0.2-0,\ 0.7-1,\ 0.4-1]=[0.2,-0.3,-0.6] $$ **Step 2: square every error.** $$ e\odot e=[0.04,0.09,0.36] $$ **Step 3: add the three squared errors.** $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ **Step 4: apply the requested loss divisors.** $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245,\qquad \mathrm{MSE}=\frac{0.49}{3}\approx0.163333333 $$ **Step 5: multiply every error by 2 for the SSE gradient.** $$ \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2] $$ **Step 6: use each error itself for the half-SSE gradient.** $$ \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6] $$ **Step 7: multiply every error by 2/3 for the MSE gradient.** $$ \nabla_{\hat y}\mathrm{MSE}=\frac23[0.2,-0.3,-0.6]=[0.133333333,-0.2,-0.4] $$ The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate. --- ## TEE · A vector is a list and a matrix is a table Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s0 The input [2, 3] is a vector containing two features. A matrix is a rectangle of numbers. “2 by 3” means two rows and three columns. A column vector with two entries has shape 2 by 1. Always write rows before columns. ![Count rows downward and columns across. The shape describes the arrangement of entries.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Count rows downward and columns across. The shape describes the arrangement of entries.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Count rows downward and columns across. The shape describes the arrangement of entries. Tap the diagram to enlarge. In this course, unless a question’s diagram explicitly uses another convention, **each row of a weight matrix belongs to one receiving neuron**. Thus W has shape number of output neurons by number of input features, and the input is a column vector. $$ W=\begin{bmatrix}0.11&0.21\\0.12&0.08\end{bmatrix},\quad x=\begin{bmatrix}2\\3\end{bmatrix},\quad Wx=\begin{bmatrix}0.11(2)+0.21(3)\\0.12(2)+0.08(3)\end{bmatrix}=\begin{bmatrix}0.85\\0.48\end{bmatrix} $$ **Do it slowly:** row 1 says 0.11 × 2 = 0.22 and 0.21 × 3 = 0.63, so its answer is 0.85. Row 2 says 0.12 × 2 = 0.24 and 0.08 × 3 = 0.24, so its answer is 0.48. Matrix multiplication is several weighted sums written together. **Shape check:** (2 by 2) × (2 by 1) gives (2 by 1). The inside numbers must match; the outside numbers are the answer shape. This check catches many transposed-weight errors before arithmetic begins. Transpose swaps rows and columns. Elementwise multiplication does not do row-by-column sums: [2, 3] ⊙ [4, 5] = [8, 15]. Dot product does add: [2, 3] · [4, 5] = 8 + 15 = 23. LSTM gates use elementwise multiplication, not a dot product. **Practice.** Multiply W = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by x = [2, 1] as a column. Answer: first row 1 × 2 + 2 × 1 = 4; second row 3 × 2 + 4 × 1 = 10. With b = [1, −1], Wx + b = [5, 9]. --- ## TEE · Matrix multiplication: every row has a job Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s1 ### Row, column, vector and matrix — read these first A **row** runs horizontally, from left to right. A **column** runs vertically, from top to bottom. A **matrix** is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows. A **vector** is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors. ### Transpose: rows become columns The superscript T means **transpose**, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m. $$ [A^T]_{ij}=A_{ji} $$ The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1. **Yaad rakhne ka cue:** transpose = row ko column bana; number aur sign ko haath mat laga. ### Transpose example 1 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}2&1&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 1 row(s) and 3 column(s), so its size is 1 × 3. **Step 2: move original row 1 into new column 1.** Copy [2, 1, 3] down column 1, in the same order. **Step 3: write the transposed matrix.** $$ A^T=\begin{bmatrix}2\\1\\3\end{bmatrix} $$ **Step 4: state its new size.** The result has 3 row(s) and 1 column(s): 3 × 1. **Final answer:** Aᵀ has size 3 × 1. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 1 rows × 3 columns A: 1 rows × 3 columns | | | | | --- | --- | --- | | 2 | 1 | 3 | Read each colored row left to right. Aᵀ: 3 rows × 1 columns Aᵀ: 3 rows × 1 columns | | | --- | | 2 | | 1 | | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 2 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 3 column(s), so its size is 2 × 3. **Step 2: move original row 1 into new column 1.** Copy [1, 2, 3] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4, 5, 6] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix} $$ **Step 5: state its new size.** The result has 3 row(s) and 2 column(s): 3 × 2. **Final answer:** Aᵀ has size 3 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 3 columns A: 2 rows × 3 columns | | | | | --- | --- | --- | | 1 | 2 | 3 | | 4 | 5 | 6 | Read each colored row left to right. Aᵀ: 3 rows × 2 columns Aᵀ: 3 rows × 2 columns | | | | --- | --- | | 1 | 4 | | 2 | 5 | | 3 | 6 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 3 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&-2\\0&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 2 column(s), so its size is 2 × 2. **Step 2: move original row 1 into new column 1.** Copy [1, -2] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [0, 3] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix} $$ **Step 5: state its new size.** The result has 2 row(s) and 2 column(s): 2 × 2. **Final answer:** Aᵀ has size 2 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 2 columns A: 2 rows × 2 columns | | | | --- | --- | | 1 | −2 | | 0 | 3 | Read each colored row left to right. Aᵀ: 2 rows × 2 columns Aᵀ: 2 rows × 2 columns | | | | --- | --- | | 1 | 0 | | −2 | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 4 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}-1\\4\\2\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 3 row(s) and 1 column(s), so its size is 3 × 1. **Step 2: move original row 1 into new column 1.** Copy [-1] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4] down column 2, in the same order. **Step 4: move original row 3 into new column 3.** Copy [2] down column 3, in the same order. **Step 5: write the transposed matrix.** $$ A^T=\begin{bmatrix}-1&4&2\end{bmatrix} $$ **Step 6: state its new size.** The result has 1 row(s) and 3 column(s): 1 × 3. **Final answer:** Aᵀ has size 1 × 3. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 3 rows × 1 columns A: 3 rows × 1 columns | | | --- | | −1 | | 4 | | 2 | Read each colored row left to right. Aᵀ: 1 rows × 3 columns Aᵀ: 1 rows × 3 columns | | | | | --- | --- | --- | | −1 | 4 | 2 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Dot product: multiply matching entries, then add A **dot product** turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed. $$ w^Tx=w_1x_1+w_2x_2+w_3x_3 $$ This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products. ### Dot-product example 1 **Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.** ### Given Weights w = [2, 3]. Matching inputs x = [4, 1]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (2)(4)=8 $$ **Step 2: multiply matching pair 2.** $$ (3)(1)=3 $$ **Step 3: add the products.** $$ w^Tx=(8)+(3)=11 $$ **Final answer:** wᵀx = 11. ### Dot-product example 2 **Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.** ### Given Weights w = [1, -2]. Matching inputs x = [2, 3]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (1)(2)=2 $$ **Step 2: multiply matching pair 2.** $$ (-2)(3)=-6 $$ **Step 3: add the products.** $$ w^Tx=(2)+(-6)=-4 $$ **Final answer:** wᵀx = -4. ### Dot-product example 3 **Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.** ### Given Weights w = [0.5, 1, -1]. Matching inputs x = [4, 2, 3]. Both lists contain 3 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (0.5)(4)=2 $$ **Step 2: multiply matching pair 2.** $$ (1)(2)=2 $$ **Step 3: multiply matching pair 3.** $$ (-1)(3)=-3 $$ **Step 4: add the products.** $$ w^Tx=(2)+(2)+(-3)=1 $$ **Final answer:** wᵀx = 1. ### Matrix multiplication: each row makes one output To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B. $$ C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj} $$ For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once. **Yaad rakhne ka cue:** multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column. **Definition.** A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation. $$ z=Wx+b $$ For d input features and m receiving neurons, the dimensions are: $$ W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1} $$ The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product. ![Each row of the weight matrix calculates one receiving neuron's preactivation.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge. **Question.** Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b. ### Given Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ. ### To find Find both components of z = Wx+b. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. ### Solution — step by step **Step 1: verify compatible dimensions.** W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column. **Step 2: calculate the first receiving row products.** $$ 1(2)=2 $$ $$ (-2)(1)=-2 $$ **Step 3: sum row 1 and add its bias.** $$ z_1=2+(-2)+0.1=0.1 $$ **Step 4: calculate the second receiving row products.** $$ 0.5(2)=1 $$ $$ 3(1)=3 $$ **Step 5: sum row 2 and add its bias.** $$ z_2=1+3-0.2=3.8 $$ $$ z_1=1(2)+(-2)(1)+0.1=0.1 $$ Row 1 produces output component z1 1. Select weight row 1 1. Select weight row 1 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 2 | −2 | First × first; second × second. Sum the products**0** + Bias for this row**0.1** = z1**0.1** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z_2=0.5(2)+3(1)-0.2=3.8 $$ Row 2 produces output component z2 1. Select weight row 2 1. Select weight row 2 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 1 | 3 | First × first; second × second. Sum the products**4** + Bias for this row**−0.2** = z2**3.8** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z=\begin{bmatrix}0.1\\3.8\end{bmatrix} $$ An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W. ### Two output columns **Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.** ### Given $$ A=\begin{bmatrix}1&2\\3&4\end{bmatrix} $$ $$ B=\begin{bmatrix}2&0\\1&3\end{bmatrix} $$ ### To find The 2 × 2 matrix C = AB. ### Formula — which one and why Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication. Cᵢⱼ is the entry in output row i, column j. $$ C_{ij}=\sum_k A_{ik}B_{kj} $$ **Yaad rakhne ka cue:** Output ki position dekho: wahi row, wahi column. ### Solution — step by step **Step 1: check the dimensions.** (2 × 2)(2 × 2) is valid; output size = 2 × 2. **Step 2: row 1 with column 1.** $$ C_{11}=1(2)+2(1)=2+2=4 $$ **Step 3: row 1 with column 2.** $$ C_{12}=1(0)+2(3)=0+6=6 $$ **Step 4: row 2 with column 1.** $$ C_{21}=3(2)+4(1)=6+4=10 $$ **Step 5: row 2 with column 2.** $$ C_{22}=3(0)+4(3)=0+12=12 $$ **Step 6: assemble the output by position.** $$ C=\begin{bmatrix}4&6\\10&12\end{bmatrix} $$ **Final answer:** C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2. ### Two products that must not be confused **Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.** ### Given a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5. ### To find The 2 × 1 entry-by-entry product and the 2 × 2 outer product. ### Formula — which one and why ⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons. ⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row. $$ (a\odot b)_i=a_i b_i $$ $$ (ab^T)_{ij}=a_i b_j $$ **Yaad rakhne ka cue:** ⊙ mein same jagah; outer product mein har pair. ### Solution — step by step **Step 1: multiply matching entries.** $$ a\odot b=\begin{bmatrix}8\\15\end{bmatrix} $$ **Step 2: transpose b.** $$ b^T=\begin{bmatrix}4&5\end{bmatrix} $$ **Step 3: form row 1 of the outer product.** $$ 2(4)=8,\qquad2(5)=10 $$ **Step 4: form row 2 of the outer product.** $$ 3(4)=12,\qquad3(5)=15 $$ **Step 5: write both rows.** $$ ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix} $$ **Final answer:** Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ . --- ## TEE · A graph and a slope Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s2 A graph shows how one value changes with another. Put the weight w on the horizontal axis and the loss L on the vertical axis. A slope is change in height divided by change in horizontal position. If L rises by 6 when w rises by 2, the average slope is 3. A derivative is the slope for an extremely small change at one location. A positive derivative says increasing the weight initially increases loss. A negative derivative says increasing the weight initially decreases loss. Gradient descent moves against that local slope. **Gradient batata hai loss kis taraf badhega; hum ulte taraf chhota step lete hain.** $$ L(w)=w^2,\qquad \frac{dL}{dw}=2w $$ At w = 3, slope = 6. With learning rate 0.1, the update is 3 − 0.1 × 6 = 2.4. Loss falls from 9 to 5.76. At w = −3, slope = −6, so the update is −3 − 0.1 × (−6) = −2.4. Both moves approach zero. ![A slope tells you the local direction of increasing loss](https://deep-learning-study-room.vercel.app/assets/figures/gradient.svg) Diagram: A slope tells you the local direction of increasing loss. Labels: w | Loss | Negative slope | increase w | Positive slope | decrease w | New weight = old weight − learning rate × gradient A slope tells you the local direction of increasing loss Tap the diagram to enlarge. --- ## TEE · Derivatives without a calculus course Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s3 **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector. $$ \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix} $$ The rules needed for the supplied neural-network numericals are: | Function or operation | Derivative | | --- | --- | | Constant c | 0 | | xⁿ | nxⁿ⁻¹ | | c f(x) | c f′(x) | | f(x) + g(x) | f′(x) + g′(x) | | f(x)g(x) | f′(x)g(x) + f(x)g′(x) | | eˣ | eˣ | | ln x, x > 0 | 1/x | | f(g(x)) | f′(g(x))g′(x) | ### A parameter derivative is not an input derivative For z = wx + b, the variable named in the denominator determines which derivative is required. $$ \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1 $$ At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities. ### Gradient direction For a small parameter displacement Δθ, first-order Taylor expansion gives: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large. ### Differentiate before substituting: 3w^2+5 **Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.** ### Given J(w) = 3w^2+5; w = 2. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(2)=3(2)^2+5=17 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=3(2)w^{2-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(2)=3(2)(2)^{1}=12 $$ **Final answer:** J = 17; derivative = 12. ### Differentiate before substituting: 0.7w^3+0 **Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.** ### Given J(w) = 0.7w^3+0; w = 1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(1)=0.7(1)^3+0=0.7 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=0.7(3)w^{3-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(1)=0.7(3)(1)^{2}=2.1 $$ **Final answer:** J = 0.7; derivative = 2.1. ### Differentiate before substituting: 2w^1+4 **Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.** ### Given J(w) = 2w^1+4; w = -1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(-1)=2(-1)^1+4=2 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=2(1)w^{1-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(-1)=2(1)(-1)^{0}=2 $$ **Final answer:** J = 2; derivative = 2. --- ## TEE · Derivatives: from an ordinary change to a local slope Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s4 ### 1. Function value and change are different numbers Let L(w) = w². At w = 3, loss is 9. If we increase w to 4, loss becomes 16. The loss change is 16 − 9 = 7 and the weight change is 4 − 3 = 1. Their ratio is 7. This is an average slope over a whole interval. ![Start at weight 3 each time. As the increase becomes smaller, the slope estimate approaches 6.](https://deep-learning-study-room.vercel.app/assets/figures/v_slope_limit.svg) Diagram: Start at weight 3 each time. As the increase becomes smaller, the slope estimate approaches 6.. Labels: Shrink the interval to see the local slope | Shrink the interval to see the local slope | Use the same starting point every time: w = 3, L = 9. | Average slope = change in loss ÷ change in weight | Weight increase | New loss | Slope estimate | 1 | 16 | (16 − 9) ÷ 1 = 7 | 0.1 | 9.61 | 0.61 ÷ 0.1 = 6.1 | 0.01 | 9.0601 | 0.0601 ÷ 0.01 = 6.01 | 7 → 6.1 → 6.01 → … → 6. The derivative at w = 3 is 6, not the loss value 9. Start at weight 3 each time. As the increase becomes smaller, the slope estimate approaches 6. Tap the diagram to enlarge. $$ \text{average slope}=\frac{\text{new loss}-\text{old loss}}{\text{new weight}-\text{old weight}} $$ ### 2. Shrink the interval For the same starting weight 3, use an increase of 0.1. The new weight is 3.1 and its square is 9.61. The slope estimate is (9.61 − 9)/0.1 = 6.1. With an increase of 0.01, it is (9.0601 − 9)/0.01 = 6.01. Smaller intervals approach 6. | Increase in w | New w | New loss w² | Change in loss | Change ratio | | --- | --- | --- | --- | --- | | 1 | 4 | 16 | 7 | 7 | | 0.1 | 3.1 | 9.61 | 0.61 | 6.1 | | 0.01 | 3.01 | 9.0601 | 0.0601 | 6.01 | The derivative is the limiting local slope as that interval approaches zero. We do not substitute a literal zero into a division by the interval. ### 3. Why the square has derivative 2w Call a small nonzero change Δw, read “delta w”. Expanding the new square gives: $$ (w+\Delta w)^2=w^2+2w\Delta w+(\Delta w)^2 $$ Subtract the old square w² and divide by the change Δw: $$ \frac{(w+\Delta w)^2-w^2}{\Delta w}=2w+\Delta w $$ As Δw approaches zero, this approaches 2w. That is the origin of the rule d(w²)/dw = 2w. At w = 3 it gives 6, matching the table's trend. ### 4. Reading derivative notation In dL/dw, the top names the quantity whose response we study, and the bottom names what we vary. Read it as “the derivative of L with respect to w”. The notation is not asking you to divide the current loss 9 by the current weight 3; that would give 3 and answer a different question. For a function of several variables, ∂L/∂w is a partial derivative: change w while holding the other independent variables fixed. The gradient groups those partial derivatives for all parameters. ### 5. Why subtract the gradient? If the derivative is positive, increasing w locally increases loss, so we try a small decrease. If it is negative, increasing w locally decreases loss, so we try a small increase. The subtraction in gradient descent automatically handles both signs. $$ w_{\mathrm{new}}=w-\eta g $$ This is local information. A very large step can cross a minimum and make loss worse. The learning rate η keeps control of the step; the derivative does not promise that any step size is safe. --- ## TEE · Partial derivatives hold other things still Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s5 For z = wx + b, the derivative with respect to w is x, because x is held fixed. The derivative with respect to b is 1. The derivative with respect to x is w. This is the entire local maths of a neuron’s weighted sum. ![Choose one changing parameter. Everything else is held fixed while you find its effect.](https://deep-learning-study-room.vercel.app/assets/figures/v_partial.svg) Diagram: Choose one changing parameter. Everything else is held fixed while you find its effect.. Labels: Change one parameter. Hold the others fixed. | Change one parameter. Hold the others fixed. | For z = wx + b, choose which parameter you are changing. | Change w; keep x and b fixed | ∂z/∂w = x | If x = 3, the slope is 3. | Change b; keep x and w fixed | ∂z/∂b = 1 | Adding 1 to b adds 1 to z. | “Partial” identifies one chosen variable while the other variables stay fixed. Choose one changing parameter. Everything else is held fixed while you find its effect. Tap the diagram to enlarge. $$ \frac{\partial z}{\partial w}=x,\qquad \frac{\partial z}{\partial b}=1,\qquad \frac{\partial z}{\partial x}=w $$ **Question:** z = 3w + b, w = 2, b = 1. Find z, ∂z/∂w and ∂z/∂b. ### Given z = 3w+b; w = 2; b = 1. ### To find Find z, ∂z/∂w and ∂z/∂b. ### Formula — which one and why **Partial derivatives.** Differentiate only with respect to the named variable; all other independent inputs and parameters remain fixed. ∂z/∂w reads “rate of change of z when w changes”. For z = wx+b, x is the coefficient of w, while b has coefficient 1. $$ \frac{\partial z}{\partial w}=x $$ $$ \frac{\partial z}{\partial b}=1 $$ **Yaad rakhne ka cue:** Neeche jis letter ka naam hai, sirf usko change samjho. ### Solution — step by step **Step 1: calculate the function value.** $$ z=3(2)+1=6+1=7 $$ **Step 2: differentiate with respect to w while b is fixed.** $$ \frac{\partial z}{\partial w}=3+0=3 $$ **Step 3: differentiate with respect to b while w is fixed.** $$ \frac{\partial z}{\partial b}=0+1=1 $$ **Final answer:** z = 7; weight derivative = 3; bias derivative = 1. A function value and its rate of change are different quantities. --- ## TEE · The chain rule is multiplying linked effects Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s6 **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions. ### First understand why the rates multiply Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency. The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w. **Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add. ### Chain-rule example 1: two linear links **Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.** ### Given w = 1; u = 3w+1; v = 2u. ### To find The two function values and the rate dv/dw. ### Formula — which one and why v depends on w through u. Both links must be included, so multiply the two local derivatives. du/dw is the slope of 3w+1. dv/du is the slope of 2u. $$ \frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw} $$ **Yaad rakhne ka cue:** Andar ka slope × bahar ka slope. ### Solution — step by step **Step 1: calculate the intermediate value.** $$ u=3(1)+1=4 $$ **Step 2: calculate the final value.** $$ v=2(4)=8 $$ **Step 3: differentiate the first link.** $$ \frac{du}{dw}=3 $$ **Step 4: differentiate the second link.** $$ \frac{dv}{du}=2 $$ **Step 5: multiply along the path.** $$ \frac{dv}{dw}=2(3)=6 $$ **Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different. ### 2. One output: calculate both parameter gradients **Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1. ### Given Input x = 2; target y = 4. Old weight w = 1; old bias b = 0. Linear prediction z = wx+b; loss L = ½(z−y)². Learning rate η = 0.1; update both parameters once. ### To find Find the loss gradient for w and for b, then w\_new and b\_new. Finally evaluate the changed prediction and loss. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ![The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.](https://deep-learning-study-room.vercel.app/assets/figures/chain_linear_revision.svg) Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1). The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient. Tap the diagram to enlarge. **Step 1 — evaluate the original model.** Use the given parameters before applying any update. $$ z=wx+b=1(2)+0=2 $$ $$ e=z-y=2-4=-2 $$ $$ L=\frac12e^2=\frac12(-2)^2=2 $$ Forward pass: three distinct quantities Prediction z**2**1 × 2 + 0 → Error e**−2**Prediction 2 − target 4 → Half squared loss**2**½ × (−2)² The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide. **Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it. $$ \frac{\partial L}{\partial e}=\frac12(2e)=e $$ $$ \frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1 $$ $$ \delta_z=\frac{\partial L}{\partial z}=e(1)=-2 $$ The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection. **Step 3 — differentiate the neuron's affine expression.** When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1. $$ \frac{\partial z}{\partial w}=x=2 $$ $$ \frac{\partial z}{\partial b}=1 $$ **Step 4 — multiply along each parameter's path.** $$ g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4 $$ $$ g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2 $$ Backward pass: a separate derivative for each parameter Shared loss derivative δz**−2**Both parameter paths start here. Weight path: multiply by x = 2**−4**∂L/∂w = (−2) × 2 Bias path: multiply by 1**−2**∂L/∂b = (−2) × 1 Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate. **Step 5 — update both parameters simultaneously.** Gradient descent subtracts the learning rate times the corresponding gradient. $$ w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4 $$ $$ b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2 $$ One simultaneous parameter update Weight**1 → 1.4**Subtract 0.1 × (−4) Bias**0 → 0.2**Subtract 0.1 × (−2) Both updates use gradients from the same original forward pass. Re-evaluate the model afterward. Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure. **Step 6 — perform a new forward pass to evaluate the update.** $$ z_{\mathrm{new}}=1.4(2)+0.2=3 $$ $$ L_{\mathrm{new}}=\frac12(3-4)^2=0.5 $$ **Final answer:** weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3. ### 3. Two outputs: why backward contributions are added A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward. **Question.** A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed. ### Given h = wx+b; outputs o₁ = 2h and o₂ = −h. x = 2; w = 0.5; b = 0.1; targets = [1,0]. Loss = half the sum of the two squared output errors. Output multipliers 2 and −1 stay fixed. ### To find Find total loss and its derivatives with respect to h, w and b. ### Formula — which one and why **Chain rule with more than one path.** The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b. oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h} $$ $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h} $$ **Yaad rakhne ka cue:** Har raaste ka effect pehle multiply, junction par sab add. ### Solution — step by step ![The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.](https://deep-learning-study-room.vercel.app/assets/figures/chain_branch_revision.svg) Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5 The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias. Tap the diagram to enlarge. **Step 1 — forward values.** $$ h=0.5(2)+0.1=1.1 $$ $$ o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1 $$ $$ L_1=\frac12(2.2-1)^2=0.72 $$ $$ L_2=\frac12(-1.1-0)^2=0.605 $$ $$ L=L_1+L_2=0.72+0.605=1.325 $$ **Step 2 — output derivatives.** Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output. $$ \frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2 $$ $$ \frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1 $$ **Step 3 — calculate the contribution of each complete path.** The derivative of 2h with respect to h is 2; the derivative of −h is −1. $$ \frac{\partial L_1}{\partial h}=(1.2)(2)=2.4 $$ $$ \frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1 $$ **Step 4 — add the two contributions at the shared hidden value.** $$ \frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h} $$ $$ \frac{\partial L}{\partial h}=2.4+1.1=3.5 $$ Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path. **Step 5 — continue to the hidden weight and bias.** Since h = wx + b, its derivatives with respect to w and b are x and 1. $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7 $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5 $$ **Final answer:** L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation. ### Chain-rule example 4: a square after a linear link **Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.** ### Given w = 1; z = 3w+1; L = z². ### To find z, L and the derivative of L with respect to w. ### Formula — which one and why The square acts on z, not directly on w. Differentiate the square first and include the change of z with w. dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ $$ \frac{d(z^2)}{dz}=2z $$ **Yaad rakhne ka cue:** Square ka 2z; andar ka 3; dono multiply. ### Solution — step by step **Step 1: evaluate the inner expression.** $$ z=3(1)+1=4 $$ **Step 2: evaluate the loss expression.** $$ L=4^2=16 $$ **Step 3: differentiate the square and use z = 4.** $$ \frac{dL}{dz}=2z=2(4)=8 $$ **Step 4: differentiate the inner expression.** $$ \frac{dz}{dw}=3 $$ **Step 5: multiply, because the changes occur in sequence.** $$ \frac{dL}{dw}=8(3)=24 $$ **Final answer:** z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3. --- ## TEE · The derivative of prediction error Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s7 For the half squared loss, use prediction minus target. Let target be y and prediction be ŷ. ![The half and the averaging factor belong to the loss definition, so they also affect its derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: The half and the averaging factor belong to the loss definition, so they also affect its derivative.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. The half and the averaging factor belong to the loss definition, so they also affect its derivative. Tap the diagram to enlarge. $$ L=\frac12(\hat y-y)^2,\qquad \frac{\partial L}{\partial \hat y}=\hat y-y $$ The 2 from differentiating a square cancels the half. Writing the loss as ½(y − ŷ)² gives the same derivative: the derivative of the inside y − ŷ adds a minus sign. This is why “target minus prediction” without the extra minus gives the wrong gradient. If the question uses plain squared error, its derivative is 2(ŷ − y). If it averages errors over n values, also divide by n. Read the stated loss before using a memorized gradient. --- ## TEE · Loss scaling: derive the gradient instead of guessing a factor Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s8 ### 1. One error, three possible objectives Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square: ![Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge. $$ \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1 $$ Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below. | Stated loss over n outputs | Derivative with respect to one prediction aⱼ | | --- | --- | | SSE: sum of all squared errors | 2(aⱼ − yⱼ) | | Half SSE: half of that sum | aⱼ − yⱼ | | MSE: that sum divided by n | 2(aⱼ − yⱼ)/n | | Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n | ### 2. Why only the matching output appears Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor. Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages. ### 3. The activation contributes another factor If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore: $$ \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a) $$ For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question. ### 4. A numerical comparison For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042. The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used. --- ## TEE · Probability mean and variance in plain language Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s9 **Definitions.** A probability assigns a value between 0 and 1 to an event. A discrete probability distribution assigns probabilities to mutually exclusive outcomes whose probabilities sum to 1. Expectation is a probability-weighted average. Variance measures expected squared deviation from the mean. $$ \sum_i p_i=1,\qquad\mathbb E[X]=\sum_i p_i x_i $$ $$ \operatorname{Var}(X)=\mathbb E[(X-\mu)^2]=\mathbb E[X^2]-\mu^2 $$ ### Conditional probability and classification P(A|B) is the probability of A conditional on B, defined when P(B) > 0. Bayes' rule relates it to the reverse conditional probability. $$ P(A\mid B)=\frac{P(A\cap B)}{P(B)}=\frac{P(B\mid A)P(A)}{P(B)} $$ For classification, the network estimates P(class | input). A softmax output is normalized across mutually exclusive classes. Independent sigmoid outputs are used for multiple labels that can be present together. ### Empirical mean, variance and standardization For N observed values, use the stated variance convention. These notes use division by N for the population-style variance in the numerical; an unbiased sample-variance estimate uses N − 1. $$ \mu=\frac1N\sum_i x_i,\qquad\sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2},\qquad x_{\mathrm{std}}=\frac{x-\mu}{\sigma} $$ For values [3, 3, 7, 7], μ = 5, squared deviations are [4, 4, 4, 4], σ² = 16/4 = 4 and σ = 2. Thus a value 7 standardizes to (7 − 5)/2 = 1: one standard deviation above the mean. For preprocessing, fit μ and σ on training data and reuse them for validation/test data; σ must be nonzero. ### Likelihood and cross-entropy If the correct class has predicted probability p, its negative-log loss is −ln p. For p = 0.8, this is approximately 0.223144. For p = 0.1, it is approximately 2.302585. Increasing correct-class probability lowers this loss. Over independent examples, maximizing the product of correct-class probabilities is equivalent to minimizing the sum of negative log probabilities. ### Population mean and spread: [1, 2, 3] **Question. For the population [1, 2, 3], find mean, population variance and standard deviation.** ### Given Values = [1, 2, 3]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 1+2+3=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (1-2)=-1 $$ $$ (-1)^2=1 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (3-2)=1 $$ $$ (1)^2=1 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{1+0+1}{3}\approx0.666666667 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.666666667}\approx0.816496581 $$ **Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581. ### Population mean and spread: [2, 2, 2] **Question. For the population [2, 2, 2], find mean, population variance and standard deviation.** ### Given Values = [2, 2, 2]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 2+2+2=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{0+0+0}{3}\approx0.000000000 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.000000000}\approx0.000000000 $$ **Final answer:** μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation. --- ## TEE · Mean, variance and standard deviation from ordinary arithmetic Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s10 **Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i,\qquad \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2,\qquad \sigma=\sqrt{\sigma^2} $$ These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable. **Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].** ### Given Observations = [3,3,7,7]; N = 4. Use population variance (divide by N). ### To find Find the population mean μ, variance σ² and standard deviation σ. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3. --- ## TEE · Standardization: what the answer 1 actually means Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s11 **Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units. $$ z=\frac{x-\mu_{\mathrm{train}}}{\sigma_{\mathrm{train}}},\qquad \sigma_{\mathrm{train}}>0 $$ Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1]. **Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.** ### Given Training observations = [3,3,7,7]. New observation = 9. Estimate the population standard deviation from the training values. ### To find Find the training mean and standard deviation; standardize all four training values and the new observation. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. **Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units. x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Step 8: standardize each original training observation.** $$ \frac{3-5}{2}=-1 $$ $$ \frac{7-5}{2}=1 $$ The repeated observations give [-1, −1, 1, 1]. **Step 9: subtract the training mean from the new observation.** $$ 9-5=4 $$ **Step 10: divide by the training standard deviation.** $$ z_{\mathrm{new}}=\frac42=2 $$ **Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy. ### Standardization with x = 3 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.** ### Given x = 3; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=3-5=-2 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{-2}{2}=-1 $$ **Step 3: interpret sign and size.** The value is 1 standard-deviation unit(s) below the training mean. **Final answer:** z = -1. Use the same training statistics when transforming a new test input. ### Standardization with x = 5 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.** ### Given x = 5; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=5-5=0 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{0}{2}=0 $$ **Step 3: interpret sign and size.** The value is 0 standard-deviation unit(s) away from the training mean. **Final answer:** z = 0. Use the same training statistics when transforming a new test input. ### Standardization with x = 9 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.** ### Given x = 9; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=9-5=4 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{4}{2}=2 $$ **Step 3: interpret sign and size.** The value is 2 standard-deviation unit(s) above the training mean. **Final answer:** z = 2. Use the same training statistics when transforming a new test input. --- ## TEE · Chain rule: the complete calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c3-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/3/tee-c3-s12 When one value depends on another, a change passes through both calculations. The total derivative multiplies the two local derivatives. It does not add them. Here w is the number being changed, z is an intermediate value, and L is the final value. ![Follow w to z to loss. The local slopes on that dependency path multiply.](https://deep-learning-study-room.vercel.app/assets/figures/v_chain_rule.svg) Diagram: Follow w to z to loss. The local slopes on that dependency path multiply.. Labels: Linked changes multiply along a path | Linked changes multiply along a path | Example: z = 3w + 1 and L = z², evaluated at w = 1. | w = 1 | dz/dw = 3 | z = 4 | dL/dz = 2z = 8 | L = 16 | dL/dw = 8 × 3 = 24 | w affects z; z affects L. Multiply the two local slopes. Follow w to z to loss. The local slopes on that dependency path multiply. Tap the diagram to enlarge. Let z = 3w + 1 and L = z². We will calculate the derivative of L with respect to w at w = 1. **Step 1: find the intermediate value.** Replace w by 1: z = 3 × 1 + 1 = 4. **Step 2: differentiate the last calculation.** For L = z², the power rule gives dL/dz = 2z. At z = 4, this is 2 × 4 = 8. This measures how L responds to z. **Step 3: differentiate the first calculation.** For z = 3w + 1, the derivative of 3w is 3 and the derivative of the constant 1 is 0. Therefore dz/dw = 3. This measures how z responds to w. **Step 4: connect the two effects.** Multiply 8 by 3. The final derivative is 24. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw}=(2z)(3)=8\times3=24 $$ The function value is L = 4² = 16. The derivative is 24. These have different meanings: 16 is the current value of L; 24 is its local rate of change per unit change in w. --- ## TEE · What does learning from data mean Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s0 **Definition.** Machine learning develops models from data so that they can perform a task on new examples. A model is a calculation rule with adjustable numbers. Deep learning uses neural networks with multiple trainable layers; each layer transforms the previous layer's outputs. ### Start with one complete training example A **sample** means one example in the dataset. Suppose one example says: an input number is 2 and the required answer is 4. We write the input as x = 2 and the correct answer, or **target**, as y = 4. These letters are labels for the two numbers. An **input feature** is one recorded property used to make a prediction. One house might have area, number of rooms and age as three features. The three values together describe one sample. An **input vector** is just an ordered list of those feature values. Order matters: the first model weight must always multiply the first feature. The model's calculated answer is the **prediction**, written ŷ, read “y hat”. The hat helps us distinguish the calculated answer ŷ from the required answer y. A prediction is a probability only when the task and output function give it that interpretation. ### The simplest model: multiply, then add In the following model, x is the input, w is a weight and b is a bias. A **weight** multiplies the input. A **bias** is added once after multiplication. Both w and b are adjustable model numbers, called **parameters**. $$ \hat y=wx+b $$ The model first multiplies w and x; then it adds b. For several inputs it uses one matching weight per input, adds their products, and then adds the bias. **Yaad rakhne ka cue:** x diya hua input; y sahi answer; ŷ model ka answer; w aur b training mein badlenge. ![Input numbers are multiplied by weights, their contributions are added with a bias, and an activation gives the neuron output. The next lesson explains each part.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Input numbers are multiplied by weights, their contributions are added with a bias, and an activation gives the neuron output. The next lesson explains each part.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Input numbers are multiplied by weights, their contributions are added with a bias, and an activation gives the neuron output. The next lesson explains each part. Tap the diagram to enlarge. ### Model example 1: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 2, w = 1 and b = 0, calculate its prediction.** ### Given Input x = 2. Weight w = 1; bias b = 0. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(1)(2)=2 $$ **Step 2: add the bias once.** $$ \hat y=2+(0)=2 $$ **Final answer:** ŷ = 2. This is a prediction, not a loss or a learned weight. ### Model example 2: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 2, w = 1.5 and b = 1, calculate its prediction.** ### Given Input x = 2. Weight w = 1.5; bias b = 1. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(1.5)(2)=3 $$ **Step 2: add the bias once.** $$ \hat y=3+(1)=4 $$ **Final answer:** ŷ = 4. This is a prediction, not a loss or a learned weight. ### Model example 3: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 3, w = 1.5 and b = 1, calculate its prediction.** ### Given Input x = 3. Weight w = 1.5; bias b = 1. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(1.5)(3)=4.5 $$ **Step 2: add the bias once.** $$ \hat y=4.5+(1)=5.5 $$ **Final answer:** ŷ = 5.5. This is a prediction, not a loss or a learned weight. ### Model example 4: calculate a prediction **Question. A linear model uses ŷ = wx+b. For x = 2, w = -0.5 and b = 3, calculate its prediction.** ### Given Input x = 2. Weight w = -0.5; bias b = 3. The output is linear: no extra activation is specified. ### To find The predicted value ŷ. ### Formula — which one and why Use the stated model. The multiplication gives the input contribution; the bias shifts the result. x is the input; w is its multiplier; b is the added offset; ŷ is the calculated output. $$ \hat y=wx+b $$ **Yaad rakhne ka cue:** Pehle input × weight, phir bias add. ### Solution — step by step **Step 1: multiply the input by its weight.** $$ wx=(-0.5)(2)=-1 $$ **Step 2: add the bias once.** $$ \hat y=-1+(3)=2 $$ **Final answer:** ŷ = 2. This is a prediction, not a loss or a learned weight. ### How the model learns An initial model can predict 2 when the target is 4. A **loss function** measures the discrepancy. Training calculates how changing each parameter changes the loss; that rate is a **gradient**. An **optimizer** uses the gradient to adjust the parameters. The learning rate controls the size of this adjustment. The loss, derivative and complete update are explained with full examples in the next lessons. The learning rate is a **hyperparameter**: a setting chosen for training. A learned weight is a **parameter**: an adjustable number inside the model. During ordinary **inference**, the trained weights stay fixed and the model calculates an output for a new input. For many training samples, the objective normally averages the individual losses. θ (theta) means the collection of adjustable parameters; N is the number of training examples; Lᵢ is the loss for example i. $$ J(\theta)=\frac1N\sum_{i=1}^{N}L_i(\theta) $$ Σ is an instruction to add the listed terms. For three losses it means L₁ + L₂ + L₃, followed by division by 3. Regularization may add a penalty; the regularization chapter explains that separate term. ### Why several layers help A **layer** contains several neurons that calculate outputs from the same incoming values. In an image model, earlier layers can respond to simple patterns such as edges; later layers combine earlier responses into more complex features. This hierarchy is learned from data. A deep network is not guaranteed to learn useful features merely because it has many layers; training data, architecture and optimization still matter. **Generalization** means performing well on new examples. Training data adjusts weights; validation data helps choose settings and stopping time; held-out test data evaluates the chosen model. Small training loss alone does not establish good generalization. **Yaad rakhne ka cue:** training = answer nikalo → loss nikalo → gradients nikalo → parameters update karo; inference = trained model se answer nikalo. ### Average training loss **Question. Three training examples have losses 2, 0.5 and 0.5. Calculate their mean training loss.** ### Given L₁ = 2; L₂ = 0.5; L₃ = 0.5; N = 3. ### To find The mean loss J. ### Formula — which one and why The question asks for the mean, so add all sample losses and divide by the number of samples. Lᵢ means loss of example i. N counts examples. $$ J=\frac{L_1+L_2+L_3}{N} $$ **Yaad rakhne ka cue:** Mean = total loss ÷ examples. ### Solution — step by step **Step 1: add the losses.** $$ 2+0.5+0.5=3 $$ **Step 2: divide by the number of examples.** $$ J=\frac33=1 $$ **Final answer:** Mean training loss = 1. --- ## TEE · Your first complete training step Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s1 **Question.** A linear model is defined by $\hat y=wx$ . For the training example x = 2, y = 4, use initial weight w₀ = 1, learning rate η = 0.1 and half squared loss. Perform two gradient-descent updates and report the prediction and loss after each update. ### Given Model: ŷ = wx; bias is absent and is not updated. x = 2; y = 4; initial w₀ = 1; η = 0.1. Loss = half squared error; perform two updates. ### To find Find the weight, prediction and loss after each update. ### Formula — which one and why **Chain rule with only a trainable weight.** The weight changes the prediction and the prediction changes the half squared loss. Multiply both rates, then subtract the scaled gradient. The stated model has no bias parameter. ŷ = prediction; y = target; x = input; w = weight; η = learning rate; g = loss derivative with respect to w. $$ \hat y=wx $$ $$ L=\frac12(\hat y-y)^2 $$ $$ g=(\hat y-y)x $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Prediction minus target, phir input se multiply; naya weight = purana − ηg. ### Solution — step by step ### Model, objective and gradient The input and target remain fixed; only the weight is trainable. There is no bias parameter in this question. $$ \hat y=wx,\qquad L(w)=\frac12(wx-y)^2,\qquad w_t=w_{t-1}-\eta g_t $$ Apply the chain rule through the prediction: $$ g_t=\left.\frac{\partial L}{\partial w}\right|_{w_{t-1}}=\frac{\partial L}{\partial\hat y}\frac{\partial\hat y}{\partial w}=(\hat y-y)x $$ The factor 1/2 in the objective cancels the factor 2 from differentiating the square. The remaining factor x comes from differentiating wx with respect to w. ![Forward evaluation, gradient calculation and a parameter update for the same training example.](https://deep-learning-study-room.vercel.app/assets/figures/first_training.png) ### Visible text and connections in this diagram HOW ONE WEIGHT LEARNS 1. PREDICT Input x = 2 Weight w = 1 Prediction = 1 × 2 = 2 Target y = 4 Prediction is 2 too small. 2. MEASURE AND CORRECT Loss = ½ × (2 − 4)² = 2 Gradient = (2 − 4) × 2 = −4 New weight = 1 − 0.1 × (−4) = 1.4 Learning rate = 0.1 3. PREDICT AGAIN Same input x = 2 New weight w = 1.4 New prediction = 1.4 × 2 = 2.8 Target stays 4 Loss falls from 2 to 0.72. Predict row: input to weight to prediction Measure and correct row: loss to gradient to new weight Predict again row: same input to new weight to new prediction Diagram: Forward evaluation, gradient calculation and a parameter update for the same training example. Forward evaluation, gradient calculation and a parameter update for the same training example. Tap the diagram to enlarge. ### Iteration 1 **Step 1: forward evaluation at the current parameter.** $$ \hat y_0=w_0x=1(2)=2,\qquad e_0=\hat y_0-y=2-4=-2 $$ $$ L_0=\frac12(-2)^2=2 $$ **Step 2: gradient at the current parameter.** $$ g_1=(\hat y_0-y)x=(-2)(2)=-4 $$ **Step 3: parameter update.** $$ w_1=w_0-\eta g_1=1-0.1(-4)=1.4 $$ **Step 4: evaluate the updated model.** $$ \hat y_1=1.4(2)=2.8,\qquad L_1=\frac12(2.8-4)^2=0.72 $$ ### Iteration 2 Use w₁ = 1.4 and recompute the gradient; reusing the previous gradient would not be gradient descent on this objective. $$ g_2=(\hat y_1-y)x=(2.8-4)(2)=-2.4 $$ $$ w_2=w_1-\eta g_2=1.4-0.1(-2.4)=1.64 $$ $$ \hat y_2=1.64(2)=3.28,\qquad L_2=\frac12(3.28-4)^2=0.2592 $$ | Model state | Weight | Prediction | Half squared loss | | --- | --- | --- | --- | | Initial | 1 | 2 | 2 | | After update 1 | 1.4 | 2.8 | 0.72 | | After update 2 | 1.64 | 3.28 | 0.2592 | **Interpretation.** Both gradients are negative, so the updates increase the weight toward the optimum w = y/x = 2. The loss decreases in these iterations. Gradient descent does not guarantee a decrease for every learning rate; step size matters. --- ## TEE · Classification regression and learning types Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s2 **Definition.** A learning problem is specified by its data, desired output and objective. Classification estimates a discrete label or class distribution; regression estimates numerical responses. The supervision type describes the feedback available during training. ### Explanation and working **Regression** predicts a numerical quantity, such as temperature, price or a continuous rating. **Classification** predicts a category, such as defective/non-defective or cat/dog/bird. Binary classification has two classes. Multiclass classification chooses one of several classes. Multilabel classification can choose several labels together, such as “beach” and “sunset” for the same image. ![Look at the required answer: a numerical quantity is regression; a category is classification.](https://deep-learning-study-room.vercel.app/assets/figures/v_tasks.svg) Diagram: Look at the required answer: a numerical quantity is regression; a category is classification.. Labels: What kind of answer do you need? | What kind of answer do you need? | The output decides whether this is regression or classification. | REGRESSION: a number | Input: house features | Output: predicted price | Example: Rs 50 lakh | CLASSIFICATION: a category | Input: an image | Output: cat / dog / bird | Example: dog | Supervised training supplies correct input–target pairs for these tasks. Look at the required answer: a numerical quantity is regression; a category is classification. Tap the diagram to enlarge. **Supervised learning** uses known input–target pairs. Most examination numericals in this course use it. **Unsupervised learning** searches for patterns without supplied target labels, for example grouping similar samples. **Reinforcement learning** learns actions using rewards from an environment. You only need these distinctions as vocabulary here; detailed algorithms from last semester are not prerequisites for this course. --- ## TEE · Training validation and test data Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s3 **Definition.** Dataset partitioning separates parameter estimation, model selection and final performance assessment. This separation is essential to evaluate generalization without contaminating the evaluation with selection decisions. ### Explanation and working Training data changes the weights. Validation data helps choose settings such as layer size, learning rate and the stopping epoch. Test data is held back for the final estimate of performance on unseen samples. A possible split is 70% training, 15% validation, 15% testing, but the best split depends on the task and dataset size. ![The three splits have different jobs. Only the training part is used to fit preprocessing and model parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_splits.svg) Diagram: The three splits have different jobs. Only the training part is used to fit preprocessing and model parameters.. Labels: Give each data split one job | Give each data split one job | Example: 1,000 independent samples split 70% / 15% / 15%. | 700 train | 150 validate | 150 test | Training | Adjust weights. | Validation | Choose settings. | Test | Final evaluation. | Fit preprocessing on training data. Reuse the same rule for validation and test. The three splits have different jobs. Only the training part is used to fit preprocessing and model parameters. Tap the diagram to enlarge. For 1,000 independent samples, this split gives 700 training, 150 validation and 150 testing. Avoid putting duplicates or near-identical views of the same subject across splits. For time series, later observations normally belong after earlier observations rather than being mixed randomly without considering time. **Data leakage** occurs when information that would not be available at prediction time enters training or model selection. Fitting a normalization rule to the whole dataset before splitting can leak information. Estimate preprocessing quantities from training data and apply the same rule to validation and test data. **Normalization example.** To map a training range 0 to 100 into 0 to 1, divide by 100. A value 40 becomes 0.4. Standardization instead subtracts the training mean and divides by the training standard deviation. Scaling can make gradients easier to optimize; it does not create labels or guarantee accuracy. --- ## TEE · Parameters and hyperparameters Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s4 **Definition.** Model parameters are quantities estimated during training, whereas hyperparameters specify the architecture or training procedure and are selected separately. Optimizer states are algorithmic accumulators associated with the training procedure. ### Explanation and working Weights and biases are parameters learned during training. Learning rate, batch size, number of epochs, network architecture, dropout rate and regularization strength are hyperparameters chosen or tuned using validation performance. Adam’s moving averages are optimizer state: they are updated by the algorithm but are not extra network connection weights. ![Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters.svg) Diagram: Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately.. Labels: What is learned, and what is chosen? | What is learned, and what is chosen? | Two different kinds of quantities appear in training. | PARAMETERS | Weights and biases | Changed by training | Example: w = 0.5 → 0.6 | HYPERPARAMETERS | Learning rate, batch size | Chosen or tuned separately | Example: learning rate 0.1 | Optimizer memories are algorithm state; they are not extra network connection | weights. Weights change during training. Settings such as batch size and learning rate are chosen or tuned separately. Tap the diagram to enlarge. An **epoch** is one complete pass through the training dataset. A **batch** is the group used to calculate one update. An **iteration** here means one update. If there are 100 samples and batch size 10, one epoch has 10 updates. One epoch does not generally mean one update. --- ## TEE · Artificial intelligence machine learning and deep learning Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s5 **Definition.** Deep learning is a subfield of machine learning that learns representations through multiple trainable processing layers. Each layer transforms the preceding representation, allowing the complete model to learn a composition of functions from data. ### Position within artificial intelligence Artificial intelligence studies systems that perform tasks such as perception, reasoning and decision-making. Machine learning develops models whose parameters or behavior are estimated from data. Deep learning uses layered neural models within machine learning. $$ \text{Deep learning}\subset\text{Machine learning}\subset\text{Artificial intelligence} $$ ![AI is the broad field. Machine learning learns from data; deep learning learns through multiple representation layers.](https://deep-learning-study-room.vercel.app/assets/figures/v_ai_ml_dl.svg) Diagram: AI is the broad field. Machine learning learns from data; deep learning learns through multiple representation layers.. Labels: Deep learning belongs inside machine learning | Deep learning belongs inside machine learning | These are nested areas, not three unrelated technologies. | ARTIFICIAL INTELLIGENCE | MACHINE LEARNING | DEEP LEARNING | Neural networks with many layers | AI is the broad field. ML learns from data. DL uses deep neural networks. AI is the broad field. Machine learning learns from data; deep learning learns through multiple representation layers. Tap the diagram to enlarge. ### Motivation: why use multiple learned layers? **Representation learning.** Hand-designing every useful image or speech feature is difficult. A deep model can learn intermediate representations together with the final prediction objective. In a CNN, early layers may respond to local edges, while later layers combine information over larger regions. These patterns depend on training and are not guaranteed to have a particular human meaning. **Nonlinear relationships.** Real tasks often depend on combinations of input variables. Composing affine transformations with nonlinear activations can represent interactions that a single linear decision boundary cannot. Depth supplies repeated transformations; increasing only the number of neurons in one layer changes width instead. **End-to-end optimization.** The final task loss supplies gradients to the preceding trainable layers. The feature extractor and prediction head can therefore be adjusted for the same task. Preprocessing and human decisions about architecture, data and evaluation still remain necessary. ### Factors that made deep learning practical | Factor | How it contributes | | --- | --- | | Larger datasets | Provide more examples of the variations a complex model must learn | | GPUs and parallel computation | Accelerate the large matrix and convolution operations used in training | | Improved training methods | Backpropagation, suitable activations, initialization and optimizers make fitting deep models more practical | | Pretrained models | Reuse learned features so a new task may require less labeled data and training effort | ### Applications in different domains An application answer should identify the input, the prediction and the role of the learned representation. | Domain | Example input | Required output | Role of deep learning | | --- | --- | --- | --- | | Computer vision | Photograph or video frame | Image class, object locations or pixel labels | Learn visual features with a CNN and an appropriate task head | | Language | Review, sentence or conversation | Sentiment label, translated words or a response | Learn representations of text and its context | | Speech | Audio sequence | Words or a speaker label | Model patterns in acoustic features over time | | Healthcare | Medical image or patient measurements | A task-specific class or numerical estimate | Learn associations from labeled examples; assess results on suitable held-out data | | Autonomous systems and robotics | Camera and other sensor measurements | Perception estimates used by a larger control system | Recognize objects and relevant environmental features | | Financial services | Transaction features or a time-ordered record | Fraud score, risk estimate or forecast | Model patterns and interactions among observations | ### Advantages, limitations and comparison with traditional ML | Aspect | Traditional machine learning often uses | Deep learning often uses | | --- | --- | --- | | Representation | Selected or engineered input features | Features learned jointly across layers | | Useful architectures | Linear models, decision trees, SVMs | MLPs, CNNs, RNNs and gated recurrent models | | Training from scratch | Can be effective with modest structured datasets | Large models often need substantial data and computation | | Interpretation | Some models have directly inspectable rules or coefficients | Learned internal representations can be harder to interpret | Feature learning, flexible nonlinear modeling and reuse of pretrained representations are major advantages. Limitations include computation and memory costs, overfitting with insufficient representative data, difficulty interpreting predictions, and reduced reliability when deployment data differs from training data. High training accuracy does not establish generalization. These are tendencies, not absolute divisions. Traditional ML can also learn representations; deep models can process tabular data. A pretrained deep model may be useful even with a small target dataset. Select the model according to the task, available data, resources and validation results. **Question. Explain the motivation for deep learning and describe applications in at least three domains.** **Answer.** Deep learning is motivated by the need to learn useful hierarchical representations and nonlinear relationships directly from complex data. Multiple trainable layers compose feature transformations, and the task loss can train those transformations jointly with the predictor. Larger datasets, GPU computation and improved optimization methods have made this practical. In computer vision, CNNs learn visual features for image classification and object localization. In language processing, neural sequence models use word representations and context for sentiment analysis or translation. In speech processing, neural models map acoustic patterns to words. The benefits must be balanced against data requirements, computation, overfitting and interpretability limitations. --- ## TEE · What one artificial neuron does Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s6 **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output. ### What goes into a neuron? An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive. The **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**. **Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output. ### Mathematical model and notation $$ z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b $$ $$ a=f(z) $$ | Symbol | Meaning and role | | --- | --- | | xᵢ | Input component i; the neuron receives d such components | | wᵢ | Trainable weight multiplying that input component | | b | Trainable bias added once to this neuron's total | | z | Preactivation: the result before applying the activation function | | f | Chosen activation function, such as sigmoid or ReLU | | a | Activation: the neuron's output after applying f | The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs. ### Why both a bias and an activation are needed The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin. The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values. ![Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron. Tap the diagram to enlarge. ### Worked numerical **Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output. ### Given Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5]. Bias = 0.1; activation = sigmoid. ### To find Find the preactivation z, then the sigmoid output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. ### From one neuron to a layer With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron: $$ z=Wx+b,\qquad a=f(z) $$ Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together. ### Neuron example 2: ReLU output **Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2]. Weights = [3]; bias = 1. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(3)=6 $$ **Step 2: add the input contributions.** $$ s=(6)=6 $$ **Step 3: add the bias once.** $$ z=6+(1)=7 $$ **Step 4: apply the stated activation.** $$ a=\max(0,7)=7 $$ **Final answer:** z = 7; a = 7. ### Neuron example 3: ReLU output **Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.** ### Given Inputs = [2, 1]. Weights = [-1, 0.5]; bias = 0.2. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(-1)=-2 $$ **Step 2: multiply input 2 by its own weight.** $$ (1)(0.5)=0.5 $$ **Step 3: add the input contributions.** $$ s=(-2)+(0.5)=-1.5 $$ **Step 4: add the bias once.** $$ z=-1.5+(0.2)=-1.3 $$ **Step 5: apply the stated activation.** $$ a=\max(0,-1.3)=0 $$ **Final answer:** z = -1.3; a = 0. ### Neuron example 4: sigmoid output **Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.** ### Given Inputs = [1, 2]. Weights = [0.2, -0.1]; bias = 0. Activation = sigmoid. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (1)(0.2)=0.2 $$ **Step 2: multiply input 2 by its own weight.** $$ (2)(-0.1)=-0.2 $$ **Step 3: add the input contributions.** $$ s=(0.2)+(-0.2)=0 $$ **Step 4: add the bias once.** $$ z=0+(0)=0 $$ **Step 5: apply the stated activation.** $$ a=\frac1{1+e^{0}}=\frac1{1+1}=0.5 $$ **Final answer:** z = 0; a = 0.5. --- ## TEE · Model answer: explain an artificial neuron Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s7 ## Question **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example. ## Answer — step by step An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers. ![Follow each input through its weight, add one bias at the receiving neuron, then apply activation.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Follow each input through its weight, add one bias at the receiving neuron, then apply activation. Tap the diagram to enlarge. ### Formula and symbols $$ z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z) $$ | Part | Meaning | What it does | | --- | --- | --- | | x₁, x₂, … | Input features | Supply the information for this calculation | | w₁, w₂, … | Weights | Control the contribution of each matching input | | b | Bias | Shifts the weighted sum independently of the inputs | | z | Net input | The number before activation | | f | Activation rule | Transforms z, often nonlinearly | | a | Activated output | The number sent onward | ### Worked ReLU neuron example **Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU. ### To find Preactivation z and activated output a. ### Formula — which one and why Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total. z means before activation; a means after activation. $$ z=w_1x_1+w_2x_2+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Product, product, sum, bias, phir activation. ### Solution — step by step **Step 1: multiply the first pair.** $$ 0.5(2)=1 $$ **Step 2: multiply the second pair.** $$ (-0.2)(1)=-0.2 $$ **Step 3: add both contributions.** $$ 1+(-0.2)=0.8 $$ **Step 4: add the bias once.** $$ z=0.8+0.1=0.9 $$ **Step 5: apply ReLU.** $$ a=\max(0,0.9)=0.9 $$ **Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged. The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once. --- ## TEE · Perceptrons and the threshold Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s8 A perceptron is a simple neuron used for classification with a threshold decision. If a question says output 1 for z ≥ 0 and 0 otherwise, then z = 0 gives 1. If it says output 1 for z > 0 and 0 otherwise, then z = 0 gives 0. That single equality sign can change an entire training table. ![The weighted sum is a real number. The threshold turns that number into a hard decision.](https://deep-learning-study-room.vercel.app/assets/figures/v_threshold.svg) Diagram: The weighted sum is a real number. The threshold turns that number into a hard decision.. Labels: A perceptron makes a threshold decision | A perceptron makes a threshold decision | Example threshold: return 1 when the weighted sum is greater than 0. | Inputs | x₁ and x₂ | Weighted sum | z = w₁x₁ + w₂x₂ + b | Threshold | z > 0: 1; otherwise: | 0 | Unlike sigmoid, this rule makes a hard 0-or-1 decision. The weighted sum is a real number. The threshold turns that number into a hard decision. Tap the diagram to enlarge. **Question.** Implement binary AND with w₁ = w₂ = 0.6, b = −0.8 and output 1 when z ≥ 0. ### Given w₁ = w₂ = 0.6; bias b = −0.8. Binary inputs; output 1 when z ≥ 0. ### To find Calculate the output for 00,01,10,11 and show that the result is AND. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step ### Input (0, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0+0=0 $$ **Step 4: add the bias once.** $$ z=0+(-0.8)=-0.8 $$ **Step 5: apply the stated threshold.** z = -0.8; output = 0 because output is 1 exactly when z ≥ 0. ### Input (0, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0+0.6=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0.6+0=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0.6+0.6=1.2 $$ **Step 4: add the bias once.** $$ z=1.2+(-0.8)=0.4 $$ **Step 5: apply the stated threshold.** z = 0.4; output = 1 because output is 1 exactly when z ≥ 0. **Final answer:** outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND. ### Summary of the four cases | Inputs | Full net calculation | Net z | Output | AND target | | --- | --- | --- | --- | --- | | 0, 0 | 0.6(0) + 0.6(0) − 0.8 | −0.8 | 0 | 0 | | 0, 1 | 0.6(0) + 0.6(1) − 0.8 | −0.2 | 0 | 0 | | 1, 0 | 0.6(1) + 0.6(0) − 0.8 | −0.2 | 0 | 0 | | 1, 1 | 0.6(1) + 0.6(1) − 0.8 | 0.4 | 1 | 1 | The neuron fires only when both inputs are 1. No weight update was requested, so this is a forward evaluation, not a training problem. For binary targets, a common learning rule is w ← w + η(t − a)x and b ← b + η(t − a). For bipolar targets −1 and +1, the class notes use a different rule: on a mismatch, w ← w + η tx and b ← b + ηt; otherwise keep them. The notes give output 0 at zero threshold, +1 above it and −1 below it. Never combine the activation from one convention with an unexplained update from another. **Training table routine:** copy current weights → calculate z → apply the stated threshold → compare output with target → compute every weight change → update bias → carry new values to the next row. Reset nothing between rows or epochs unless instructed. ### Binary perceptron: target 1 **Question. A binary perceptron has x = [1,2], weights [-0.2, 0.1], bias 0 and target 1. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.** ### Given x = [1,2]; w = [-0.2, 0.1]; b = 0; target = 1; η = 0.1. ### To find Net, threshold output, correction and new weights/bias. ### Formula — which one and why This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias. t is target; o is threshold output; Δ means the amount added to the current parameter. $$ z=\sum_iw_ix_i+b $$ $$ \Delta w_i=\eta(t-o)x_i $$ $$ \Delta b=\eta(t-o) $$ **Yaad rakhne ka cue:** Target minus output; correct prediction ho toh correction zero. ### Solution — step by step **Step 1: calculate the net input.** $$ z=(-0.2)(1)+(0.1)(2)+0=0 $$ **Step 2: apply the threshold.** Net = 0; output = 1 under the stated ≥ 0 rule. **Step 3: calculate the correction.** $$ t-o=1-1=0 $$ **Step 4: update weight 1.** $$ w_1^{\mathrm{new}}=-0.2+0.1(0)(1)=-0.2 $$ **Step 5: update weight 2.** $$ w_2^{\mathrm{new}}=0.1+0.1(0)(2)=0.1 $$ **Step 6: update the bias.** $$ b_{\mathrm{new}}=0+0.1(0)=0 $$ **Final answer:** New weights = [-0.2,0.1]; new bias = 0. ### Binary perceptron: target 0 **Question. A binary perceptron has x = [1,2], weights [0.2, 0.1], bias 0 and target 0. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.** ### Given x = [1,2]; w = [0.2, 0.1]; b = 0; target = 0; η = 0.1. ### To find Net, threshold output, correction and new weights/bias. ### Formula — which one and why This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias. t is target; o is threshold output; Δ means the amount added to the current parameter. $$ z=\sum_iw_ix_i+b $$ $$ \Delta w_i=\eta(t-o)x_i $$ $$ \Delta b=\eta(t-o) $$ **Yaad rakhne ka cue:** Target minus output; correct prediction ho toh correction zero. ### Solution — step by step **Step 1: calculate the net input.** $$ z=(0.2)(1)+(0.1)(2)+0=0.4 $$ **Step 2: apply the threshold.** Net = 0.4; output = 1 under the stated ≥ 0 rule. **Step 3: calculate the correction.** $$ t-o=0-1=-1 $$ **Step 4: update weight 1.** $$ w_1^{\mathrm{new}}=0.2+0.1(-1)(1)=0.1 $$ **Step 5: update weight 2.** $$ w_2^{\mathrm{new}}=0.1+0.1(-1)(2)=-0.1 $$ **Step 6: update the bias.** $$ b_{\mathrm{new}}=0+0.1(-1)=-0.1 $$ **Final answer:** New weights = [0.1,-0.1]; new bias = -0.1. --- ## TEE · Why XOR needs more than one perceptron Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s9 XOR outputs 1 for [0,1] and [1,0], but 0 for [0,0] and [1,1]. On a graph the positive points occupy opposite corners. A single straight line cannot put both positive points on one side and both negative points on the other. A single perceptron has only one linear decision boundary. ![XOR needs a combination of boundaries](https://deep-learning-study-room.vercel.app/assets/figures/xor.svg) Diagram: XOR needs a combination of boundaries. Labels: x1 | x2 | 0 | 1 | 1 | 0 | XOR labels occupy opposite corners | 0 | 1 | 0 | 1 | One straight decision boundary cannot separate the two classes. XOR needs a combination of boundaries Tap the diagram to enlarge. A two-layer construction solves it. Let h₁ = step(x₁ + x₂ − 0.5), which is OR. Let h₂ = step(x₁ + x₂ − 1.5), which is AND. With step(z) = 1 for z ≥ 0, output XOR = step(h₁ − 2h₂ − 0.5). Output XNOR = step(−h₁ + 2h₂ + 0.5). | x₁ x₂ | h₁ OR | h₂ AND | XOR net | XOR | XNOR | | --- | --- | --- | --- | --- | --- | | 0 0 | 0 | 0 | −0.5 | 0 | 1 | | 0 1 | 1 | 0 | 0.5 | 1 | 0 | | 1 0 | 1 | 0 | 0.5 | 1 | 0 | | 1 1 | 1 | 1 | −1.5 | 0 | 1 | More layers help only if appropriate nonlinearities are used. Two purely affine layers collapse into one affine map: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + W₂b₁ + b₂. Affine means linear plus a bias. --- ## TEE · Activation functions and how to calculate them Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s10 ### What an activation function does, and why it is needed A neuron first produces the weighted sum z. The activation function f is a rule that transforms that sum into the output a = f(z). Applying an activation does not change the stored weights; it calculates a new output from the current sum. If every layer only multiplies and adds, several layers still combine into a single affine transformation. A nonlinear activation lets the network represent more complicated relationships. The output activation must also suit the task: sigmoid for a binary-class probability, softmax for mutually exclusive class probabilities, and often a linear output for unrestricted regression. The **activation value** says what the neuron outputs. The **activation derivative** says how quickly that output changes when z changes, which is needed during backpropagation. For sigmoid at z = 0, the value is 0.5 but the derivative is 0.25. They serve different purposes. **Yaad rakhne ka cue:** z = activation se pehle ka sum; a = uske baad ka answer; f′ = answer kitni tezi se badlega. **Definition.** An activation function maps a neuron's preactivation to its output. Nonlinear activations allow a multilayer network to represent nonlinear functions; the output activation also encodes the prediction type. Without a nonlinear activation, a composition of affine layers remains affine. ### Explanation and working $$ \sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ Sigmoid converts any finite real input into a value strictly between 0 and 1. At zero it is 0.5. It is useful for a binary-class probability or independent multilabel outputs. At very positive or negative inputs it saturates, so its derivative becomes small. Its derivative is at most 0.25. ![Trace the output of each curve as the same raw input moves from negative to positive.](https://deep-learning-study-room.vercel.app/assets/figures/v_activations.svg) Diagram: Trace the output of each curve as the same raw input moves from negative to positive.. Labels: See what each activation does | See what each activation does | The horizontal axis is the raw score z; the vertical axis is the output. | Sigmoid | 0 | z | 0 < output < 1 | Tanh | 0 | z | −1 < output < 1 | ReLU | 0 | z | Negative → 0; positive → z | The shape of an activation changes both the output and the gradient passed | backward. Trace the output of each curve as the same raw input moves from negative to positive. Tap the diagram to enlarge. $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1},\quad \tanh'(z)=1-\tanh^2(z) $$ Tanh lies between −1 and +1 and is zero-centered. It is common in recurrent hidden states and LSTM candidates. It also saturates. “Zero-centered” does not mean “no vanishing gradients”. $$ \operatorname{ReLU}(z)=\max(0,z),\quad f'(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ ReLU returns the input if positive and zero if negative. It is cheap to compute and has derivative 1 on its positive side. Units can become inactive on all relevant inputs, known as dying ReLU. At z = 0 the mathematical derivative is undefined; a convention such as 0 is used in implementations and this course’s exercises when needed. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\az&z<0\end{cases},\qquad a=0.01\ \text{unless specified} $$ The small negative slope a lets some gradient through for negative inputs. At z = −2 and a = 0.01, output = −0.02. Do not take the absolute value. PReLU learns the negative slope; Leaky ReLU normally fixes it. $$ p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Softmax acts on a group of class scores, called logits. All probabilities are positive and sum to 1. Subtracting the same maximum c from all logits leaves the result unchanged and avoids very large exponentials. It is suited to mutually exclusive multiclass output. **Full softmax example.** For [−1, 0, 1], exponentials are [0.367879, 1, 2.718282]. Sum = 4.086161. Divide each by that same sum: [0.090031, 0.244728, 0.665241]. Check their sum is approximately 1. Three separate sigmoids would give [0.268941, 0.5, 0.731059], which are different and do not sum to 1. ### Derivation of the sigmoid derivative Write sigmoid as a negative power. The outer derivative contributes −1, and differentiating the inner exponential contributes a second negative sign: $$ \sigma(z)=(1+e^{-z})^{-1} $$ $$ \sigma'(z)=-(1+e^{-z})^{-2}(-e^{-z})=\frac{e^{-z}}{(1+e^{-z})^2} $$ Separate the expression into two factors: $$ \sigma'(z)=\frac{1}{1+e^{-z}}\frac{e^{-z}}{1+e^{-z}}=\sigma(z)[1-\sigma(z)] $$ The second factor is 1−σ(z) because subtracting 1/(1+e⁻ᶻ) from 1 leaves e⁻ᶻ/(1+e⁻ᶻ). Once a forward pass has supplied σ(z), its derivative requires no further exponential evaluation. ### Derivation of the tanh derivative Use the quotient rule on the exponential definition. Here u = e²ᶻ−1 and v = e²ᶻ+1, with u′ = v′ = 2e²ᶻ. $$ \frac{d}{dz}\tanh z=\frac{2e^{2z}(e^{2z}+1)-2e^{2z}(e^{2z}-1)}{(e^{2z}+1)^2}=\frac{4e^{2z}}{(e^{2z}+1)^2} $$ The identity 1−tanh²(z) gives the same expression, since the numerator becomes (e²ᶻ+1)²−(e²ᶻ−1)² = 4e²ᶻ. Therefore the backward-pass factor is 1−a² when a = tanh(z). ### Why nonlinear activations are required For two affine layers without a nonlinear activation: $$ h=W_1x+b_1,\qquad \hat y=W_2h+b_2=W_2W_1x+(W_2b_1+b_2) $$ The composition is still one affine transformation. A nonlinear activation between layers changes the family of functions the network can represent. This is the mathematical reason for using nonlinear hidden activations; merely adding affine layers does not provide this expressive advantage. ### sigmoid example 1 **Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.** ### Given z = -2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(-2)=2 $$ **Step 2: calculate the exponential.** $$ e^{2}\approx7.389056099 $$ **Step 3: add one in the denominator.** $$ 1+7.389056099\approx8.389056099 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: use the activated output in the slope.** $$ f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585 $$ **Final answer:** a ≈ 0.119202922. Derivative ≈ 0.104993585. ### sigmoid example 2 **Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.** ### Given z = 0; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: add one in the denominator.** $$ 1+1.000000000\approx2.000000000 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: use the activated output in the slope.** $$ f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000 $$ **Final answer:** a ≈ 0.500000000. Derivative ≈ 0.250000000. ### sigmoid example 3 **Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.** ### Given z = 2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(2)=-2 $$ **Step 2: calculate the exponential.** $$ e^{-2}\approx0.135335283 $$ **Step 3: add one in the denominator.** $$ 1+0.135335283\approx1.135335283 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: use the activated output in the slope.** $$ f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585 $$ **Final answer:** a ≈ 0.880797078. Derivative ≈ 0.104993585. ### tanh example 1 **Question. For preactivation z = -2, calculate the tanh output and state its derivative.** ### Given z = -2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(-2)=-4 $$ **Step 2: calculate the exponential.** $$ e^{-4}\approx0.018315639 $$ **Step 3: calculate numerator and denominator separately.** $$ 0.018315639-1\approx-0.981684361 $$ $$ 0.018315639+1\approx1.018315639 $$ **Step 4: divide.** $$ a=\frac{-0.981684361}{1.018315639}\approx-0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(-2)=1-(-0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ -0.964027580. Derivative ≈ 0.070650825. ### tanh example 2 **Question. For preactivation z = 0, calculate the tanh output and state its derivative.** ### Given z = 0; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: calculate numerator and denominator separately.** $$ 1.000000000-1\approx0.000000000 $$ $$ 1.000000000+1\approx2.000000000 $$ **Step 4: divide.** $$ a=\frac{0.000000000}{2.000000000}\approx0.000000000 $$ **Step 5: square the activated output for its slope.** $$ f\prime(0)=1-(0.000000000)^2\approx1.000000000 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 1.000000000. ### tanh example 3 **Question. For preactivation z = 2, calculate the tanh output and state its derivative.** ### Given z = 2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(2)=4 $$ **Step 2: calculate the exponential.** $$ e^{4}\approx54.598150033 $$ **Step 3: calculate numerator and denominator separately.** $$ 54.598150033-1\approx53.598150033 $$ $$ 54.598150033+1\approx55.598150033 $$ **Step 4: divide.** $$ a=\frac{53.598150033}{55.598150033}\approx0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(2)=1-(0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ 0.964027580. Derivative ≈ 0.070650825. ### ReLU example 1 **Question. For preactivation z = -2, calculate the ReLU output and state its derivative.** ### Given z = -2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 0.000000000. ### ReLU example 2 **Question. For preactivation z = 0, calculate the ReLU output and state its derivative.** ### Given z = 0; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### ReLU example 3 **Question. For preactivation z = 2, calculate the ReLU output and state its derivative.** ### Given z = 2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ### Leaky ReLU example 1 **Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = -2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=-0.2 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0.1 $$ **Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000. ### Leaky ReLU example 2 **Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 0; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### Leaky ReLU example 3 **Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ### Softmax for [0, 0, 0] **Question. Find stable softmax probabilities for logits [0, 0, 0].** ### Given Logits = [0, 0, 0]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}0&0&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{0}\approx1.000000000 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{0}\approx1.000000000 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=1.000000000+1.000000000+1.000000000\approx3.000000000 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding. ### Softmax for [1, 2, 3] **Question. Find stable softmax probabilities for logits [1, 2, 3].** ### Given Logits = [1, 2, 3]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}-2&-1&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{-2}\approx0.135335283 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{-1}\approx0.367879441 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=0.135335283+0.367879441+1.000000000\approx1.503214724 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{0.135335283}{1.503214724}\approx0.090030573 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{0.367879441}{1.503214724}\approx0.244728471 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{1.503214724}\approx0.665240956 $$ **Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding. --- ## TEE · Model answer: why a network needs nonlinear activation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s11 ## Question **Question.** Why are activation functions required in a multilayer neural network? Compare sigmoid, tanh and ReLU, including their limitations. ## Answer — step by step An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary. ![Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.](https://deep-learning-study-room.vercel.app/assets/figures/v_nonlinear.svg) Diagram: Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.. Labels: Two affine layers still make one affine rule | Two affine layers still make one affine rule | Without a nonlinear activation, these two stages can be combined. | Layer 1 and layer 2 | h = 2x + 1 | y = 3h + 4 | Substitute and simplify | y = 3(2x + 1) + 4 | y = 6x + 7 | A nonlinear activation between layers prevents this simple affine collapse. Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation. Tap the diagram to enlarge. ### Why two linear layers are still linear in the input For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second: $$ y=3(2x+1)+4=6x+3+4=6x+7 $$ This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent. | Activation | Output range | Useful property | Main limitation | | --- | --- | --- | --- | | Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small | | Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients | | ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient | Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score. **Exam conclusion:** select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction. --- ## TEE · Choosing a loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s12 **Definition.** A loss function quantifies the discrepancy between a prediction and its target. The training objective aggregates per-example losses and may add regularization. The loss must be compatible with the task and output parameterization. ### Explanation and working Activation produces the prediction. Loss measures how wrong that prediction is. An optimizer uses gradients to update parameters. These are three different roles. ![First identify what the prediction means. Then choose the loss paired with that task and output.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_choice.svg) Diagram: First identify what the prediction means. Then choose the loss paired with that task and output.. Labels: Match the loss to what the output means | Match the loss to what the output means | A probability and an unrestricted numerical prediction need different | interpretations. | Predict a numerical quantity | Output: a price or temperature | Typical loss: MSE or MAE | Predict a class probability | Binary: sigmoid + BCE | Multiclass: softmax + CCE | The loss measures the mistake. The activation determines the output | transformation. First identify what the prediction means. Then choose the loss paired with that task and output. Tap the diagram to enlarge. $$ \operatorname{MSE}=\frac1n\sum_{i=1}^n(\hat y_i-y_i)^2,\qquad \operatorname{MAE}=\frac1n\sum_{i=1}^n|\hat y_i-y_i| $$ **Question.** Targets are [2, 4], predictions are [1, 6]. Find MSE and MAE. ### Given Targets = [2,4]; predictions = [1,6]; n = 2. ### To find Find mean squared error and mean absolute error. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. **Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes. n = number of predictions. |e| means the nonnegative magnitude of error e. $$ \mathrm{MAE}=\frac1n\sum_i|\hat y_i-y_i| $$ **Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square. ### Solution — step by step **Step 1: calculate prediction minus target.** $$ e_1=1-2=-1 $$ $$ e_2=6-4=2 $$ **Step 2: square both errors.** $$ e_1^2=(-1)^2=1 $$ $$ e_2^2=2^2=4 $$ **Step 3: add squares and divide by two observations.** $$ \mathrm{MSE}=\frac{1+4}{2}=2.5 $$ **Step 4: take the absolute value of each original error.** $$ |-1|=1,\qquad |2|=2 $$ **Step 5: add absolute errors and divide by two.** $$ \mathrm{MAE}=\frac{1+2}{2}=1.5 $$ **Final answer:** MSE = 2.5; MAE = 1.5. Squaring gives larger errors more influence; MAE is not differentiable at zero error. $$ L_{\mathrm{BCE}}=-[y\ln p+(1-y)\ln(1-p)],\qquad L_{\mathrm{CCE}}=-\sum_k y_k\ln p_k $$ For binary y = 1, BCE becomes −ln(p). For y = 0, it becomes −ln(1 − p). For one-hot multiclass labels, CCE is simply minus the natural log of the predicted probability assigned to the correct class. **Question.** True class is Mild, encoded [0,1,0], and probabilities are [0.1,0.8,0.1]. Find CCE. ### Given True class = Mild; one-hot target = [0,1,0]. Predicted probabilities = [0.1,0.8,0.1]. ### To find Find the categorical cross-entropy. ### Formula — which one and why **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step **Step 1: write one weighted log term for each class.** $$ L=-[0\ln(0.1)+1\ln(0.8)+0\ln(0.1)] $$ **Step 2: remove terms with zero target weight.** $$ L=-\ln(0.8) $$ **Step 3: evaluate the natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 4: apply the outer minus sign.** $$ L\approx-(-0.223143551)=0.223143551 $$ **Final answer:** CCE ≈ 0.223144. Only the correct-class probability contributes for this one-hot target. | Task | Typical output | Typical loss | | --- | --- | --- | | Unbounded numerical regression | One linear output | MSE or MAE | | Binary classification | One sigmoid output | Binary cross-entropy | | Exactly one of K classes | K softmax outputs | Categorical cross-entropy | | Several independent labels | K sigmoid outputs | Binary cross-entropy per label | A continuous rating restricted to 0–10 can use a linear output with a stated range policy, or 10 × sigmoid with an appropriate regression loss. If ratings are treated as discrete ordered categories, modelling choices differ. State the interpretation instead of selecting a loss merely from the number 10. ### Squared-error scaling in backpropagation For n included scalar predictions, let eᵢ = ŷᵢ−yᵢ. Three frequently used objectives are: $$ \mathrm{SSE}=\sum_i e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_i e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ Differentiating with respect to one prediction produces different factors: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_i}=2e_i,\quad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_i}=e_i,\quad \frac{\partial\mathrm{MSE}}{\partial\hat y_i}=\frac2n e_i $$ Use the exact objective in the question. The teacher's two-output backpropagation example uses half SSE. Do not automatically insert the MSE factor into that example. ### Why cross-entropy uses the negative logarithm For a one-hot target, the likelihood of the observed class is the probability assigned to it. Maximizing that likelihood is equivalent to minimizing its negative logarithm. For independent training examples, multiplying probabilities becomes adding their logarithms. A low correct-class probability therefore produces a large loss. ### Sigmoid with binary cross-entropy: output delta Let p = σ(z). First differentiate the loss with respect to p, holding y constant: $$ \frac{\partial L}{\partial p}=-\frac yp+\frac{1-y}{1-p}=\frac{p-y}{p(1-p)} $$ Then multiply by the sigmoid derivative: $$ \frac{\partial L}{\partial z}=\frac{\partial L}{\partial p}\frac{\partial p}{\partial z}=\frac{p-y}{p(1-p)}p(1-p)=p-y $$ Thus the logit delta is p−y for this paired output and loss. With sigmoid and half squared error it is instead (p−y)p(1−p). Confusing these two cases changes every subsequent gradient. ### Softmax with categorical cross-entropy: output delta For a normalized target vector, the derivative of softmax component pᵢ with respect to logit zⱼ is pᵢ times (δᵢⱼ−pⱼ), where δᵢⱼ is 1 if i = j and 0 otherwise. Therefore: $$ \frac{\partial L}{\partial z_j}=-\sum_i\frac{y_i}{p_i}p_i(\delta_{ij}-p_j)=-y_j+p_j\sum_i y_i=p_j-y_j $$ This result assumes the target components sum to 1. For a mean loss over a batch, include the corresponding averaging factor as well. ### Three squared-error conventions: [1, 3] **Question. Predictions are [1, 3] and targets are [2, 3]. Find SSE, half SSE and MSE.** ### Given Predictions = [1, 3]. Targets = [2, 3]. Number of scalar predictions n = 2. ### To find SSE, half SSE and MSE separately. ### Formula — which one and why All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention. ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Square sab mein; half mein 2, mean mein n se divide. ### Solution — step by step **Step 1: calculate error 1.** $$ e_1=1-2=-1 $$ **Step 2: square error 1.** $$ e_1^2=(-1)^2=1 $$ **Step 3: calculate error 2.** $$ e_2=3-3=0 $$ **Step 4: square error 2.** $$ e_2^2=(0)^2=0 $$ **Step 5: add squared errors.** $$ \mathrm{SSE}=1+0=1 $$ **Step 6: divide by 2 for half SSE.** $$ L_{\mathrm{half}}=\frac{1}{2}=0.5 $$ **Step 7: divide by the number of predictions for MSE.** $$ \mathrm{MSE}=\frac{1}{2}\approx0.500000000 $$ **Final answer:** SSE = 1; half SSE = 0.5; MSE ≈ 0.500000000. ### Three squared-error conventions: [2, 5, 1] **Question. Predictions are [2, 5, 1] and targets are [1, 3, 2]. Find SSE, half SSE and MSE.** ### Given Predictions = [2, 5, 1]. Targets = [1, 3, 2]. Number of scalar predictions n = 3. ### To find SSE, half SSE and MSE separately. ### Formula — which one and why All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention. ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Square sab mein; half mein 2, mean mein n se divide. ### Solution — step by step **Step 1: calculate error 1.** $$ e_1=2-1=1 $$ **Step 2: square error 1.** $$ e_1^2=(1)^2=1 $$ **Step 3: calculate error 2.** $$ e_2=5-3=2 $$ **Step 4: square error 2.** $$ e_2^2=(2)^2=4 $$ **Step 5: calculate error 3.** $$ e_3=1-2=-1 $$ **Step 6: square error 3.** $$ e_3^2=(-1)^2=1 $$ **Step 7: add squared errors.** $$ \mathrm{SSE}=1+4+1=6 $$ **Step 8: divide by 2 for half SSE.** $$ L_{\mathrm{half}}=\frac{6}{2}=3 $$ **Step 9: divide by the number of predictions for MSE.** $$ \mathrm{MSE}=\frac{6}{3}\approx2.000000000 $$ **Final answer:** SSE = 6; half SSE = 3; MSE ≈ 2.000000000. ### Three squared-error conventions: [4] **Question. Predictions are [4] and targets are [1]. Find SSE, half SSE and MSE.** ### Given Predictions = [4]. Targets = [1]. Number of scalar predictions n = 1. ### To find SSE, half SSE and MSE separately. ### Formula — which one and why All three use the same squared errors but divide the sum differently. The question explicitly asks for each convention. ŷᵢ is a prediction; yᵢ its target; n counts predictions, not network layers. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Square sab mein; half mein 2, mean mein n se divide. ### Solution — step by step **Step 1: calculate error 1.** $$ e_1=4-1=3 $$ **Step 2: square error 1.** $$ e_1^2=(3)^2=9 $$ **Step 3: add squared errors.** $$ \mathrm{SSE}=9=9 $$ **Step 4: divide by 2 for half SSE.** $$ L_{\mathrm{half}}=\frac{9}{2}=4.5 $$ **Step 5: divide by the number of predictions for MSE.** $$ \mathrm{MSE}=\frac{9}{1}\approx9.000000000 $$ **Final answer:** SSE = 9; half SSE = 4.5; MSE ≈ 9.000000000. ### Binary cross-entropy: target 1, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.8 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.223143551)=0.223143551 $$ **Final answer:** Loss ≈ 0.223143551 nats. ### Binary cross-entropy: target 0, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 0; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ 1-p=1-0.8=0.2 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.2)\approx-1.609437912 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-1.609437912)=1.609437912 $$ **Final answer:** Loss ≈ 1.609437912 nats. ### Binary cross-entropy: target 1, predicted positive-class probability 0.5 **Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.5; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.5 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.693147181)=0.693147181 $$ **Final answer:** Loss ≈ 0.693147181 nats. ### Sigmoid+BCE logit gradient: p=0.8, y=1 **Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 1; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(-1.25)(0.16)=-0.2 $$ **Final answer:** dL/dz = -0.2. No further sigmoid derivative is needed. ### Sigmoid+BCE logit gradient: p=0.8, y=0 **Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 0; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-0}{0.16}=5 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(5)(0.16)=0.8 $$ **Final answer:** dL/dz = 0.8. No further sigmoid derivative is needed. --- ## TEE · Forward propagation through multiple neurons Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s13 **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations. ### Input, hidden and output layers The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task. In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections. ![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg) Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights. A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge. ### General layer equations and dimensions Let a⁽⁰⁾ = x. For layer l: $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f^{(l)}(z^{(l)}) $$ If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j. $$ W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1} $$ For one hidden layer, these computations compose to give: $$ h=f^{(1)}(W^{(1)}x+b^{(1)}) $$ $$ \hat y=f^{(2)}(W^{(2)}h+b^{(2)}) $$ The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer. ### Forward propagation versus training Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture. ### Complete two-layer numerical **Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.** ### Given A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. ### To find Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg) Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge. **Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output. **Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias. $$ z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3 $$ **Step 3: calculate the second hidden preactivation.** $$ z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5 $$ **Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them. $$ a^{(1)}_1=\max(0,0.3)=0.3 $$ $$ a^{(1)}_2=\max(0,0.5)=0.5 $$ $$ a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix} $$ **Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs. $$ z^{(2)}=0.5(0.3)+0.6(0.5)+0.1 $$ $$ z^{(2)}=0.15+0.30+0.10=0.55 $$ **Step 6: apply the output activation.** $$ \hat y=\max(0,0.55)=0.55 $$ **Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass. --- ## TEE · Backpropagation with one hidden neuron Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s14 **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. --- ## TEE · More outputs and vectorized backpropagation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s15 **Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path. ### Why the hidden derivative is a sum If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives: $$ \frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2} $$ For any number of connected outputs, sum their contributions and then differentiate the hidden activation: $$ \delta_{h_j}=\left(\sum_k W_{2,kj}\delta_{o_k}\right)f'(z_{h_j}) $$ Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss. ![The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_branch_gradient.svg) Diagram: The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.. Labels: Two output paths both affect the hidden neuron | Two output paths both affect the hidden neuron | Example: hidden activation h = 0.6, using sigmoid in the hidden neuron. | h = 0.6 | Output 1 delta = 0.2 | Connecting weight = 0.4 | Output 2 delta = −0.1 | Connecting weight = 0.5 | Downstream sum = 0.2 × 0.4 + (−0.1) × 0.5 = 0.03 | Hidden delta = 0.03 × 0.6 × (1 − 0.6) = 0.0072 | Add the two downstream paths first. Then multiply by the hidden activation | derivative. The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative. Tap the diagram to enlarge. ### Matrix equations for one training sample Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂: $$ L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2 $$ $$ \delta_2=(\hat y-y)\odot f_2'(z_2) $$ $$ \frac{\partial L}{\partial W_2}=\delta_2a_1^T,\qquad\frac{\partial L}{\partial b_2}=\delta_2 $$ $$ \delta_1=(W_2^T\delta_2)\odot f_1'(z_1) $$ $$ \frac{\partial L}{\partial W_1}=\delta_1x^T,\qquad\frac{\partial L}{\partial b_1}=\delta_1 $$ Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation. | Quantity | Shape | Reason | | --- | --- | --- | | W₁ and its gradient | m × d | m receiving hidden neurons, d inputs | | W₂ and its gradient | k × m | k receiving output neurons, m hidden activations | | δ₂ | k × 1 | One derivative per output preactivation | | W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit | | δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ | ### How the loss changes the output delta For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target: $$ \delta_{\mathrm{output}}=\hat y-y $$ That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required. For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question. --- ## TEE · Model answer: forward propagation, backpropagation and updating Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s16 ## Question **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations. ## Answer — step by step Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs. ![Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge. | Operation | Direction or action | Result | | --- | --- | --- | | Forward propagation | Inputs move through successive layers | Predictions and intermediate activations | | Loss calculation | Compare predictions with targets | A scalar objective for the example or batch | | Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters | | Optimizer update | Use those gradients to change parameters | A new set of weights and biases | ### 1. Forward propagation At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ ### 2. Loss Use the loss specified by the task. For one regression output, half squared loss is one possible choice: $$ L=\frac12(a-y)^2 $$ Here a is the prediction and y is the target. Other stated losses produce different derivatives. ### 3. Backpropagation Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it. ### 4. Update and repeat $$ w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w} $$ Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations. --- ## TEE · Counting parameters in a dense network Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s17/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s17 **Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases. ### Deriving the parameter count For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters: $$ P_{\mathrm{dense}}=mn+n=(m+1)n $$ Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram. ![Count all input-to-neuron connections and then one bias per receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/v_dense_count.svg) Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters. Count all input-to-neuron connections and then one bias per receiving neuron. Tap the diagram to enlarge. ### Complete numerical **Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters. ### Given Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs. Every hidden/output neuron has one bias. ### To find Count weights and biases at each receiving layer, then the total. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step **Step 1: identify the consecutive layer widths.** $$ 10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4 $$ There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters. **Step 2: count weights and biases for each transformation.** | Transformation | Weights | Biases | Layer total | | --- | --- | --- | --- | | Input → hidden 1 | 10 × 5 = 50 | 5 | 55 | | Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 | | Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 | | Hidden 3 → output | 5 × 4 = 20 | 4 | 24 | **Step 3: add the layer totals.** $$ P=55+30+30+24=139 $$ Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139. **Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network. --- ## TEE · What you should understand before optimization Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s18/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s18 You should be able to explain why a target and prediction differ, compute a weighted sum, choose an activation, measure a loss, find a one-neuron gradient and update a weight. If these steps are weak, repeat the first training example and the ReLU backprop example. They contain the entire calculation pattern that the next chapters extend. ![These four operations form the training loop used by the optimizers in the next chapter.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: These four operations form the training loop used by the optimizers in the next chapter.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. These four operations form the training loop used by the optimizers in the next chapter. Tap the diagram to enlarge. --- ## TEE · Complete sigmoid training example Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s19/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s19 **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. --- ## TEE · Cross-entropy for probability distributions Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s20/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s20 **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities. $$ H(p,q)=-\sum_{i=1}^{K}p_i\log q_i $$ Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms. ### Logarithm base and units Natural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2: $$ \log_2 q=\frac{\ln q}{\ln 2} $$ For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities. ### A target distribution with two nonzero entries **Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.** ### Given A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms. ### To find Find cross-entropy in the requested units. ### Formula — which one and why **Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute. pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ H_2(p,q)=\frac{H(p,q)}{\ln2} $$ **Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai. ### Solution — step by step **Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded. $$ H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)] $$ **Step 2: calculate the logarithms.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: weight each negative logarithm by its target probability.** $$ -0.75\ln(0.5)\approx0.519860385 $$ $$ -0.25\ln(0.5)\approx0.173286795 $$ **Step 4: add the contributions, retaining calculator precision.** $$ H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats} $$ **Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1. $$ H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit} $$ **Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale. ### Cross-entropy, entropy and relative entropy These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p) $$ For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus. **Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.** ### Given For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats. ### To find Find entropy and KL divergence in nats. ### Formula — which one and why **Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p. p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p) $$ **Yaad rakhne ka cue:** KL = cross-entropy minus target ki own entropy. ### Solution — step by step **Step 1: calculate the target entropy.** $$ H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)] $$ $$ \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361 $$ $$ H(p)\approx0.215761554+0.346573590=0.562335145 $$ **Step 2: subtract target entropy from cross-entropy.** $$ D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036 $$ **Final answer:** H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ. --- ## TEE · Biological neurons and the McCulloch-Pitts model Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c4-s21/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/4/tee-c4-s21 **Definition.** An artificial neuron is a simplified computational model inspired by the collection, combination and transmission of signals in biological neurons. This analogy motivates weighted inputs and an output response; it is not a full simulation of a living neuron. ### Biological and artificial components | Biological component | Simplified artificial counterpart | Function in the analogy | | --- | --- | --- | | Dendrites | Input connections | Receive incoming signals | | Synapses | Connection weights | Influence how strongly each incoming signal contributes | | Cell body or soma | Aggregation stage | Combines the weighted signals | | Firing behavior | Threshold or activation function | Determines the model's output response | | Axon | Output connection | Passes the resulting signal onward | For an artificial threshold unit, a threshold θ on the weighted sum can be written as a bias b = −θ, followed by a threshold at zero. The equality convention must be preserved. $$ \sum_i w_ix_i\ge\theta\quad\Longleftrightarrow\quad\sum_i w_ix_i-\theta\ge0 $$ ### McCulloch–Pitts model and perceptron The McCulloch–Pitts model is an early binary threshold model. Its logical connections and threshold are prescribed to implement a desired behavior; it has no data-driven weight-update procedure. The original formulation also distinguishes excitatory inputs from inhibitory inputs, which can prevent firing. In elementary logic-gate examples using only excitatory inputs, a threshold on their sum is sufficient. A perceptron introduces adjustable connection weights and a learning rule. It calculates a weighted sum, applies a threshold and updates parameters when its classification disagrees with the target. Both single threshold units have a linear decision boundary; neither single unit solves XOR in the original two-input space. **Question. Implement AND and OR for binary inputs x₁, x₂ using the sum x₁ + x₂ and output 1 when the sum is at least θ. Use θ = 2 for AND and θ = 1 for OR.** ### Given Implement AND and OR for binary inputs x₁, x₂ using the sum x₁ + x₂ and output 1 when the sum is at least θ. Use θ = 2 for AND and θ = 1 for OR. ### To find Write every row of the truth table and the requested network output. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step Calculate the sum once for each input pair, then compare it with the relevant threshold. ### Binary input (0,0) **Step 1: add the inputs.** $$ s=0+0=0 $$ **Step 2: compare with AND threshold 2.** 0 ≥ 2 is false; AND output = 0. **Step 3: compare with OR threshold 1.** 0 ≥ 1 is false; OR output = 0. ### Binary input (0,1) **Step 1: add the inputs.** $$ s=0+1=1 $$ **Step 2: compare with AND threshold 2.** 1 ≥ 2 is false; AND output = 0. **Step 3: compare with OR threshold 1.** 1 ≥ 1 is true; OR output = 1. ### Binary input (1,0) **Step 1: add the inputs.** $$ s=1+0=1 $$ **Step 2: compare with AND threshold 2.** 1 ≥ 2 is false; AND output = 0. **Step 3: compare with OR threshold 1.** 1 ≥ 1 is true; OR output = 1. ### Binary input (1,1) **Step 1: add the inputs.** $$ s=1+1=2 $$ **Step 2: compare with AND threshold 2.** 2 ≥ 2 is true; AND output = 1. **Step 3: compare with OR threshold 1.** 2 ≥ 1 is true; OR output = 1. ### Truth-table summary | x₁ | x₂ | x₁ + x₂ | AND: sum ≥ 2 | OR: sum ≥ 1 | | --- | --- | --- | --- | --- | | 0 | 0 | 0 + 0 = 0 | 0 | 0 | | 0 | 1 | 0 + 1 = 1 | 0 | 1 | | 1 | 0 | 1 + 0 = 1 | 0 | 1 | | 1 | 1 | 1 + 1 = 2 | 1 | 1 | The parameters have been chosen to implement the truth tables. This calculation is a logic-gate evaluation, not a perceptron training epoch. --- ## TEE · Reading an optimizer formula without getting lost Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s0 An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated. ![Old value, current gradient and new value belong to different moments in the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. Old value, current gradient and new value belong to different moments in the update. Tap the diagram to enlarge. ### Old, current and new At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position. $$ \theta_t=\theta_{t-1}-\eta g_t $$ η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition. ### What a stored state means An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares. ### A weighted average from ordinary arithmetic If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square. $$ 0.9\times4=3.6,\qquad(1-0.9)\times9=0.9 $$ $$ s_{\mathrm{new}}=3.6+0.9=4.5 $$ The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum. ### Squared memory and signed direction For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g². ### Epsilon and hats ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second. --- ## TEE · Optimizer notation: old value, memory, new value Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s1 ### 1. A parameter and a stored state are different θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries. ![The time label tells you which value to use. Carry the previous memory into the current update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: The time label tells you which value to use. Carry the previous memory into the current update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. The time label tells you which value to use. Carry the previous memory into the current update. Tap the diagram to enlarge. At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values. ### 2. A supplied gradient versus a supplied objective If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead. For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64. ### 3. A squared gradient keeps size and loses sign Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square. ### 4. Square root and epsilon belong to the denominator If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero. ### 5. A table prevents mixing iterations Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1. The worked examples below use this exact order, with every state shown. --- ## TEE · What an optimizer is trying to do Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s2 **Definition.** Optimization is the process of selecting model parameters to minimize a scalar objective. In neural-network training, a gradient-based optimizer updates weights and biases using derivatives of the training objective. ### Objective and gradient For empirical risk with an optional regularizer: $$ J(\theta)=\frac1N\sum_{i=1}^{N}L_i(\theta)+\lambda\Omega(\theta) $$ The loss value indicates the current objective. Its gradient indicates local sensitivity to parameters. Backpropagation computes this gradient efficiently; the optimizer determines how the parameters are updated from it. A gradient and a loss value cannot be interchanged in the update equation. ### Why gradient descent subtracts the gradient For a small displacement Δθ, first-order Taylor expansion approximates the objective by: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ Choose Δθ = −η∇J(θ), with η > 0. Substitution gives: $$ J(\theta-\eta\nabla J)\approx J(\theta)-\eta\|\nabla J\|_2^2 $$ The squared norm is nonnegative. Therefore the negative-gradient direction is locally a descent direction when the gradient is nonzero. The approximation explains the sign; it does not guarantee that an arbitrarily large finite step decreases loss. ![A gradient specifies a local direction; the learning rate determines the finite parameter step.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: A gradient specifies a local direction; the learning rate determines the finite parameter step.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. A gradient specifies a local direction; the learning rate determines the finite parameter step. Tap the diagram to enlarge. ### Local minima, global minima and saddle points A local minimum has objective value no larger than nearby points. A global minimum is minimal over the whole feasible domain. At a saddle point, the objective increases in some directions and decreases in others. A small gradient alone does not establish a minimum: a stationary point can also be a maximum or saddle. Neural-network objectives are generally nonconvex. Training can encounter flat regions, ill-conditioned valleys, noisy gradient estimates and poor step sizes. Momentum modifies the update direction using previous steps; adaptive methods rescale parameter-specific updates. These methods improve particular optimization behavior, without guaranteeing the global optimum or best test performance. --- ## TEE · The ordinary gradient descent update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s3 **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate. ### Explanation and working $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent. Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction. ![Calculate the slope first, multiply it by the learning rate, then subtract that correction.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: Calculate the slope first, multiply it by the learning rate, then subtract that correction.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. Calculate the slope first, multiply it by the learning rate, then subtract that correction. Tap the diagram to enlarge. **Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations. ### Given J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1. ### To find Differentiate J and perform two gradient-descent iterations. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ². **Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79. **Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061. **Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**. The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum. **Common error:** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients. --- ## TEE · Batch stochastic and mini batch gradient descent Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s4 **Definition.** Batch, stochastic and mini-batch gradient descent differ in the subset of training examples used to estimate the gradient for each parameter update. Their update equation is the same; their gradient estimates and update frequency differ. ### Explanation and working All three use the same update idea. They differ in how much data contributes to one gradient. ![Count outlined groups, not individual cells, to count gradient updates in one epoch.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: Count outlined groups, not individual cells, to count gradient updates in one epoch.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. Count outlined groups, not individual cells, to count gradient updates in one epoch. Tap the diagram to enlarge. | Method | Data for one update | Updates in one epoch with N samples | Typical behaviour | | --- | --- | --- | --- | | Batch GD | All N samples | 1 | Stable full-data gradient; costly updates | | Stochastic GD | One sample | N | Frequent noisy updates | | Mini-batch GD | B samples | Ceiling of N/B if final partial batch is retained | Efficient parallel calculation with moderate noise | **Question.** N = 1,000 and B = 100. Find updates per epoch. ### Given N = 1,000 examples; mini-batch size B = 100; one epoch. ### To find Count full-batch, stochastic and mini-batch updates per epoch. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step **Step 1: count full-batch updates.** All 1,000 samples form one group, so one epoch has 1 update. **Step 2: count stochastic updates.** One sample per update gives 1,000 updates per epoch. **Step 3: count mini-batches.** $$ \frac{N}{B}=\frac{1000}{100}=10 $$ **Final answer:** batch = 1; SGD = 1,000; mini-batch = 10 updates per epoch. ### Mini-batch calculation with a remainder **Question. A dataset contains 105 samples and the mini-batch size is 20. Keeping the incomplete last batch, find the updates per epoch.** ### Given N = 105; B = 20; retain the final incomplete batch. ### To find The number of updates per epoch. ### Formula — which one and why Every processed group causes one update. A remaining nonempty group also counts. The ceiling symbol rounds upward to the next integer. $$ U=\lceil N/B\rceil $$ **Yaad rakhne ka cue:** Bacha hua batch process kiya toh ek aur update. ### Solution — step by step **Step 1: count complete batches.** $$ 5(20)=100 $$ **Step 2: count remaining examples.** $$ 105-100=5 $$ **Step 3: include the nonempty last batch.** $$ U=5+1=6 $$ **Final answer:** 6 updates. If a question explicitly drops the final incomplete batch, it instead gives 5. --- ## TEE · Model answer: batch, stochastic and mini-batch training Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s5 ## Question **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration. ## Answer — step by step All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update. ![All methods visit the same six examples. Their grouping determines how often they update.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: All methods visit the same six examples. Their grouping determines how often they update.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge. A **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations. | Basis | Batch GD | Stochastic GD | Mini-batch GD | | --- | --- | --- | --- | | Examples in one gradient | Entire training dataset | One example | A smaller group of B examples | | Updates per epoch | One | N | Ceiling of N/B if the remainder is retained | | Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging | | Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation | ### Example to distinguish the three counts With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch. The basic parameter update is unchanged: $$ \theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g $$ For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples. --- ## TEE · A tiny dataset shows why the methods differ Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s6 **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order. ### Given Model ŷ = wx; initial w = 1; η = 0.1. Samples in order: (1,2), (2,4). Mean half squared loss for the batch. ### To find Compare one batch update with a sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### One batch update **Step 1: predict sample 1 with the original weight.** $$ \hat y_1=1(1)=1 $$ **Step 2: calculate its error.** $$ e_1=1-2=-1 $$ **Step 3: multiply its error by its input.** $$ g_1=(-1)(1)=-1 $$ **Step 4: predict sample 2 with the same original weight.** $$ \hat y_2=1(2)=2 $$ **Step 5: calculate its error.** $$ e_2=2-4=-2 $$ **Step 6: multiply its error by its input.** $$ g_2=(-2)(2)=-4 $$ **Step 7: average the two gradients.** $$ g=\frac{-1+(-4)}2=-2.5 $$ **Step 8: update once.** $$ w_{\mathrm{new}}=1-0.1(-2.5)=1.25 $$ ![This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge. ### One sequential SGD epoch **Step 1: use the first sample gradient.** $$ g_1=-1 $$ **Step 2: update after the first sample.** $$ w_1=1-0.1(-1)=1.1 $$ **Step 3: predict sample 2 using the changed weight.** $$ \hat y_2=1.1(2)=2.2 $$ **Step 4: find sample 2 error.** $$ e_2=2.2-4=-1.8 $$ **Step 5: find the new sample 2 gradient.** $$ g_2=(-1.8)(2)=-3.6 $$ **Step 6: update after sample 2.** $$ w_2=1.1-0.1(-3.6)=1.46 $$ **Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples. Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim. --- ## TEE · Learning rate and common training problems Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s7 **Definition.** The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss. ### Explanation and working A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate. ![Illustrative loss traces show why smoothness and progress are different](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative loss traces show why smoothness and progress are different. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative loss traces show why smoothness and progress are different Tap the diagram to enlarge. | Observation | Possible reason | Sensible first action | | --- | --- | --- | | Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation | | Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate | | Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum | | Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping | | All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights | Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum. --- ## TEE · A moving average remembers the recent past Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s8 **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients. $$ m_t=\beta m_{t-1}+(1-\beta)g_t,\qquad0\le\beta<1 $$ For m₀ = 0, expanding the recurrence gives: $$ m_t=(1-\beta)\sum_{k=1}^{t}\beta^{t-k}g_k $$ The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history. ### Signed gradients versus squared gradients An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form. **Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂. ### Given β = 0.9; initial m₀ = 0. Gradients in order: g₁ = 2, g₂ = −1. ### To find Find m₁ and m₂, preserving the sign of each gradient. ### Formula — which one and why **Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state. β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state. $$ m_t=\beta m_{t-1}+(1-\beta)g_t $$ **Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai. ### Solution — step by step **Step 1: find the recent-gradient coefficient.** $$ 1-0.9=0.1 $$ **Step 2: retain the old memory for step 1.** $$ 0.9(0)=0 $$ **Step 3: weight the first gradient.** $$ 0.1(2)=0.2 $$ **Step 4: add to obtain the first memory.** $$ m_1=0+0.2=0.2 $$ **Step 5: retain the first memory for step 2.** $$ 0.9(0.2)=0.18 $$ **Step 6: weight the second gradient.** $$ 0.1(-1)=-0.1 $$ **Step 7: add to obtain the second memory.** $$ m_2=0.18+(-0.1)=0.08 $$ **Final answer:** m₁ = 0.2; m₂ = 0.08. The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable. --- ## TEE · Momentum remembers movement Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s9 **Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions. ### Update equations and convention This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter: $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters. ![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg) Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients. Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration. **Step 1: Iteration 1: gradient, velocity, parameter.** $$ g_1=J'(1)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Iteration 2: use θ₁ and v₁.** $$ g_2=J'(0.9)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ | Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter | | --- | --- | --- | --- | --- | | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.9 | −0.18 | 0.72 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient. ### Properties and alternative notation Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective. Some texts store a positive gradient accumulator u instead: $$ u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t $$ With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions. --- ## TEE · Nesterov looks before correcting Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s10 **Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update. ### What changes relative to momentum Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured: $$ \theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look},t}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁. ![Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective. **Iteration 1 — step 1: locate the lookahead point.** $$ \theta_{\mathrm{look},1}=1+0.9(0)=1 $$ **Step 2: evaluate the gradient there.** $$ g_1=J'(1)=1 $$ **Step 3: calculate the new velocity and actual parameter.** $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Iteration 2 — step 1: use the previous actual parameter and velocity.** $$ \theta_{\mathrm{look},2}=0.9+0.9(-0.1)=0.81 $$ **Step 2: evaluate the gradient at 0.81, not at 0.9.** $$ g_2=J'(0.81)=0.81 $$ **Step 3: calculate the new velocity.** $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ **Step 4: update the old actual parameter.** $$ \theta_2=0.9+(-0.171)=0.729 $$ | Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ | | --- | --- | --- | --- | --- | --- | | 1 | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice. --- ## TEE · Model answer: momentum and Nesterov Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s11 ## Question **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ? ## Answer — step by step Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions. ![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge. ### Momentum formulas This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained. $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ### Nesterov's change Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look}}) $$ The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point. | Basis | Momentum | Nesterov | | --- | --- | --- | | Gradient evaluated at | Current parameter | Lookahead parameter | | Information retained | Previous velocity | Previous velocity | | Intuition | Continue useful movement | Inspect the slope where old momentum points | | Initial zero velocity | No movement contribution yet | First lookahead equals current parameter | Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently. --- ## TEE · AdaGrad adds every squared gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s12 **Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator. ### State, effective learning rate and update For one parameter θ, let gₜ be its current gradient and s₀ = 0: $$ s_t=s_{t-1}+g_t^2 $$ $$ \eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon} $$ $$ \theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t $$ s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign. ![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg) Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update. The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge. ### Complete numerical: supplied gradients **Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter. **Iteration 1 — step 1: update the accumulator.** $$ s_1=0+2^2=4 $$ **Step 2: calculate the effective learning rate.** $$ \eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05 $$ **Step 3: apply the signed gradient.** $$ \theta_1=1-0.05(2)=0.9 $$ **Iteration 2 — step 1: retain s₁ and add the new squared gradient.** $$ s_2=4+1^2=5 $$ **Step 2: calculate the new coefficient.** $$ \eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360 $$ **Step 3: update from θ₁.** $$ \theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640 $$ **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5. ### Advantage, limitation and epsilon convention Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes. ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical. --- ## TEE · RMSProp remembers recent squared gradients Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s13 **Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator. ### Recurrence and interpretation Using decay β and zero initial state: $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update. If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever. ![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg) Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient. Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β = 0.9. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step **Iteration 1 — step 1: calculate the squared-gradient average.** $$ s_1=0.9(0)+0.1(2^2)=0.4 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_1}=\sqrt{0.4}\approx0.632455532 $$ $$ \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766 $$ **Step 3: update the parameter.** $$ \theta_1=1-0.316227766\approx0.683772234 $$ **Iteration 2 — step 1: use s₁, not the original zero state.** $$ s_2=0.9(0.4)+0.1(1^2)=0.46 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_2}=\sqrt{0.46}\approx0.678232998 $$ $$ \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956 $$ **Step 3: update from θ₁, retaining calculator precision.** $$ \theta_2\approx0.683772234-0.147441956\approx0.536330278 $$ **Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46. ### Comparison with AdaGrad and Adam AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum. --- ## TEE · Adam remembers both direction and squared size Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s14 **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. --- ## TEE · Adam bias correction: why the denominator is one minus beta to a power Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s15 **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate. ### Explanation and working ### 1. What the moving average actually contains Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization. ![The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam_correction.svg) Diagram: The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.. Labels: Correct the early zero-initialized memory | Correct the early zero-initialized memory | Example: β₁ = 0.9, first gradient = 2, initial m = 0. | Raw memory | m₁ = 0.9 × 0 + 0.1 × | 2 = 0.2 | Correction factor | 1 − 0.9¹ = 0.1 | Corrected memory | m̂₁ = 0.2 ÷ 0.1 = 2 | The small raw value is partly caused by the starting zero. Bias correction | removes that effect. The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor. Tap the diagram to enlarge. $$ m_1=(1-\beta)g_1 $$ At the second step, substitute the first expression into the recurrence: $$ m_2=\beta(1-\beta)g_1+(1-\beta)g_2 $$ ### 2. Add the coefficients After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term: $$ \beta(1-\beta)+(1-\beta)=(1-\beta)(1+\beta)=1-\beta^2 $$ After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ The squared-gradient memory uses its own decay β₂ and the same reasoning: $$ \widehat s_t=\frac{s_t}{1-\beta_2^t} $$ ### 3. A constant-gradient example makes the purpose visible Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38. $$ \widehat m_1=\frac{0.2}{1-0.9}=2 $$ $$ \widehat m_2=\frac{0.38}{1-0.9^2}=\frac{0.38}{0.19}=2 $$ Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient. ### 4. Two betas mean two separate powers At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators. ### 5. Correct first, then combine The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows. This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude. --- ## TEE · Model answer: AdaGrad, RMSProp and Adam Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s16 ## Question **Question.** Compare AdaGrad, RMSProp and Adam. Explain what each method stores and how that memory changes a parameter update. ## Answer — step by step AdaGrad, RMSProp and Adam are adaptive gradient-based optimizers. Each stores separate state for each trainable parameter and uses that state to scale its update. Let gₜ be the current gradient for parameter θ, η the base learning rate and ε a small positive stabilizer. The equations below put ε outside the square root. ### AdaGrad: cumulative squared-gradient history $$ s_t=s_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Starting at s₀ = 0, every squared gradient contributes permanently. Parameters with frequent large gradients acquire smaller effective learning rates. This can be useful for sparse features, but the growing denominator can make later updates too small. The base learning rate still has to be chosen. ### RMSProp: a decaying squared-gradient average $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Old contributions decay by repeated multiplication by β. The state can decrease when recent squared gradients are small, avoiding AdaGrad's permanent accumulation. Basic RMSProp has one squared-gradient state and no separate signed-gradient average. ### Adam: first and second moments with bias correction Initialize m₀ = s₀ = 0 and count the first update as t = 1: $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad\hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The first moment stores signed directional information; the second stores squared-gradient scale. Correction compensates for their zero initialization. The corrected first moment is already the numerator's gradient estimate, so the update is not multiplied by gₜ again. Raw moments are carried into the next iteration. | Aspect | AdaGrad | RMSProp | Adam | | --- | --- | --- | --- | | Squared-gradient history | Cumulative sum | Decaying average | Decaying average | | Separate signed-gradient average | No | No in the basic method | Yes | | Standard initialization correction | No | No in the basic method | Corrects both moments | | State values per parameter | One | One | Two | | Main concern | Effective rate can become very small | Learning rate and decay need tuning | More state; hyperparameters still matter | All three adapt updates coordinate by coordinate. None removes the need for validation or guarantees a global minimum for a general neural-network objective. --- ## TEE · One comparison to rebuild all optimizer formulas Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s17/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s17 | Method | What is remembered | What changes the step | | --- | --- | --- | | GD or SGD | No optimizer memory in the basic rule | Current gradient × η | | Momentum | Previous movement | Old velocity plus current correction | | Nesterov | Previous movement | Gradient evaluated at lookahead | | AdaGrad | Sum of all squared gradients | Divide current gradient by accumulated root | | RMSProp | Moving average of squared gradients | Divide current gradient by recent root | | Adam | Moving averages of gradients and squares | Divide corrected mean by corrected root | **Working order.** Start each method on a separate line with its states initialized. The solved examples use the same supplied gradients to show how their stored states differ. They do not have to give the same parameter value. The prescribed gradient sequence is practice data; if a question instead gives J(θ), recalculate its gradient at the appropriate current or lookahead parameter. ![Compare the memory rules first. Then use the matching memory in the denominator of the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad_rms.svg) Diagram: Compare the memory rules first. Then use the matching memory in the denominator of the update.. Labels: What happens to an old squared gradient? | What happens to an old squared gradient? | Start with old memory 4. The current gradient is −3, so its square is 9. | AdaGrad: keep adding | Old sum G + new g² | 4 + 9 = 13 | An old contribution stays. | RMSProp: decay and add | ρs + (1 − ρ)g² | 0.9 × 4 + 0.1 × 9 = 4.5 | Old contributions fade. | Both divide the signed current gradient by a square-root scale plus epsilon. Compare the memory rules first. Then use the matching memory in the denominator of the update. Tap the diagram to enlarge. --- ## TEE · AdaGrad and RMSProp with a negative gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c5-s18/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/5/tee-c5-s18 The gradient has a sign, but its square is nonnegative. Both methods store squared gradients to control update size. The signed gradient is still used separately in the update numerator to preserve direction. ![The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad_rms.svg) Diagram: The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory.. Labels: What happens to an old squared gradient? | What happens to an old squared gradient? | Start with old memory 4. The current gradient is −3, so its square is 9. | AdaGrad: keep adding | Old sum G + new g² | 4 + 9 = 13 | An old contribution stays. | RMSProp: decay and add | ρs + (1 − ρ)g² | 0.9 × 4 + 0.1 × 9 = 4.5 | Old contributions fade. | Both divide the signed current gradient by a square-root scale plus epsilon. The gradient sign remains in the numerator. Its square contributes a nonnegative value to the memory. Tap the diagram to enlarge. For an old stored state of 4 and a new gradient of −3, first square the gradient: (−3) × (−3) = 9. **AdaGrad:** add the new square to the entire old sum: 4 + 9 = 13. **RMSProp with decay 0.9:** keep 0.9 × 4 = 3.6 of the old state; add (1 − 0.9) × 9 = 0.1 × 9 = 0.9 of the new square. The new state is 3.6 + 0.9 = 4.5. AdaGrad retains all earlier squares in a growing sum. RMSProp gradually discounts earlier squares. This difference explains why their effective step sizes change differently during training. --- ## TEE · Recognizing overfitting and underfitting Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s0 **Definitions.** Generalization is a model's ability to perform on unseen examples from the relevant task distribution. Underfitting occurs when the model fails to capture the task's important relationships. Overfitting occurs when fitting the training data produces substantially poorer generalization, often by modeling noise or incidental details. ### Interpreting learning curves Underfitting commonly gives high training and validation errors. Overfitting commonly gives low training error with a substantially higher validation error, or validation loss that rises while training loss continues to fall. These patterns are diagnostic clues; distribution shift, leakage and an unrepresentative split can also affect the curves. ![Compare training and validation behavior when selecting model capacity or a stopping point.](https://deep-learning-study-room.vercel.app/assets/figures/overfitting.svg) Diagram: Compare training and validation behavior when selecting model capacity or a stopping point.. Labels: Loss | Epochs | Validation | Training | Best validation region | Save the best validation checkpoint; do not choose by test loss. Compare training and validation behavior when selecting model capacity or a stopping point. Tap the diagram to enlarge. | Observation | Plausible explanation | Appropriate response | | --- | --- | --- | | Both errors remain high | Insufficient capacity, weak features or inadequate optimization | Improve representation, train appropriately, reconsider architecture or reduce excessive regularization | | Training error low; validation error high | Overfitting or a distribution/split problem | Inspect the split; use suitable regularization, more representative data or early stopping | | Both losses fluctuate or diverge | Optimization instability | Examine learning rate, data scaling and gradient magnitude | ### Bias–variance perspective Insufficient model flexibility can create high approximation bias. A highly flexible model can be sensitive to the particular training sample, increasing variance. Regularization trades some training fit for constraints that can improve generalization. A more complex network is not automatically more accurate on unseen data. The validation set is used to choose hyperparameters and checkpoints. The test set should remain outside these choices. Repeatedly selecting models from test results turns the test data into part of the selection process and weakens its role as an unbiased final evaluation. --- ## TEE · L1 and L2 add a cost for large weights Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s1 **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. --- ## TEE · Model answer: regularization and the L1–L2 difference Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s2 ## Question **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength. ## Answer — step by step Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost. ![Follow the stated penalty convention, including lambda and any half factor, when you form the objective.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: Follow the stated penalty convention, including lambda and any half factor, when you form the objective.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge. ### Objectives used in this course $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question. | Basis | L1 | L2 | | --- | --- | --- | | Quantity penalized | Absolute weight magnitude | Squared weight magnitude | | Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention | | Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes | | Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights | The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total. Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective. --- ## TEE · Dropout removes activations during training Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s3 **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training. ### Explanation and working Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count. ![Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout.svg) Diagram: Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.. Labels: Dropout temporarily removes selected activations | Dropout temporarily removes selected activations | Example: keep probability q = 0.5. Inverted dropout during training. | Original | 0.8 | -0.4 | 0.6 | 0.2 | Mask | 1 | 0 | 1 | 0 | After mask ÷ q | 1.6 | 0 | 1.2 | 0 | The kept values double: 0.8 ÷ 0.5 = 1.6 and 0.6 ÷ 0.5 = 1.2. At inference, no | mask is used. Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge. Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions. $$ a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout} $$ $$ a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout} $$ **Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs. ### Given Activations = [0.8,−0.4,0.6,0.2]. Dropout rate = 50%, so keep probability q = 0.5. Chosen mask = [1,0,1,0]. ### To find Find training outputs under inverted dropout. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step **Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key. **Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer. Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition. --- ## TEE · Early stopping uses the best validation point Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s4 **Definition.** Early stopping is a regularization and model-selection procedure that terminates training when a monitored validation criterion no longer improves according to a specified rule, typically retaining the best checkpoint. ### Explanation and working Monitor validation loss after epochs. When it stops improving for a chosen patience interval, stop training and restore the best validation checkpoint. Training loss can fluctuate; it does not always decrease monotonically. Never select the stopping epoch by repeatedly consulting the test set. ![Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. Training loss keeps falling while validation loss starts rising. Keep the checkpoint from the validation minimum. Tap the diagram to enlarge. **Question.** Validation losses after epochs 1–5 are [0.8,0.6,0.5,0.53,0.55]. Patience is 2 with no required minimum improvement. When do you stop and what do you restore? ### Given Validation losses at epochs 1–5 = [0.8,0.6,0.5,0.53,0.55]. Patience = 2; no required minimum improvement. ### To find Find the stop epoch and the best checkpoint to restore. ### Formula — which one and why **Early stopping rule.** Compare each validation loss with the best one so far. A new best resets the waiting count; a non-improvement increases it. Stop when the count reaches patience and restore the best checkpoint. Patience = allowed consecutive epochs without improvement. With minimum improvement zero, a strictly smaller validation loss counts as improvement. **Yaad rakhne ka cue:** Best validation save; patience khatam toh stop aur best restore. ### Solution — step by step **Step 1: process epoch 1.** Loss 0.8 establishes the first best checkpoint; wait = 0. **Step 2: process epoch 2.** 0.6 < 0.8: save epoch 2; wait = 0. **Step 3: process epoch 3.** 0.5 < 0.6: save epoch 3; wait = 0. **Step 4: process epoch 4.** 0.53 > 0.5: no improvement; wait = 1. **Step 5: process epoch 5.** 0.55 > 0.5: no improvement; wait = 2, equal to patience. Stop. **Step 6: restore the best checkpoint.** The smallest observed validation loss is 0.5 at epoch 3. Restore its weights. **Final answer:** stop after epoch 5; restore epoch 3. These are different epochs. ### Early stopping with [0.8, 0.7, 0.72, 0.68, 0.69, 0.7] **Question. Validation losses are [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.** ### Given Validation losses in epoch order = [0.8, 0.7, 0.72, 0.68, 0.69, 0.7]. Patience = 2; smaller loss is better; ties are not improvements. ### To find The stopping epoch and best checkpoint. ### Formula — which one and why Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs. Patience counts consecutive non-improvements. A new best resets the count. **Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore. ### Solution — step by step **Step 1: process epoch 1.** Validation loss = 0.8; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.8. **Step 2: process epoch 2.** Validation loss = 0.7; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 2, loss 0.7. **Step 3: process epoch 3.** Validation loss = 0.72; no strict improvement: wait count becomes 1. Best checkpoint is epoch 2, loss 0.7. **Step 4: process epoch 4.** Validation loss = 0.68; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 4, loss 0.68. **Step 5: process epoch 5.** Validation loss = 0.69; no strict improvement: wait count becomes 1. Best checkpoint is epoch 4, loss 0.68. **Step 6: process epoch 6.** Validation loss = 0.7; no strict improvement: wait count becomes 2. Best checkpoint is epoch 4, loss 0.68. **Final answer:** Stop after epoch 6; restore epoch 4, with validation loss 0.68. ### Early stopping with [0.6, 0.6, 0.61, 0.62] **Question. Validation losses are [0.6, 0.6, 0.61, 0.62]. With patience 2 and strict improvement, identify the stopping epoch and checkpoint to restore.** ### Given Validation losses in epoch order = [0.6, 0.6, 0.61, 0.62]. Patience = 2; smaller loss is better; ties are not improvements. ### To find The stopping epoch and best checkpoint. ### Formula — which one and why Save an epoch when its validation loss is strictly lower than the best so far. Stop after the stated number of consecutive non-improving epochs. Patience counts consecutive non-improvements. A new best resets the count. **Yaad rakhne ka cue:** Naya best mila toh save aur wait zero; patience khatam toh best restore. ### Solution — step by step **Step 1: process epoch 1.** Validation loss = 0.6; strict improvement: save this checkpoint and reset wait count to 0. Best checkpoint is epoch 1, loss 0.6. **Step 2: process epoch 2.** Validation loss = 0.6; no strict improvement: wait count becomes 1. Best checkpoint is epoch 1, loss 0.6. **Step 3: process epoch 3.** Validation loss = 0.61; no strict improvement: wait count becomes 2. Best checkpoint is epoch 1, loss 0.6. **Final answer:** Stop after epoch 3; restore epoch 1, with validation loss 0.6. --- ## TEE · Model answer: generalization and early stopping Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s5 ## Question **Question.** Distinguish underfitting and overfitting. Explain how validation data and early stopping help control overfitting. ## Answer — step by step Underfitting occurs when a model has not captured enough useful structure in the training data. Both training and validation errors are usually high relative to a sensible task baseline. Causes can include insufficient capacity, inadequate features, excessive regularization or incomplete optimization. ![The best training score is not necessarily the best score on unseen validation examples.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: The best training score is not necessarily the best score on unseen validation examples.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. The best training score is not necessarily the best score on unseen validation examples. Tap the diagram to enlarge. Overfitting occurs when a model fits training-specific details or noise that do not generalize. Training error is low while validation error is substantially higher. A mismatch between training and validation data can produce a similar gap, so the split must also be examined. ### How early stopping works 1. Train using the training set; do not learn parameters from the validation targets as training samples. 2. After an epoch or another chosen interval, evaluate validation loss. 3. Save the parameters whenever validation loss reaches a new best value. 4. Count consecutive intervals with no qualifying improvement. 5. When the count reaches the chosen patience, stop and restore the saved best parameters. For validation losses 0.8, 0.6, 0.5, 0.53 and 0.55, epoch 3 is best. With patience 2, epochs 4 and 5 supply two non-improvements: stop after epoch 5 and restore epoch 3. The stopping epoch and the retained model epoch are different. Other remedies for overfitting include appropriate L1/L2 penalties, dropout, augmentation and more representative training data. Underfitting may instead require a better model, adequate training or less regularization. Use the test set for final evaluation after those choices, not for repeatedly choosing the stopping point. --- ## TEE · Choosing a remedy Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s6 For overfitting: try L1/L2, dropout, suitable data augmentation, a smaller model, more representative training data and early stopping. For underfitting: improve features or architecture, train adequately, check optimization, or reduce excessive regularization. Evaluate changes on validation data. ![Read the training and validation pattern before selecting a remedy.](https://deep-learning-study-room.vercel.app/assets/figures/v_remedies.svg) Diagram: Read the training and validation pattern before selecting a remedy.. Labels: Match the remedy to the observed problem | Match the remedy to the observed problem | Compare training and validation behavior before changing the model. | Both losses high: underfitting | Model may be too simple. | Training may be insufficient. | Check optimization and features. | Train low, validation high | Possible overfitting. | Use regularization or more data. | Consider early stopping. | A remedy is a choice based on the pattern; no single setting fixes every problem. Read the training and validation pattern before selecting a remedy. Tap the diagram to enlarge. Data augmentation makes label-preserving variations of training examples, such as suitable crops or brightness changes in images. It must respect the task: flipping a digit or a medical image may change meaning. Augmentation happens within the training split; do not leak augmented copies into validation. | Change | Usual bias tendency | Usual variance tendency | | --- | --- | --- | | Stronger weight regularization | Increase | Decrease | | More hidden units with other factors fixed | Decrease | Increase | | Dropout | Increase | Decrease | | More representative data from same distribution | Roughly unchanged model bias | Decrease | These are classical tendencies for exam discussion, not exact laws for every deep-network training run. Distinguish model capacity from optimization failure. --- ## TEE · Inverted dropout during training and inference Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s7 **Definition.** Inverted dropout divides retained training activations by their keep probability so that their expectation equals the original activation. At inference, dropout is disabled and no compensating activation scaling is required. ### Explanation and working Inverted dropout first chooses which activations to keep, then divides each kept activation by the keep probability. This compensates for the missing activations in expectation. A dropped activation becomes zero. ![Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. Follow both possible training outcomes for activation 3. Their average remains 3; inference uses that ordinary activation. Tap the diagram to enlarge. For activation 3 and keep probability q = 0.5, a kept value becomes 3 ÷ 0.5 = 6. A dropped value becomes 0. Since half the masks keep it, the average is 0.5 × 6 + 0.5 × 0 = 3, equal to the original activation. At inference, dropout is disabled and the ordinary activation 3 is used. Under this inverted-dropout convention, no extra multiplication by q is required at inference. ### Expectation calculation For a fixed activation a and a Bernoulli keep mask m with P(m = 1) = q: $$ \mathbb E\left[\frac{ma}{q}\right]=\frac aq\mathbb E[m]=\frac aq\,q=a $$ The equality concerns the activation's expectation over masks. It does not say every sampled layer preserves its sum, nor does it generally make a nonlinear network's complete stochastic prediction equal to its deterministic inference prediction. --- ## TEE · Model answer: dropout with training and inference explained Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c6-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/6/tee-c6-s8 ## Question **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling. ## Answer — step by step Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights. ![The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge. Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop. ### Inverted dropout during training $$ a'=\frac{ma}{q} $$ A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks. For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is: $$ 0.5(6)+0.5(0)=3 $$ ### Inference Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions. Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data. --- ## TEE · Reading image shapes before doing CNN arithmetic Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s0 **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension. ### Explanation and working ### Height, width and channels The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors. ![Count height, width and channels separately. Three channels are stacked at the same spatial positions.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_shape.svg) Diagram: Count height, width and channels separately. Three channels are stacked at the same spatial positions.. Labels: Read height, width and channels separately | Read height, width and channels separately | The text uses 4 × 5 × 3: four rows, five columns and three channels. | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 3 stacked channels | H × W × C | H = 4 rows | W = 5 columns | C = 3 channels | Total entries = 60 | A filter spans the incoming channels. Its spatial width is a different quantity. Count height, width and channels separately. Three channels are stacked at the same spatial positions. Tap the diagram to enlarge. ### One position versus the whole map A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels. ### Why filter depth matches input channels A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels. ### Reading a complete shape calculation The spatial-size formula computes height and width. It does not compute the channel count. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position. ### Shape, parameters and computation are three different counts **Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights. In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count. --- ## TEE · A computer sees an image as numbers Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s1 **Definition.** A digital image is a spatially arranged numerical tensor. In a CNN, convolutions transform input channels into learned feature maps using local connectivity and spatially shared kernel parameters. ### Explanation and working A pixel is one position in an image. A grayscale image stores one intensity per position. A colour RGB image usually stores three numbers per position: red, green and blue. Therefore 32 × 32 × 3 means height 32, width 32 and 3 input channels. It contains 32 × 32 × 3 = 3,072 numbers, not 32 neurons. ![Each visible pixel corresponds to a matrix entry. Brightness is a number the network can calculate with.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_grid.svg) Diagram: Each visible pixel corresponds to a matrix entry. Brightness is a number the network can calculate with.. Labels: Pixels are numerical entries | Pixels are numerical entries | A small grayscale image can be stored as a matrix of brightness values. | 0 | 0 | 0 | 0 | 0 | 180 | 180 | 0 | 0 | 180 | 180 | 0 | 0 | 0 | 0 | 0 | Image | 4 rows × 4 columns | An RGB image has three such channels. A pixel location has a red, green and blue | value. Each visible pixel corresponds to a matrix entry. Brightness is a number the network can calculate with. Tap the diagram to enlarge. A **feature** is a useful pattern, such as an edge or texture. A **feature map** is a grid of responses that says where a learned filter matches the input. Early CNN layers often detect local simple patterns; later layers combine them into parts and more complex patterns. A filter, also called a kernel in many exam questions, is a small set of learned weights. Slide it across the image, multiply corresponding numbers, and add. The same weights are reused at every spatial position. This is **weight sharing**. --- ## TEE · The convolution calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s2 **Definition.** A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention. ### Explanation and working In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it. **Question.** Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias. ### Given Input X = $\begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}$ . Kernel K = $\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$ . Stride = 1; padding = 0; bias = 0; no kernel flip. ### To find Find the 2 × 2 output map. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_cd39cabe48.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 0 | 0 | −1 | Matrix 2 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 Given matrices for this question Tap the diagram to enlarge. $$ X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix} $$ **Step 1. Top-left window.** 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4. **Step 2. Move right by one.** 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4. **Step 3. Move down and return left.** 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4. **Step 4. Bottom-right window.** 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is $\begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}$ . If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested. ![Four windows produce four output cells](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: Four windows produce four output cells. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. Four windows produce four output cells Tap the diagram to enlarge. --- ## TEE · Stride padding and floor Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s3 **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output. ### Explanation and working Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension. ![The border adds to both sides of the input. The stride changes the next starting position of the filter window.](https://deep-learning-study-room.vercel.app/assets/figures/v_stride_padding.svg) Diagram: The border adds to both sides of the input. The stride changes the next starting position of the filter window.. Labels: Padding adds a border. Stride moves the window. | Padding adds a border. Stride moves the window. | Example: a 3 × 3 input becomes 5 × 5 after one zero border. | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 3 | 0 | 0 | 4 | 5 | 6 | 0 | 0 | 7 | 8 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | Padding P = 1 | 0 | 1 | 2 | 3 | 4 | Stride S = 2 | Starting positions: 0, 2, … | For each spatial dimension: add 2P, subtract K, divide by S, take floor, then add | 1. The border adds to both sides of the input. The stride changes the next starting position of the filter window. Tap the diagram to enlarge. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: $\left\lfloor 2.5\right\rfloor$ = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels. **Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1. ### Output dimension with a fractional intermediate result **Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.** ### Given N = 6; K = 3; S = 2; P = 0; dilation = 1. ### To find The integer output side length. ### Formula — which one and why The formula counts complete kernel placements. Flooring removes an incomplete final placement. N is input side; P is padding on each side; K is kernel side; S is stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1. ### Solution — step by step **Step 1: include padding and subtract kernel size.** $$ 6+2(0)-3=3 $$ **Step 2: divide by the stride.** $$ \frac32=1.5 $$ **Step 3: take the floor.** $$ \lfloor1.5\rfloor=1 $$ **Step 4: count the first placement.** $$ N_{\mathrm{out}}=1+1=2 $$ **Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter. ### Padding on both sides **Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.** ### Given N = 5; K = 3; S = 1; P = 1 on each side. ### To find Padded input and output dimensions. ### Formula — which one and why Padding adds P entries on each of two opposite sides; the output counts complete kernel placements. N = unpadded side; K = kernel side; S = stride; P = per-side padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\lfloor(N+2P-K)/S\rfloor+1 $$ **Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain. ### Solution — step by step **Step 1: count both padding edges.** $$ 2P=2(1)=2 $$ **Step 2: calculate the padded input side.** $$ N+2P=5+2=7 $$ **Step 3: subtract kernel width.** $$ 7-3=4 $$ **Step 4: divide by stride and take the floor.** $$ \lfloor4/1\rfloor=4 $$ **Step 5: include the first placement.** $$ N_{out}=4+1=5 $$ **Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter. “Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case. --- ## TEE · Convolution over several channels Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s4 **Definition.** A standard convolutional filter spans every input channel. At one spatial position, channel-wise weighted sums are accumulated and one filter bias is added, producing one scalar output in that filter's feature map. ### Explanation and working A 3 × 3 filter receiving RGB input has depth 3, so it contains 3 × 3 × 3 weights. At one position, compute a weighted sum in each of the three channels, add those sums, and then add the filter’s one bias. One filter produces one output channel. Sixteen filters produce sixteen output channels. ![Add the channel contributions into one output cell. The filter bias is added once after those contributions.](https://deep-learning-study-room.vercel.app/assets/figures/v_channels.svg) Diagram: Add the channel contributions into one output cell. The filter bias is added once after those contributions.. Labels: One filter combines all incoming channels | One filter combines all incoming channels | The three channel contributions join into one cell, not three separate maps. | Red channel | Weighted sum = 2 | Green channel | Weighted sum = −1 | Blue channel | Weighted sum = 4 | Add contributions and one bias | 2 + (−1) + 4 + 0.5 = 5.5 | The filter spans all three channels. Add its bias once after summing the channel | contributions. Add the channel contributions into one output cell. The filter bias is added once after those contributions. Tap the diagram to enlarge. **Tiny example.** At one position the red, green and blue channel sums are 2, −1 and 4. With bias 0.5, the output is 2 − 1 + 4 + 0.5 = 5.5. Do not keep three separate outputs for this one filter; they have been added. --- ## TEE · One filter over multiple channels: all contributions go into one cell Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s5 **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels. ### Explanation and working ### 1. Separate spatial size from depth Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3. ![Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.](https://deep-learning-study-room.vercel.app/assets/figures/v_channel_products.svg) Diagram: Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.. Labels: A 1 × 1 filter still mixes all three channels | A 1 × 1 filter still mixes all three channels | Use input [2, 5, 1], weights [0.1, −0.2, 0.5] and one bias 0.3. | Channel 1 | 2 × 0.1 = 0.2 | Channel 2 | 5 × (−0.2) = −1 | Channel 3 | 1 × 0.5 = 0.5 | One filter → one output value | 0.2 + (−1) + 0.5 + 0.3 = 0 | One spatial position can contain several channel values. The one-by-one filter | mixes them. Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example. Tap the diagram to enlarge. $$ z=0.1(2)+(-0.2)(5)+0.5(1)+0.3 $$ $$ z=0.2-1+0.5+0.3=0 $$ The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels. ### 2. A larger filter has one spatial grid per input channel A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias. The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified. ### 3. What changes when there are several filters? Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth. For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters. ### 4. One-by-one does not mean one weight A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed. This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately. --- ## TEE · Choosing more or fewer filters Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s6 **Definition.** The number of convolutional filters determines the number of output channels. Kernel height, kernel width, stride and padding determine spatial processing; input-channel count determines each filter's depth. ### Explanation and working Each filter produces one output channel and can learn a different pattern. More filters increase representational capacity, output depth, parameter count and activation memory. They may improve performance but can overfit small datasets or exceed a compute budget. Choose counts using validation results and resource constraints. More filters in one layer increase its width, not the number of network layers. ![Each filter has the full incoming depth. Each produces one map, so the number of filters sets the output depth.](https://deep-learning-study-room.vercel.app/assets/figures/v_filters.svg) Diagram: Each filter has the full incoming depth. Each produces one map, so the number of filters sets the output depth.. Labels: Each filter produces one output feature map | Each filter produces one output feature map | Three incoming channels; two filters; two output maps. | Input | H × W × 3 | Filter A | K × K × 3 | Filter B | K × K × 3 | Map A | Hout × Wout | Map B | Hout × Wout | Two filters mean output depth 2. Filter depth matches the incoming depth 3. Each filter has the full incoming depth. Each produces one map, so the number of filters sets the output depth. Tap the diagram to enlarge. --- ## TEE · Pooling needs comparisons or averages Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s7 **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases. ### Explanation and working Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged. **Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding: ### Given The 4 × 4 input map is printed below. Window = 2 × 2; stride = 2; padding = 0. ### To find Find the pooled output by the operation(s) requested. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step $$ A=\begin{bmatrix}21&11&15&10\\12&9&10&11\\11&13&18&12\\7&4&15&12\end{bmatrix} $$ ### Top-left window **Step 1: select the four entries.** [21, 11, 12, 9] **Step 2: choose the largest entry.** $$ y_{\max}=21 $$ **Step 3: add all entries.** $$ 21+11+12+9=53 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{53}4=13.25 $$ ### Top-right window **Step 1: select the four entries.** [15, 10, 10, 11] **Step 2: choose the largest entry.** $$ y_{\max}=15 $$ **Step 3: add all entries.** $$ 15+10+10+11=46 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{46}4=11.5 $$ ### Bottom-left window **Step 1: select the four entries.** [11, 13, 7, 4] **Step 2: choose the largest entry.** $$ y_{\max}=13 $$ **Step 3: add all entries.** $$ 11+13+7+4=35 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{35}4=8.75 $$ ### Bottom-right window **Step 1: select the four entries.** [18, 12, 15, 12] **Step 2: choose the largest entry.** $$ y_{\max}=18 $$ **Step 3: add all entries.** $$ 18+12+15+12=57 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{57}4=14.25 $$ Outputs are max $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ and average $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . ![Each pooling window reduces several values to one](https://deep-learning-study-room.vercel.app/assets/figures/pooling.svg) Diagram: Each pooling window reduces several values to one. Labels: INPUT 4 × 4 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 | MAX POOL | 21 | 15 | 13 | 18 | AVERAGE POOL | 13.25 | 11.5 | 8.75 | 14.25 | 2 × 2 windows | stride 2 Each pooling window reduces several values to one Tap the diagram to enlarge. Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance. --- ## TEE · Parameters are different from activation values Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s8 **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters. ### Explanation and working An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights. ![The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations. Tap the diagram to enlarge. $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation. ### One filter, then all filters **Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.** ### Given Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter. ### To find Weights, biases, total trainable parameters and output channels. ### Formula — which one and why Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights. Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters. $$ P=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply. ### Solution — step by step **Step 1: count weights in one filter.** $$ 3(3)(3)=27 $$ **Step 2: include its one bias.** $$ 27+1=28 $$ **Step 3: count all weights.** $$ 27(16)=432 $$ **Step 4: count all biases.** $$ 1(16)=16 $$ **Step 5: add and state channel count.** $$ P=432+16=448 $$ Each filter produces one channel, so output channels = 16. **Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels. $$ P_{\mathrm{pool}}=0,\quad P_{\mathrm{flatten}}=0,\quad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values. --- ## TEE · One complete CNN pipeline Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s9 **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron. ### Given Input = 128 × 128 × 3. Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2. Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2. Flatten → dense 240 → dense 120 → dense 5. One bias per learned filter/dense neuron. ### To find Find every output shape and layer parameter count, then the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step **Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448. ![Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.](https://deep-learning-study-room.vercel.app/assets/figures/v_cnn_pipeline.svg) Diagram: Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.. Labels: Follow the complete CNN question, one layer at a time | Follow the complete CNN question, one layer at a time | Start with 128 × 128 × 3. Shape and learned-parameter count are separate. | LAYER | OUTPUT SHAPE / LENGTH | PARAMETERS | 1. Convolution 1 | 126 × 126 × 16 | 448 | 2. Pooling 1 | 63 × 63 × 16 | 0 | 3. Convolution 2 | 61 × 61 × 32 | 4,640 | 4. Pooling 2 | 30 × 30 × 32 | 0 | 5. Flatten | 28,800 | 0 | 6. Dense 240 | 240 | 6,912,240 | 7. Dense 120 | 120 | 28,920 | 8. Classifier | 5 | 605 | Total learned parameters = 6,946,853 | Each output is the next layer’s input. Pooling and flattening add no learned | parameters. Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count. Tap the diagram to enlarge. **Step 2. Pool 1 size.** $\left\lfloor\frac{126 - 2}{2}\right\rfloor+1$ = 63; output 63 × 63 × 16. Parameters = 0. **Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640. **Step 4. Pool 2 size.** $\left\lfloor\frac{61 - 2}{2}\right\rfloor+1$ = $\left\lfloor 29.5\right\rfloor$ + 1 = 30; output 30 × 30 × 32. Parameters = 0. **Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0. **Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240. **Step 7. Dense 120.** (240 + 1) × 120 = 28,920. **Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer. **Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer. --- ## TEE · CNN versus a fully connected image network Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s10 **Definition.** A convolutional neural network uses local receptive fields and spatial parameter sharing, unlike a dense layer's independent connection weights. These architectural constraints exploit spatial structure and can greatly reduce parameter counts. ### Explanation and working A fully connected layer can learn image features, but flattening pixels and connecting every pixel to every neuron ignores the useful local structure built into a CNN and can require many parameters. CNNs exploit local connections and shared filters. They learn hierarchical features and reuse a pattern detector at many locations. ![Compare where connections go and which weights are reused.](https://deep-learning-study-room.vercel.app/assets/figures/v_fc_cnn.svg) Diagram: Compare where connections go and which weights are reused.. Labels: Fully connected and convolutional layers connect differently | Fully connected and convolutional layers connect differently | Both learn weights; the connection pattern changes. | Fully connected image layer | Every output sees every input. | Each connection has a weight. | Spatial layout is not built in. | Convolutional layer | A filter sees a local patch. | Reuse it at many positions. | Preserve a spatial feature map. | Parameter sharing makes a convolution economical when the same feature can occur | in many places. Compare where connections go and which weights are reused. Tap the diagram to enlarge. For a 32 × 32 × 3 input, a dense layer with 100 neurons has (3,072 + 1) × 100 = 307,300 parameters. A convolution with 100 filters of size 3 × 3 has (27 + 1) × 100 = 2,800 parameters. The outputs are different shapes, so this is a parameter-efficiency illustration, not two identical computational tasks. --- ## TEE · Model answer: CNN structure and parameter sharing Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s11 ## Question **Question.** Explain the main layers of a convolutional neural network and why CNNs are suitable for image data. Compare them with a fully connected image network. ## Answer — step by step A CNN processes spatially arranged data using filters that examine local regions. The same filter weights are reused at different positions. This combination of local connectivity and weight sharing builds an image-specific structure into the model and can greatly reduce the number of learned weights. ![The highlighted patch uses the same filter that will be reused at the next position.](https://deep-learning-study-room.vercel.app/assets/figures/v_weight_sharing.svg) Diagram: The highlighted patch uses the same filter that will be reused at the next position.. Labels: Slide the same filter; reuse its weights | Slide the same filter; reuse its weights | A CNN does not learn a fresh filter for every image location. | 1 | 2 | 0 | 1 | 0 | 1 | 3 | 2 | 2 | 0 | 1 | 1 | 1 | 2 | 0 | 3 | 1 | 0 | 0 | -1 | Same four weights | 0 | -1 | -2 | 0 | 0 | 2 | 0 | 0 | -2 | Output map | Move the window to calculate the next output cell. | Local connections restrict the visible patch. Weight sharing reuses the learned | pattern. The highlighted patch uses the same filter that will be reused at the next position. Tap the diagram to enlarge. ### Typical layer sequence **Input → convolution → nonlinear activation → optional pooling → further feature extraction → classifier.** | Layer | Operation | Learned parameters? | | --- | --- | --- | | Convolution | Multiply a local window by a filter, sum and add its bias | Filter weights and usually one bias per filter | | ReLU | Replace negative values by zero | No ordinary ReLU parameters | | Pooling | Summarize each window by maximum or average | None for ordinary max/average pooling | | Flatten | Rearrange H × W × C values into a vector | None | | Dense layer | Form weighted combinations of all incoming values | Weights and receiving-neuron biases | | Softmax | Convert class scores into probabilities | No additional weights of its own | Early filters can learn local patterns such as edges. Later layers combine features over larger receptive fields. A filter responding at several image positions uses one shared set of weights, not a new set at every location. Flattening an image and using a dense layer does not enforce this local, shared structure. A dense layer has a separate connection weight for each input–output pair. CNNs can therefore be more parameter-efficient for visual patterns, although they still need suitable data and training. Pooling can provide some robustness to small shifts; it does not guarantee complete invariance to every translation. --- ## TEE · Output size and parameter count are different Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s12 **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters. ### Explanation and working **Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias. ### Given Input = 7 × 7 × 3; two 3 × 3 filters. Stride = 2; padding = 1; include biases. ### To find Find the output shape and trainable parameter count. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step **Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2. ![Output size counts produced values; parameter count counts adjustable weights and biases.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Output size counts produced values; parameter count counts adjustable weights and biases.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Output size counts produced values; parameter count counts adjustable weights and biases. Tap the diagram to enlarge. **Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters. --- ## TEE · Model answer: shape, activations and parameters Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s13 ## Question **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them. ## Answer — step by step Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions. ![Keep output shape, activation count and parameter count as three separate quantities.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Keep output shape, activation count and parameter count as three separate quantities.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Keep output shape, activation count and parameter count as three separate quantities. Tap the diagram to enlarge. ### Spatial output shape For an ordinary convolution with dilation 1: $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters. ### Output activations $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ This is the number of output values for one example, excluding any batch dimension. ### Trainable parameters $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions. ### Small example A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels. --- ## TEE · Faculty example: a complete binary-image convolution Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s14 **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)** ### Given Image X = $\begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}$ . Filter K = $\begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}$ . These are the values in the Unit 4 lecture example. ### To find The output shape and all nine feature-map values. ### Formula — which one and why The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window. ![Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.](https://deep-learning-study-room.vercel.app/assets/figures/u4_binary_windows.svg) Diagram: Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.. Labels: u4 binary windows | One filter, nine window positions | Input: 5 × 5 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | Shared filter: 3 × 3 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | → | Output: 3 × 3 | 4 | 3 | 4 | 2 | 4 | 3 | 2 | 3 | 4 | Orange window → orange output cell = 4. Move one cell at a time. Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions. Tap the diagram to enlarge. ### Solution — step by step **Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3. **Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**. **Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**. **Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**. **Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**. **Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**. **Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**. **Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**. **Final answer:** Y = $\begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}$ . All nine outputs use the same nine filter weights. They do not introduce nine different filters. **Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain. --- ## TEE · Faculty example: combine three RGB channel sums Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s15 **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)** ### Given Red patch = $\begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}$ ; red kernel = $\begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}$ . Green patch = $\begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}$ ; green kernel = $\begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}$ . Blue patch = $\begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}$ ; blue kernel = $\begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}$ . Bias = 1. These are three slices of **one** full-depth filter, not three independent output filters. ### To find The contribution of each input channel and the final scalar output at this position. ### Formula — which one and why Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions. $$ z=s_R+s_G+s_B+b $$ The quantities s\_R, s\_G and s\_B are intermediate weighted sums, not separate final output channels. ### Solution — step by step **Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1. **Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**. **Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1). **Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**. **Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1. **Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**. **Step 7: add the channel contributions.** 308 −498 +164 = −26. **Step 8: add one filter bias.** −26 +1 = **−25**. **Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages. **Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo. --- ## TEE · Max, min and average pooling on the same lecture grid Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s16 **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases. **Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)** ### Given X = $\begin{bmatrix}12 & 20 & 30 & 0\\8 & 12 & 2 & 0\\34 & 70 & 37 & 4\\112 & 100 & 25 & 12\end{bmatrix}$ . Window = 2 × 2; stride = 2; padding = 0. ### To find The shared output shape and each of the four cells for each pooling method. ### Formula — which one and why Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap. $$ y_{\max}=\max(x_1,x_2,x_3,x_4) $$ $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{\mathrm{avg}}=\frac{x_1+x_2+x_3+x_4}{4} $$ ![The four colours identify four separate pooling windows. Each window creates one cell in each output map.](https://deep-learning-study-room.vercel.app/assets/figures/u4_pooling_windows.svg) Diagram: The four colours identify four separate pooling windows. Each window creates one cell in each output map.. Labels: u4 pooling windows | Same four windows, three different summaries | Input: 4 × 4 | 12 | 20 | 30 | 0 | 8 | 12 | 2 | 0 | 34 | 70 | 37 | 4 | 112 | 100 | 25 | 12 | Max | 20 | 30 | 112 | 37 | Min | 8 | 0 | 34 | 4 | Average | 13 | 8 | 79 | 19.5 | Bottom-right average: (37 + 4 + 25 + 12) ÷ 4 = 19.5 | Pooling changes values and shape; ordinary pooling learns no parameters. The four colours identify four separate pooling windows. Each window creates one cell in each output map. Tap the diagram to enlarge. ### Solution — step by step **Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2. **Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13. **Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8. **Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79. **Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20. **Step 6: put each result back in its window's position.** Max = $\begin{bmatrix}20 & 30\\112 & 37\end{bmatrix}$ . Min = $\begin{bmatrix}8 & 0\\34 & 4\end{bmatrix}$ . Average = $\begin{bmatrix}13 & 8\\79 & 19.5\end{bmatrix}$ . **Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation. ### Why the choice matters Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour. Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter. **Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide. --- ## TEE · Faculty CNN table: shapes, activations and 62958 parameters Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s17/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s17 **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)** ### Given Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question. ### To find Output shape and number of output values for each layer, its trainable parameters, and the total parameters. ### Formula — which one and why Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ A counts produced activations. P\_conv and P\_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here. ### Solution — step by step **Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0. **Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272. **Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**. **Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight. **Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600. **Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**. **Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters. **Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations. **Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations. **Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation. **Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**. | Layer | Output shape | Activations | Parameters | | --- | --- | --- | --- | | Input | 32 × 32 × 3 | 3,072 | 0 | | Conv1 | 28 × 28 × 8 | 6,272 | 608 | | Pool1 | 14 × 14 × 8 | 1,568 | 0 | | Conv2 | 10 × 10 × 16 | 1,600 | 3,216 | | Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 | | Dense 120 | 120 | 120 | 48,120 | | Dense 84 | 84 | 84 | 10,164 | | Dense 10 → softmax | 10 | 10 | 850 + 0 | **Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network. **Yaad rakho:** Shape aage carry karo; weights aur activations ko alag columns mein rakho. --- ## TEE · Even-sized kernels: SAME padding is not always symmetric Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c7-s18/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/7/tee-c7-s18 **Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides. For a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention. **Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho. ### Example 1: one entry on every side **Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)** ### Given Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter. ### To find The padded input dimensions and the convolution output dimensions. ### Formula — which one and why The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel. ### Solution — step by step **Step 1: count the two padding borders.** $$ 2P=2(1)=2 $$ **Step 2: find the padded side length.** $$ N_{\mathrm{padded}}=5+2=7 $$ The padded input is 7 × 7 × 1. **Step 3: subtract the kernel size.** $$ 7-2=5 $$ **Step 4: divide by stride, take the floor and count the first placement.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5}{1}\right\rfloor+1=5+1=6 $$ **Step 5: write height, width and channels.** Both spatial dimensions equal 6. There is one filter, so there is one output channel. **Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel. ### Example 2: preserve the size with SAME padding **Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)** ### Given Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found. ### To find Padding above, below, left and right; padded-input shape; final output shape. ### Formula — which one and why For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+P_{\mathrm{before}}+P_{\mathrm{after}}-K}{S}\right\rfloor+1 $$ At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1. $$ P_{\mathrm{total}}=P_{\mathrm{before}}+P_{\mathrm{after}}=K-1 $$ ### Solution — step by step **Step 1: calculate total padding along the height.** $$ P_{\mathrm{total}}=2-1=1 $$ Only one extra row in total is needed. **Step 2: place the row using the stated convention.** Padding above = 0; padding below = 1. **Step 3: apply the same calculation to width.** Only one extra column in total is needed. Padding left = 0; padding right = 1. **Step 4: find the padded-input shape.** $$ H_{\mathrm{padded}}=5+0+1=6 $$ $$ W_{\mathrm{padded}}=5+0+1=6 $$ The padded input is 6 × 6 × 1. **Step 5: count the output positions along each axis.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5+0+1-2}{1}\right\rfloor+1=4+1=5 $$ **Step 6: include the output channel.** One filter creates one output channel. **Final answer:** add **0 above, 1 below, 0 left and 1 right**; padded input = **6 × 6 × 1**; output = **5 × 5 × 1**. The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula. --- ## TEE · AlexNet and the reason for its components Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s0 **Definition.** AlexNet is a deep convolutional architecture introduced for ImageNet classification, conventionally described as five convolutional layers followed by three fully connected layers. Its historical significance is the effective combination of learned convolutional features, nonlinear activations, GPU training and regularization at large scale. ### Architecture and information flow Input image → convolution and ReLU stages → spatial pooling → fully connected representations → class scores and softmax. The original model used local response normalization in selected early stages and overlapping max pooling. Grouped computation across two GPUs was part of the original implementation. Exact sizes must follow the architecture specified in an examination question. ![Convolution extracts representations before the fully connected classification stages.](https://deep-learning-study-room.vercel.app/assets/figures/alexnet.svg) Diagram: Convolution extracts representations before the fully connected classification stages.. Labels: Input | 227×227×3 | Conv1 | 11×11, 96 | Pool | 3×3, s2 | Conv2 | 5×5, 256 | Pool | 3×3, s2 | Conv3 | 3×3, 384 | Conv4 | 3×3, 384 | Conv5 | 3×3, 256 | Pool + flat | 9216 | Dense | 4096 | Dense | 4096 | Output | 1000 | Read rows in order. Apply nonlinearities between learned layers. Convolution extracts representations before the fully connected classification stages. Tap the diagram to enlarge. ### Role of the components **ReLU** supplies a nonlinear transformation and avoids sigmoid-style saturation for positive preactivations. **Convolution** uses local connectivity and spatial weight sharing. **Pooling** reduces spatial dimensions and increases tolerance to small local changes. **Dropout** regularizes the fully connected layers. **Data augmentation** supplies transformed training examples to reduce dependence on a fixed view. These components perform different jobs and should not be described as interchangeable methods of reducing overfitting. ### Limitations and exam distinction The dense layers contribute a large number of parameters; the model has substantial memory and computation requirements. AlexNet is much shallower than later residual architectures. Its original use of local response normalization is different from batch normalization. For a comparison answer, identify its sequential convolutional design, then contrast it with Inception's parallel branches and ResNet's residual addition. --- ## TEE · Inception examines several spatial scales Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s1 **Definition.** An Inception module is a convolutional building block that processes the same input through parallel branches using different receptive-field operations and concatenates their output feature maps along the channel dimension. ### Structure and working A conventional early Inception module includes 1 × 1, 3 × 3 and 5 × 5 convolution branches and a pooling branch. Suitable stride and padding preserve compatible spatial dimensions. The branch outputs are then concatenated. If branch channel counts are c₁, c₂, c₃ and c₄, the output depth is c₁ + c₂ + c₃ + c₄. ![Branches operate on the same input. Matching spatial outputs are joined along the channel axis.](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Branches operate on the same input. Matching spatial outputs are joined along the channel axis.. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Branches operate on the same input. Matching spatial outputs are joined along the channel axis. Tap the diagram to enlarge. ### Why use several branches? Different branch receptive fields can capture patterns at different spatial scales. A 1 × 1 convolution mixes channel information at each position. Before an expensive larger kernel, it can reduce the number of input channels and therefore the parameter count and multiplication cost. The bottleneck calculation is developed in the following section. ### Concatenation is not addition Concatenation increases channel depth and retains each branch's feature maps separately. Elementwise addition requires equal shapes and combines corresponding values. Inception commonly uses concatenation; a standard residual block uses addition. A question giving branch sizes tests this distinction as well as the convolution formula. Early Inception designs reduce reliance on a large fully connected classifier through global average pooling. The parallel design still needs compatible branch shapes and carefully chosen channel widths; increasing the number of branches or filters increases computation. --- ## TEE · Why one by one convolution saves work Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s2 **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts. ![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg) Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications. Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge. **Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.** ### Given Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size. ### To find Direct MACs, total two-stage bottleneck MACs and percentage saved. ### Formula — which one and why Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs. ### Solution — step by step **Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400 $$ **Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels. $$ \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448 $$ **Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels. $$ \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200 $$ **Step 4: add the two bottleneck costs.** $$ \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648 $$ **Step 5: calculate the saving.** $$ \mathrm{saving}=120422400-12443648=107978752 $$ $$ \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\% $$ **Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption. ### A second bottleneck with different dimensions **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1. ### To find Direct MACs, bottleneck MACs and saving. ### Formula — which one and why Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers. ### Solution — step by step **Step 1: direct spatial convolution.** $$ 8(8)(8)(3)(3)(16)=73728 $$ **Step 2: 1 × 1 reduction.** $$ 8(8)(4)(1)(1)(16)=4096 $$ **Step 3: reduced-depth 3 × 3 convolution.** $$ 8(8)(8)(3)(3)(4)=18432 $$ **Step 4: add the stages and subtract from the direct cost.** $$ 4096+18432=22528 $$ $$ 73728-22528=51200 $$ **Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values. --- ## TEE · ResNet adds a shortcut Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s3 **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x. $$ y=F(x)+x $$ If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch. ![The two paths must have equal output shape before elementwise addition.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge. ### Effect on gradient propagation For the identity-shortcut formulation, the derivative includes a direct identity contribution. $$ \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I $$ This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients. ### Projection when dimensions differ If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride. $$ y=F(x)+W_sx $$ An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition. **Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum. ### Given x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3]. ### To find Find the three components of x+F(x). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. ### Solution — step by step **Step 1: add the first matching pair.** $$ 1+0.1=1.1 $$ **Step 2: add the second matching pair.** $$ 2+(-0.2)=1.8 $$ **Step 3: add the third matching pair.** $$ 3+0.3=3.3 $$ **Step 4: assemble in the original order.** $$ y=[1.1,1.8,3.3] $$ The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum. --- ## TEE · Transfer learning from a small dataset Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s4 **Definition.** Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers. ### Explanation and working Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort. ![Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change. Tap the diagram to enlarge. **Full scenario.** Build a face-attendance model with few images per student. Explain a training plan. **Step 1. Define and split the data.** Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization. **Step 2. Choose a suitable pretrained backbone.** For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available. **Step 3. Replace the classifier.** Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy. **Step 4. Freeze and train.** Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring. **Step 5. Fine-tune if justified.** Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint. **Step 6. Evaluate.** Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition. Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario. --- ## TEE · Model answer: feature extraction and fine-tuning Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s5 ## Question **Question.** Explain transfer learning for image classification with limited labeled data. Distinguish feature extraction from fine-tuning and describe the training procedure. ## Answer — step by step Transfer learning reuses knowledge represented by a model trained on a source task to support a related target task. A pretrained CNN can provide useful visual representations, reducing the target data and training effort needed compared with learning all features from random initialization. ![Freezing and fine-tuning differ in whether selected pretrained weights are allowed to update.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Freezing and fine-tuning differ in whether selected pretrained weights are allowed to update.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Freezing and fine-tuning differ in whether selected pretrained weights are allowed to update. Tap the diagram to enlarge. ### Procedure 1. Define the target classes and create separate training, validation and test splits. Prevent duplicate or closely related captures from leaking across splits. 2. Choose a suitable pretrained backbone and apply its required input size and preprocessing. 3. Replace the original classifier with a new head whose output count matches the target classes. 4. Freeze the backbone and train the new head. Only the unfrozen parameters receive optimizer updates. 5. If validation results justify it, unfreeze selected later backbone layers and continue training with a suitably smaller learning rate. 6. Select hyperparameters and checkpoints using validation data. Evaluate the selected model on the held-out test set. | Feature extraction | Fine-tuning | | --- | --- | | Pretrained backbone weights remain fixed | Some or all selected pretrained weights are updated | | New classifier learns from fixed representations | Representations can adapt to the target task | | Usually fewer trainable parameters | Usually more trainable parameters and greater risk of overfitting on a small dataset | The benefit depends on how useful the source features are for the target data. Transfer is not guaranteed to help every unrelated task. For a numerical, count all stored parameters separately from the trainable subset; freezing a layer does not delete its weights. --- ## TEE · How to draw the architecture diagrams Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s6 For AlexNet, draw a sequence of convolution/pooling stages followed by dense layers and classifier. For Inception, draw four parallel branches and a channel-concatenation join. For ResNet, draw one transformation branch and one shortcut meeting at a plus sign. For transfer learning, draw frozen pretrained base → new trainable head, then mark selected base layers optionally unfrozen for fine-tuning. ![Draw the sequential path, parallel branches or additive shortcut that explains the architecture.](https://deep-learning-study-room.vercel.app/assets/figures/v_architectures.svg) Diagram: Draw the sequential path, parallel branches or additive shortcut that explains the architecture.. Labels: Three architecture ideas to remember | Three architecture ideas to remember | These are different designs, not three consecutive layers of one network. | AlexNet | Sequential | convolution and | pooling stages | Inception | Parallel branches; | concatenate their | channels | ResNet | Learn a branch; add a | shortcut | A diagram should show the mechanism, not only the architecture name. Draw the sequential path, parallel branches or additive shortcut that explains the architecture. Tap the diagram to enlarge. --- ## TEE · Inception concatenation and residual addition Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s7 **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths. ### Explanation and working Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations. ![Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.](https://deep-learning-study-room.vercel.app/assets/figures/v_join_add.svg) Diagram: Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.. Labels: Concatenation and addition produce different shapes | Concatenation and addition produce different shapes | Use height and width 14 × 14 throughout. | CONCATENATE four branches | Channels: 16 + 32 + 8 + 8 | Result: 14 × 14 × 64 | Stack all 64 channels. | ADD two matching tensors | 14 × 14 × 64 + 14 × 14 × 64 | Result: 14 × 14 × 64 | Add each matching pair. | A 14 × 14 × 32 shortcut needs a projection before addition to a 14 × 14 × 64 | branch. Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape. Tap the diagram to enlarge. Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64. A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition. --- ## TEE · Model answer: AlexNet, Inception and ResNet Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s8 ## Question **Question.** Compare the main ideas of AlexNet, Inception and ResNet. Explain how to recognize their architecture diagrams. ## Answer — step by step All three are CNN architectures, but their characteristic structures differ. ![Each block illustrates the defining design idea; the full architecture diagrams give its detailed layout.](https://deep-learning-study-room.vercel.app/assets/figures/v_architectures.svg) Diagram: Each block illustrates the defining design idea; the full architecture diagrams give its detailed layout.. Labels: Three architecture ideas to remember | Three architecture ideas to remember | These are different designs, not three consecutive layers of one network. | AlexNet | Sequential | convolution and | pooling stages | Inception | Parallel branches; | concatenate their | channels | ResNet | Learn a branch; add a | shortcut | A diagram should show the mechanism, not only the architecture name. Each block illustrates the defining design idea; the full architecture diagrams give its detailed layout. Tap the diagram to enlarge. | Architecture | Main idea | What to show in a diagram | | --- | --- | --- | | AlexNet | Successive convolutional feature extraction followed by dense classification | Five convolutional layers and three fully connected layers, with the stated pooling/normalization stages | | Inception v1 | Parallel branches analyze features at different spatial scales | 1 × 1, reduced 3 × 3, reduced 5 × 5 and pooling branches joined by channel concatenation | | ResNet | Learn a residual transformation and add a shortcut | A transformation branch and input shortcut meeting at a plus sign | AlexNet's exam-relevant features include ReLU, GPU-based training, augmentation and dropout in the fully connected part. A parameter calculation must follow the given channel connectivity; a grouped implementation and a simplified fully connected-channel diagram need not have the same count. Inception uses 1 × 1 convolutions to mix channels and, when a smaller number of filters is chosen, reduce the channel depth before expensive spatial filters. Branch outputs must have compatible height and width for concatenation. Concatenation adds channel counts by joining their lists. ResNet computes a learned transformation F(x) and combines it with the shortcut: $$ H(x)=F(x)+x $$ This is entrywise addition. The two shapes must match; a projection can adapt a shortcut when they do not. The shortcut supplies an additional route for information and gradients and makes identity-like mappings easy to represent. **Key distinction:** an Inception join concatenates feature maps; a residual join adds corresponding values. Their diagram symbols and output-channel calculations must therefore differ. --- ## TEE · Applications of convolutional neural networks Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s9 **Definition.** A CNN application combines a convolutional feature extractor with an output head designed for a particular prediction task. Shared local filters learn spatial patterns; the output head determines what is predicted and the loss determines how it is trained. ### Classification, detection and segmentation | Task | Required output | What distinguishes it | | --- | --- | --- | | Image classification | One label or class distribution for the image | Answers what category the whole image belongs to | | Object detection | Object classes and bounding-box coordinates | Also identifies where each detected object is located | | Semantic segmentation | A class label or distribution for each pixel | Produces a spatial label map rather than one image-wide class | | Face recognition | Identity class or a learned face representation | Compares or classifies learned visual features | A softmax image classifier alone does not produce boxes or pixel labels. Detection and segmentation require appropriate task heads, targets and objectives. In an applications question, explain the prediction required instead of listing model names without their purpose. ### Examples across domains **Medical imaging:** classify an image or use a segmentation model to identify regions of interest. The target may be an image-level label or a pixel mask, so the architecture and loss must match that choice. **Autonomous perception:** use camera images for tasks such as detecting pedestrians and vehicles or segmenting road regions. These outputs are inputs to a larger perception and decision system. **Manufacturing:** classify product images as defective or non-defective, or localize defects when position information is needed. **Document analysis:** classify cropped handwritten characters; a complete document-reading system must also locate and order text regions. **Remote sensing:** classify land-use regions or segment different surface categories in satellite images. Spatial context and texture can help distinguish classes. ### Complete exam-style application answer **Question. Describe how a CNN can classify product photographs into three classes: acceptable, scratched and cracked. Explain the input, network stages, output activation, loss, training and evaluation.** **Step 1: define data and labels.** Each example contains an image and one of the three class labels. Use separate training, validation and test sets, keeping near-duplicate photographs of the same item out of different splits. Apply the chosen model's input size and scaling consistently. **Step 2: extract spatial features.** Convolutional layers calculate local responses using shared kernels. Nonlinear activations allow more complex combinations of those responses. Further convolution stages combine local patterns; pooling may reduce spatial dimensions. **Step 3: produce three class scores.** Convert the final representation into a feature vector, using flattening or an appropriate global pooling operation, and apply a dense layer with three outputs. These raw outputs are logits. **Step 4: calculate class probabilities and loss.** For mutually exclusive classes, use softmax and categorical cross-entropy. A one-hot label selects the probability of the correct class. $$ q_k=\frac{e^{z_k}}{\sum_{j=1}^{3}e^{z_j}},\qquad L=-\sum_{k=1}^{3}y_k\ln q_k $$ **Step 5: train the parameters.** Run forward propagation, evaluate the loss, backpropagate derivatives and update the trainable convolutional and dense parameters using the selected optimizer. Use validation results for hyperparameters and stopping decisions. **Step 6: evaluate the selected model.** Assess the final model on held-out photographs. Report overall classification performance and inspect which defects are confused. With limited data, a pretrained CNN can provide features and selected layers can be fine-tuned. **Numerical extension. If the last pooled feature vector has 64 components, how many parameters does the three-class dense head have, including bias?** **Step 1:** each output receives 64 weights, so three outputs require 64 × 3 = 192 weights. **Step 2:** each of the three outputs has one bias, giving 3 biases. $$ P_{\mathrm{head}}=64(3)+3=(64+1)3=195 $$ **Final answer:** 195 trainable parameters in this head. Softmax adds no trainable parameters. The backbone's parameters are counted separately. --- ## TEE · Reading LeNet and AlexNet layer diagrams Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s10 **A layer diagram describes two different things:** the size of the data moving through the network and the trainable operations that transform that data. A number such as 9,216 beside a flatten operation describes data length; it does not automatically mean another dense layer with 9,216 neurons. ### LeNet: follow the image as it becomes a class prediction The classroom LeNet layout begins with a 32 × 32 grayscale image. A valid 5 × 5 convolution with six filters produces 28 × 28 × 6. A 2 × 2 pooling operation with stride 2 produces 14 × 14 × 6. The next valid 5 × 5 convolution produces 10 × 10 × 16, and pooling reduces this to 5 × 5 × 16. A 5 × 5 convolution across those 16 channels produces 1 × 1 × 120. The final representation passes through 84 units and then ten class outputs. | Stage | What happens | Output shape | | --- | --- | --- | | Input | One grayscale image | 32 × 32 × 1 | | C1 | Six 5 × 5 filters, stride 1, no padding | 28 × 28 × 6 | | S2 | 2 × 2 average pooling, stride 2 | 14 × 14 × 6 | | C3 | Sixteen 5 × 5 filters, stride 1, no padding | 10 × 10 × 16 | | S4 | 2 × 2 average pooling, stride 2 | 5 × 5 × 16 | | C5 | 120 filters covering the complete 5 × 5 spatial map | 1 × 1 × 120 | | F6 | Dense transformation | 84 values | | Classifier | Modern classroom softmax classifier | Ten probabilities | The original LeNet-5 used details such as partial channel connectivity, trainable subsampling and an RBF output. A modern full-channel implementation with ordinary pooling and softmax is a different parameter-count convention. For a numerical, follow the question's diagram and state the convention. The complete LeNet parameter numerical in the solved-paper section uses the modern classroom convention. ### AlexNet: identify the flatten step correctly ![Five convolutional layers extract image features; three dense layers produce the classifier output. Flattening only rearranges the final feature maps.](https://deep-learning-study-room.vercel.app/assets/figures/alexnet.svg) Diagram: Five convolutional layers extract image features; three dense layers produce the classifier output. Flattening only rearranges the final feature maps.. Labels: Input | 227×227×3 | Conv1 | 11×11, 96 | Pool | 3×3, s2 | Conv2 | 5×5, 256 | Pool | 3×3, s2 | Conv3 | 3×3, 384 | Conv4 | 3×3, 384 | Conv5 | 3×3, 256 | Pool + flat | 9216 | Dense | 4096 | Dense | 4096 | Output | 1000 | Read rows in order. Apply nonlinearities between learned layers. Five convolutional layers extract image features; three dense layers produce the classifier output. Flattening only rearranges the final feature maps. Tap the diagram to enlarge. For the faculty's 227 × 227 × 3 classroom input, the main shape sequence is 55 × 55 × 96 → 27 × 27 × 96 → 27 × 27 × 256 → 13 × 13 × 256 → 13 × 13 × 384 → 13 × 13 × 384 → 13 × 13 × 256 → 6 × 6 × 256. Flattening gives 9,216 values; the three dense outputs are 4,096, 4,096 and 1,000. There is no extra learned 9,216-unit layer between flattening and the first 4,096-unit layer. **Question (practice, 3 marks). An AlexNet-style first layer receives a 227 × 227 RGB image and uses 96 filters of size 11 × 11, stride 4 and no padding. Find its output shape, trainable parameters including bias, and the number of output activation values.** ### Given Input height and width = 227; input channels = 3; kernel = 11; stride = 4; padding = 0; output filters = 96. One bias is used per filter. ### To find The three-dimensional output shape, parameter count and activation count. ### Formula — which one and why The output-size formula counts valid window positions. Parameter count counts stored filter coefficients, reused at every position. Activation count counts the values produced for one image. $$ H_o=W_o=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ A=H_oW_oC_{\mathrm{out}} $$ Here N is input side length, P is padding per side, K is kernel side length, S is stride, and C denotes channel count. The symbol P in the output formula means padding; the labelled quantity P\_conv means parameters. ### Solution — step by step **Step 1: calculate one spatial side.** $$ \frac{227-11}{4}+1=\frac{216}{4}+1=54+1=55 $$ **Step 2: attach the output depth.** There is one output map per filter, so the output is **55 × 55 × 96**. **Step 3: count the weights in one filter.** $$ 11(11)(3)=363 $$ **Step 4: add one bias and multiply by 96 filters.** $$ P_{\mathrm{conv}}=(363+1)(96)=34944 $$ **Step 5: count activation values for one image.** $$ A=55(55)(96)=290400 $$ **Final answer:** output = 55 × 55 × 96; parameters = 34,944; activations = 290,400. The weights are reused, so parameters and activations are different counts. The full faculty-style ungrouped AlexNet table totals 62,378,344 parameters; the original grouped model uses different channel connectivity. **Yaad rakho:** shape = kitne answers nikle; parameters = kitne numbers seekhe. --- ## TEE · Top-1 and top-5 error: what an ImageNet result means Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s11 **Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models. For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures. **Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.** ### Given Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes. ### To find The percentage of mistakes under each rule. ### Formula — which one and why Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage. $$ E_k=\frac{N-C_k}{N}\times100\% $$ N means total evaluated images; C\_k means images whose correct class appears among the top k predictions. ### Solution — step by step **Step 1: top-1 mistakes.** 100 − 78 = 22. **Step 2: top-1 error percentage.** $$ E_1=\frac{22}{100}\times100\%=22\% $$ **Step 3: top-5 mistakes.** 100 − 92 = 8. **Step 4: top-5 error percentage.** $$ E_5=\frac{8}{100}\times100\%=8\% $$ **Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions. **Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success. --- ## TEE · VGG16: small filters in a regular sequence Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s12 ![Follow the five blocks and their output sizes. The 16 counts learned layers, not pooling or flattening.](https://deep-learning-study-room.vercel.app/assets/figures/u4_vgg16.svg) Diagram: Follow the five blocks and their output sizes. The 16 counts learned layers, not pooling or flattening.. Labels: u4 vgg16 | VGG16: 13 convolution layers + 3 dense layers | Block 1 | 2 convolutions | Block 2 | 2 convolutions | Block 3 | 3 convolutions | Block 4 | 3 convolutions | Block 5 | 3 convolutions | 112 × 112 | 56 × 56 | 28 × 28 | 14 × 14 | 7 × 7 | Sizes shown after each 2 × 2 pool, starting from a 224 × 224 image. | 7 × 7 × 512 → flatten 25,088 → 4,096 → 4,096 → 1,000 Follow the five blocks and their output sizes. The 16 counts learned layers, not pooling or flattening. Tap the diagram to enlarge. **Definition.** VGG16 is a convolutional architecture with 13 convolutional layers and three dense layers. Its standard convolution blocks contain 2, 2, 3, 3 and 3 convolutions. Pooling, activations and flattening are not counted in the name's 16 weight layers. Each convolution uses a 3 × 3 kernel, stride 1 and padding 1, preserving height and width. Each block ends with 2 × 2 max pooling of stride 2. The final classifier is 4,096 → 4,096 → 1,000 for the original 1,000-class task. | Block | Convolutions | Channels after its convolutions | Size after its pooling | | --- | --- | --- | --- | | 1 | Two | 64 | 112 × 112 × 64 | | 2 | Two | 128 | 56 × 56 × 128 | | 3 | Three | 256 | 28 × 28 × 256 | | 4 | Three | 512 | 14 × 14 × 512 | | 5 | Three | 512 | 7 × 7 × 512 | This table starts with 224 × 224 × 3. The flattened length is 7 × 7 × 512 = 25,088. Flattening has zero learned parameters. Stacking small filters provides repeated nonlinear transformations while progressively combining spatial information. A limitation is the large dense classifier and its memory cost. **Question (practice, 3 marks). VGG16's first block contains two same-padded 3 × 3 convolutions with 64 output filters each, followed by 2 × 2 max pooling with stride 2. Input is 224 × 224 × 3. Find both convolution parameter counts including bias and the pooled output shape.** ### Given Conv1: 3 input channels, 64 output channels. Conv2: 64 input channels, 64 output channels. Both have kernel 3, stride 1 and padding 1. Pooling has kernel 2, stride 2 and no padding. ### To find Trainable parameters in each convolution and the output after pooling. ### Formula — which one and why $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Use each layer's own input depth; the second layer receives the first layer's 64 maps. Use the spatial-output formula for pooling. Max pooling introduces no trainable parameters. ### Solution — step by step **Step 1: first convolution.** Each filter has 3 × 3 × 3 = 27 weights and one bias. $$ P_1=(27+1)(64)=1792 $$ **Step 2: second convolution.** Each filter now spans 64 channels, giving 3 × 3 × 64 = 576 weights. $$ P_2=(576+1)(64)=36928 $$ **Step 3: add parameters.** $$ P_{\mathrm{block}}=1792+36928=38720 $$ **Step 4: calculate pooled side length.** The same-padded convolutions retained side length 224. $$ \frac{224-2}{2}+1=111+1=112 $$ **Final answer:** 1,792 and 36,928 parameters; total = 38,720; pooled shape = **112 × 112 × 64**. **Exam convention:** use the architecture explicitly supplied in a calculation. For the standard VGG16 architecture, remember **2–2–3–3–3 convolutions and 2 × 2 pooling**; 25,088 is the flattened feature length. --- ## TEE · Global average pooling: one average for each channel Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s13 ![Average each channel separately: these are the exact values used in the worked example.](https://deep-learning-study-room.vercel.app/assets/figures/u4_gap.svg) Diagram: Average each channel separately: these are the exact values used in the worked example.. Labels: u4 gap | Global average pooling keeps channels separate | Channel A | 1 | 3 | 5 | 7 | → | 4 | 16 ÷ 4 | Channel B | 2 | 4 | 6 | 8 | → | 5 | 20 ÷ 4 | 2 × 2 × 2 input → two channel means → [4, 5] Average each channel separately: these are the exact values used in the worked example. Tap the diagram to enlarge. **Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier. **Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = $\begin{bmatrix}1 & 3\\5 & 7\end{bmatrix}$ and channel B = $\begin{bmatrix}2 & 4\\6 & 8\end{bmatrix}$ . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.** ### Given H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs. ### To find The GAP result and dense-head parameter count. ### Formula — which one and why Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias. $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Solution — step by step **Step 1: add channel A.** 1 + 3 + 5 + 7 = 16. **Step 2: average channel A.** 16 ÷ 4 = 4. **Step 3: add channel B.** 2 + 4 + 6 + 8 = 20. **Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**. **Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights. **Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero. **Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters. **Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao. --- ## TEE · The faculty's 480-channel bottleneck numerical Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s14 ![Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.](https://deep-learning-study-room.vercel.app/assets/figures/u4_bottleneck_480.svg) Diagram: Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.. Labels: u4 bottleneck 480 | Same output shape, different computation | Input | 14 × 14 × 480 | → | Direct 5 × 5 | 48 filters | → | Output | 14 × 14 × 48 | 112,896,000 MACs | 1 × 1 reduction | 480 → 16 channels | → | Reduced 5 × 5 | 16 → 48 channels | → | Output | 14 × 14 × 48 | 1,505,280 + 3,763,200 = 5,268,480 MACs | The lower path also starts from 14 × 14 × 480. Count both layers. Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers. Tap the diagram to enlarge. **Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.** ### Given Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters. ### To find MACs for the direct path, both bottleneck stages, their total and percentage saved. ### Formula — which one and why $$ M=H_oW_oC_oK^2C_i $$ M is the MAC count. H\_oW\_oC\_o counts output values; K²C\_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck. $$ \mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}} $$ ### Solution — step by step **Step 1: direct output count.** 14 × 14 × 48 = 9,408. **Step 2: products per direct output.** 5 × 5 × 480 = 12,000. **Step 3: direct MACs.** $$ 9408(12000)=112896000 $$ **Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values. $$ 14(14)(16)(480)=1505280 $$ **Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values. $$ 9408(400)=3763200 $$ **Step 6: add both bottleneck costs.** $$ 1505280+3763200=5268480 $$ **Step 7: calculate the saving.** $$ 112896000-5268480=107627520 $$ $$ 100\frac{107627520}{112896000}\approx95.333333\% $$ **Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone. --- ## TEE · Batch normalization inside a residual network Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s15 **Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training. For one feature with m values in a training batch: $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ The batch mean μ\_B is the centre; batch variance σ²\_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch. **Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.** ### Given x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5. ### To find Batch mean, variance, normalized values and scale-and-shift outputs. ### Formula — which one and why Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale. ### Solution — step by step **Step 1: mean.** $$ \mu_B=\frac{2+4}{2}=3 $$ **Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1. **Step 3: squared deviations and variance.** $$ \sigma_B^2=\frac{(-1)^2+1^2}{2}=\frac{1+1}{2}=1 $$ **Step 4: stable denominator.** $$ d=\sqrt{1+0.00001}\approx1.00000499999 $$ **Step 5: normalize separately.** $$ \hat{x}_1=\frac{-1}{d}\approx-0.99999500004 $$ $$ \hat{x}_2=\frac{1}{d}\approx0.99999500004 $$ **Step 6: scale the first normalized value, then shift it.** $$ y_1=2(-0.99999500004)+0.5\approx-1.49999000008 $$ **Step 7: scale the second normalized value, then shift it.** $$ y_2=2(0.99999500004)+0.5\approx2.49999000008 $$ **Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here. **Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo. --- ## TEE · Residual-block order and architecture qualifications Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s16 The original post-activation basic block has this sequence: **convolution → BN → ReLU → convolution → BN → add shortcut → ReLU**. Thus the second branch convolution is not followed by a separate ReLU before the addition in this particular block. A pre-activation ResNet is another design; use the stated diagram. ![Add the shortcut to the residual branch, then apply the post-addition ReLU when the specified block requires it.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: Add the shortcut to the residual branch, then apply the post-addition ReLU when the specified block requires it.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. Add the shortcut to the residual branch, then apply the post-addition ReLU when the specified block requires it. Tap the diagram to enlarge. An identity shortcut has no learned parameters. A projection shortcut changes the shape using learned weights. Original ImageNet ResNets include an initial max-pooling operation; a basic residual block does not need a max-pooling layer inside it. ResNet-18/34 use two-convolution basic blocks, whereas deeper standard variants use bottleneck blocks. Avoid treating every ResNet as the same two-layer diagram. The main advantage is easier optimization of deep networks: shortcuts provide an additional information and gradient path. This reduces the degradation problem but does not prove immunity to every vanishing/exploding-gradient case. Large networks still need compute, memory, suitable data and regularization. Reusing early features can help; it does not guarantee that every deeper model generalizes better. --- ## TEE · Choosing frozen layers from dataset size and similarity Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s17/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s17 Early image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom. | Target dataset | Sensible starting choice | Reason | | --- | --- | --- | | Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely | | Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters | | Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky | | Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested | These are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently. **Example.** With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate. **Yaad rakho:** data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho. --- ## TEE · Four ways to reuse a pretrained model Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c8-s18/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/8/tee-c8-s18 **Definition.** A pretrained model has already learned parameters from an earlier training task. We can reuse its complete classifier, reuse its feature extractor, or use its learned weights as the starting point for further training. The choice determines which output labels are available and which parameters change. **First identify two parts.** The **backbone** converts an input image into useful features. The **head** converts those features into the required prediction, such as class scores. A new task can reuse the backbone while replacing the head. **Yaad rakhne ka cue:** Purane model ka poora answer use karna, sirf features lena, ya uske weights ko aage train karna—yeh teen decisions alag hain. The faculty reference divides these choices into the following four uses. The two feature-extractor uses differ in whether features are saved separately or computed inside the new model. | Use | What happens to the pretrained part? | What learns for the target task? | Concrete example | | --- | --- | --- | --- | | Direct classifier | Keep the complete model and its existing output head. Run it on new inputs. | Nothing is updated during ordinary prediction. | A model already trained to classify cats and dogs labels new cat/dog photographs. It can do this because the required labels match its existing outputs. | | Standalone feature extractor | Run images through the fixed backbone and save the resulting feature vectors. A feature vector is an ordered list of numbers describing the image. | Train a separate classifier using these saved vectors. | Extract one vector per product photograph, then train a classifier to distinguish acceptable, scratched and cracked products. The extractor stays unchanged. | | Integrated feature extractor | Place the pretrained backbone inside the new network and freeze it. Add a new head. The backbone calculates features whenever the new model runs. | Train the new head while the backbone's weights remain fixed. | Use a frozen image backbone followed by a new three-output head for acceptable, scratched and cracked products. The complete image-to-class pipeline is one model. | | Weight initialization followed by fine-tuning | Start from the pretrained weights, then allow selected pretrained layers to update. Replace or adapt the head as required. | Train the new head and the unfrozen part of the backbone. | After first training a product-defect head, unfreeze the last backbone block and train it with a smaller learning rate to adapt to the factory's photographs. | ### Why the existing class labels matter A cat/dog head has two outputs with learned meanings: cat and dog. Giving it a photograph of a scratched product does not create a new output called “scratched.” For the product task, use an appropriate new head and train it on labeled product examples. Direct reuse of the original classifier is suitable only when its output meanings and preprocessing match the required task. When the task and data distribution are unchanged, this is ordinary inference rather than adaptation to a new task. ### Standalone versus integrated features Both methods can keep the backbone frozen. In the standalone method, feature vectors are calculated first and saved as a separate dataset; a later classifier reads that dataset. In the integrated method, each image passes through the backbone and head within one pipeline. Freezing means that the backbone parameters receive no training updates; it does not mean that the backbone stops performing calculations or that its stored weights disappear. ### When transfer learning is useful It is useful when labeled target data are limited and a pretrained model has learned representations relevant to the target task. It can reduce training time and the amount of target data needed. Improvement is not guaranteed: unrelated source features, unsuitable preprocessing or excessive fine-tuning on a small dataset can hurt performance. Choose the frozen or fine-tuned configuration using validation data, then evaluate the chosen model on the held-out test set. **Exam answer structure:** define transfer learning → identify the backbone and head → explain the selected reuse mode → state which weights remain fixed and which learn → give a matching example and a limitation. --- ## TEE · Complete the entire sequence chapter Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s0 All sections in this chapter belong to the TEE preparation route. Learn vanilla RNNs first, then the shared-gradient calculation, then LSTM and GRU. Do not skip the gates or BPTT because the MT2 route ends earlier. ![Begin with the vanilla state, then learn why the gated variants carry and update memory differently.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_comparison.svg) Diagram: Begin with the vanilla state, then learn why the gated variants carry and update memory differently.. Labels: What each recurrent cell carries forward | What each recurrent cell carries forward | Compare three different cell designs. These are alternatives, not consecutive | layers. | Vanilla RNN | One hidden state h | No memory gates | LSTM | Cell C and hidden h | Forget / input / | output gates | GRU | One hidden state h | Reset / update gates | At equal dimensions, a standard LSTM uses four affine transforms; a basic GRU | uses three. Begin with the vanilla state, then learn why the gated variants carry and update memory differently. Tap the diagram to enlarge. --- ## TEE · What makes a sequence different Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s1 **Definition.** A recurrent neural network (RNN) is a neural architecture that models sequential data by updating a hidden state from the current input and the previous state, using shared parameters across time steps. $$ h_t=\phi(W_{xh}x_t+W_{hh}h_{t-1}+b_h) $$ The state hₜ is a learned representation of information available up to time t. It is not a guaranteed lossless record of the entire sequence. The initial state is supplied, initialized to zero or learned, according to the model or question. ### Why order matters In language, audio and time series, the meaning of an observation can depend on earlier observations and their order. A basic feedforward network treating each input independently has no recurrent state. An RNN allows the output at a time step to depend on previous processing through hₜ₋₁. ![Unrolling shows successive evaluations of one recurrent cell with shared parameters.](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Unrolling shows successive evaluations of one recurrent cell with shared parameters.. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Unrolling shows successive evaluations of one recurrent cell with shared parameters. Tap the diagram to enlarge. ### Parameter sharing The unrolled cells use the same Wₓₕ, Wₕₕ and bₕ at every time step. A longer input sequence increases the number of evaluations, not the number of independently learned copies of these parameters. Each new hidden state must be calculated before it is used by the next step. ### MT2 interpretation For TEE, study the state equation, time-step computation, shared parameters and sequence-to-output configurations, then BPTT, vanishing/exploding gradients, LSTM and GRU. All are part of Unit 5. The faculty update received on 12 October focuses MT2 on Units 3 and 4; this complete recurrent-network chapter is TEE preparation. --- ## TEE · Model answer: why recurrent networks use a hidden state Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s2 ## Question **Question.** What is a recurrent neural network? Explain how a hidden state allows it to process a sequence and distinguish it from a feedforward network. ## Answer — step by step An RNN processes ordered inputs using a hidden state that is carried from one time step to the next. The current state combines information from the new input with information retained from earlier inputs. The same recurrent weights are reused across time steps. ![The previous hidden state brings earlier information into the current calculation alongside the new input.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: The previous hidden state brings earlier information into the current calculation alongside the new input.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. The previous hidden state brings earlier information into the current calculation alongside the new input. Tap the diagram to enlarge. For a simple tanh RNN: $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h) $$ | Symbol | Meaning | | --- | --- | | xₜ | Input at the current time step | | hₜ₋₁ | Hidden state passed from the previous step | | Wxh | Weights connecting the current input to the hidden state | | Whh | Weights connecting the previous state to the current state | | bₕ | Hidden-state bias | | hₜ | New state after the activation | For words in a sentence, earlier words affect the state used when the next word arrives. Changing the order can therefore change the final representation. The state is a learned summary; it is not a guaranteed exact copy of every earlier input. A feedforward network maps inputs through its layers without this recurrent state transition. An RNN can accept a sequence, reuse its parameters at each step and produce either an output at each step or an output from a final state, depending on the task. Examples include sentiment classification, next-word prediction and time-series modeling. Unrolling an RNN draws its repeated use at t = 1, 2, 3 and so on. Those boxes share parameters; drawing three time steps does not create three independently learned copies of Whh. --- ## TEE · Reading recurrent memory and gates Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s3 ### Time means position in the sequence t = 1 is the first input; t = 2 is the second. The state h₀ is the starting memory, often given as zero. At each step the input changes and the state changes, while the learned weights are reused. ![A gate is a multiplier. Its value controls how much of a particular contribution gets through.](https://deep-learning-study-room.vercel.app/assets/figures/v_gate.svg) Diagram: A gate is a multiplier. Its value controls how much of a particular contribution gets through.. Labels: A gate multiplies the value passing through it | A gate multiplies the value passing through it | Example old value = 8. Compare three gate settings. | Gate = 0 | 8 × 0 = 0 | Pass none. | Gate = 0.25 | 8 × 0.25 = 2 | Pass one quarter. | Gate = 1 | 8 × 1 = 8 | Pass all. | Sigmoid gates lie between 0 and 1. A negative candidate can still contribute a | negative value. A gate is a multiplier. Its value controls how much of a particular contribution gets through. Tap the diagram to enlarge. ### Raw score and activated state First multiply and add to form a raw score. Then apply tanh or the activation specified in the question. The activated result is the hidden state passed to the next time step. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h,\qquad h_t=\tanh(a_t) $$ Wxh means the weight mapping input x to hidden state h. Whh maps the previous hidden state to the new hidden calculation. These labels describe a connection; they do not mean multiply the letters together. ### What a gate does A gate is a multiplier controlling how much of a value passes through. Gate 0 keeps none of that contribution; gate 1 keeps all; gate 0.25 keeps a quarter. For example, 0.25 × 8 = 2. Sigmoid-generated gates lie between 0 and 1 for finite raw scores. ### A gate is different from its raw score If a gate's raw score is 0, sigmoid gives 1/(1 + 1) = 0.5. A supplied gate value 0, however, is already zero and must not be passed through sigmoid again. Read whether the question gives **preactivations** or **gate outputs**. ### Memory values can be negative The gate may be nonnegative while the value being gated is negative. Multiplying 0.2 by −0.5 gives −0.1. This is normal: the gate chooses an amount, and the candidate content supplies its sign. --- ## TEE · The vanilla RNN equations from scratch Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s4 **Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction. ### Hidden-state and output equations For a tanh recurrent layer: $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h $$ $$ h_t=\tanh(a_t) $$ aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit. For classification at time t, an output layer can produce logits and probabilities: $$ z_t=W_{hy}h_t+b_y,\qquad p_t=\operatorname{softmax}(z_t) $$ A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation. ![The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state. Tap the diagram to enlarge. ### Complete numerical: two recurrent time steps **Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation. ### Given h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0. x₁ = 1; x₂ = 2; activation = tanh. ### To find Find h₁ and h₂ in order. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step **Step 1: Time step 1 — calculate the preactivation.** $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ **Apply the recurrent activation.** $$ h_1=\tanh(0.5)\approx0.462117157 $$ This activated value, not a₁ = 0.5, is passed to the next time step. **Step 2: Time step 2 — combine the new input and previous state.** $$ a_2=0.5(2)+0.2h_1+0 $$ $$ a_2\approx1+0.2(0.462117157)\approx1.092423431 $$ **Apply tanh to the complete new preactivation.** $$ h_2=\tanh(a_2)\approx0.797760954 $$ | Time step | Current input | Previous state | Preactivation | New state | | --- | --- | --- | --- | --- | | 1 | 1 | 0 | 0.5 | 0.462117157 | | 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 | **Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update. --- ## TEE · A complete vector RNN calculation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s5 **Question from R26 Q5a.** xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons. ### Given Input x = [0,1]ᵀ; previous h = [0,1]ᵀ. Wxh = $\begin{bmatrix}0.6 & 0.4\\0.3 & 0.7\end{bmatrix}$ . Whh = $\begin{bmatrix}0.5 & 0.3\\0.2 & 0.4\end{bmatrix}$ ; bias = 0. Apply tanh separately to the two sums. ### To find Find the new two-component hidden state; output-layer parameters are not supplied. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_deac39f14c.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Wxh | 0.6 | 0.4 | 0.3 | 0.7 | Whh | 0.5 | 0.3 | 0.2 | 0.4 Given matrices for this question Tap the diagram to enlarge. **Step 1 new input:** Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7]. **Step 2 old memory:** Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4]. **Step 3 add:** a = [0.4+0.3, 0.7+0.4] = [0.7,1.1]. **Step 4 activate each element:** h = [tanh(0.7), tanh(1.1)] = **[0.604368,0.800499]**. An output-layer prediction cannot be numerically calculated without its weights and bias. --- ## TEE · How an RNN is trained Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s6 **Definition.** Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters. ### Explanation and working Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT. ![Unroll forward through time, then accumulate backward contributions for the shared parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_bptt.svg) Diagram: Unroll forward through time, then accumulate backward contributions for the shared parameters.. Labels: One shared weight receives contributions from many times | One shared weight receives contributions from many times | Unrolling draws repeated uses of one cell, not independent copies of its | weights. | Time 1 | Same weights W | Time 2 | Same weights W | Time 3 | Same weights W | Gradients flow backward through the state dependencies | Add shared-weight contributions | Total gradient = contribution 1 + contribution 2 + | contribution 3 | After summing the required contributions, update each shared parameter once. Unroll forward through time, then accumulate backward contributions for the shared parameters. Tap the diagram to enlarge. For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero. $$ \delta_t=\left(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1}\right)\odot(1-h_t^2) $$ $$ \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T},\quad \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^{T},\quad \frac{\partial L}{\partial b_h}=\sum_t\delta_t $$ The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters. ### One BPTT delta at h = 0.5 **Question. In a scalar tanh RNN, h\_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.** ### Given Activated h\_t = 0.5; q\_t = 0.2; δ\_(t+1) = 0.3; recurrent weight u = 0.4. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.4(0.3)=0.12 $$ **Step 2: add the current output contribution.** $$ 0.2+0.12=0.32 $$ **Step 3: calculate the current tanh slope.** $$ 1-(0.5)^2=0.75 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.32)(0.75)=0.24 $$ **Final answer:** δ\_t = 0.24. ### One BPTT delta at h = -0.2 **Question. In a scalar tanh RNN, h\_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.** ### Given Activated h\_t = -0.2; q\_t = 0.1; δ\_(t+1) = 0; recurrent weight u = 0.5. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.5(0)=0 $$ **Step 2: add the current output contribution.** $$ 0.1+0=0.1 $$ **Step 3: calculate the current tanh slope.** $$ 1-(-0.2)^2=0.96 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.1)(0.96)=0.096 $$ **Final answer:** δ\_t = 0.096. --- ## TEE · BPTT with every arithmetic step visible Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s7 **Question.** A scalar RNN uses the linear state equation hₜ = u xₜ + v hₜ₋₁ + b. Its prediction is ŷ = h₂ and its loss is L = ½(ŷ−1)². Given u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1 and x₂ = 2, calculate the gradients of u, v and b using BPTT, then update them once using learning rate η = 0.1. ### Given hₜ = uxₜ+vhₜ₋₁+b; prediction = h₂. u = 0.2; v = 0.5; b = 0; h₀ = 0; x₁ = 1; x₂ = 2. Target = 1; half squared loss; η = 0.1. ### To find Find gradients for u,v,b and update each once. ### Formula — which one and why **Backpropagation through time.** Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once. For h\_t=ux\_t+vh\_(t−1)+b with loss only at the last state: δ\_t = loss derivative at that state. The linear recurrence has activation slope 1. $$ \delta_T=h_T-y $$ $$ \delta_t=v\delta_{t+1} $$ $$ g_u=\sum_t\delta_tx_t $$ $$ g_v=\sum_t\delta_th_{t-1} $$ $$ g_b=\sum_t\delta_t $$ **Yaad rakhne ka cue:** Time ke har use ka gradient jodo; shared weight ko ek baar update karo. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Model and dependency structure u is the input weight, v is the recurrent weight and b is the hidden bias. Each is shared by both time steps. The hidden activation is linear in this question, so its derivative is 1. There is a loss only at time step 2; time step 1 influences that loss through h₂. ![Follow the two shared-parameter state calculations forward, then sum the backward contributions from each use of u, v and b.](https://deep-learning-study-room.vercel.app/assets/figures/v_bptt_worked.svg) Diagram: Follow the two shared-parameter state calculations forward, then sum the backward contributions from each use of u, v and b.. Labels: The two-step RNN in the worked question | The two-step RNN in the worked question | Shared weights u = 0.2, v = 0.5; bias 0; linear hidden activation. | Start | h₀ = 0 | Time 1: x₁ = 1 | h₁ = 0.2 | Time 2: x₂ = 2 | h₂ = 0.5 | δ₂ = 0.5 − 1 = −0.5 | Backward through v = 0.5 | δ₁ = −0.5 × 0.5 = −0.25 | Input weight gradient: (−0.25 × 1) + (−0.5 × 2) = −1.25 | Recurrent weight gradient: (−0.25 × 0) + (−0.5 × 0.2) = −0.1 | Shared bias gradient: −0.25 + (−0.5) = −0.75 | Each row is a sum over time. Update u, v and b only after calculating these three | gradients. Follow the two shared-parameter state calculations forward, then sum the backward contributions from each use of u, v and b. Tap the diagram to enlarge. ### Step 1: complete the forward pass $$ h_1=ux_1+vh_0+b $$ $$ h_1=0.2(1)+0.5(0)+0=0.2 $$ $$ h_2=ux_2+vh_1+b $$ $$ h_2=0.2(2)+0.5(0.2)+0=0.5 $$ $$ L=\frac12(0.5-1)^2=0.125 $$ ### Step 2: differentiate the loss at the final step Because the output is h₂ and the state activation is linear: $$ \delta_2=\frac{\partial L}{\partial h_2}=h_2-1=-0.5 $$ ### Step 3: propagate the derivative to the earlier state The local derivative of h₂ with respect to h₁ is the recurrent weight v = 0.5. Applying the chain rule: $$ \delta_1=\frac{\partial L}{\partial h_1}=\frac{\partial L}{\partial h_2}\frac{\partial h_2}{\partial h_1} $$ $$ \delta_1=\delta_2v=(-0.5)(0.5)=-0.25 $$ There is no separate direct loss at time step 1 to add here. Both deltas use the original recurrent weight. ### Step 4: sum the contributions to each shared parameter u appears in both h₁ and h₂, so its gradient receives one contribution from each use: $$ \frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2 $$ $$ \frac{\partial L}{\partial u}=(-0.25)(1)+(-0.5)(2)=-1.25 $$ For v, the sending activation is the previous hidden state at the corresponding time: $$ \frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1 $$ $$ \frac{\partial L}{\partial v}=(-0.25)(0)+(-0.5)(0.2)=-0.1 $$ The same additive bias is used at both steps. Its two local derivatives are 1: $$ \frac{\partial L}{\partial b}=\delta_1+\delta_2=-0.25-0.5=-0.75 $$ | Shared parameter | Contribution at time 1 | Contribution at time 2 | Total gradient | | --- | --- | --- | --- | | u | −0.25 × 1 = −0.25 | −0.5 × 2 = −1 | −1.25 | | v | −0.25 × 0 = 0 | −0.5 × 0.2 = −0.1 | −0.1 | | b | −0.25 | −0.5 | −0.75 | ### Step 5: update each shared parameter once $$ u_{\mathrm{new}}=0.2-0.1(-1.25)=0.325 $$ $$ v_{\mathrm{new}}=0.5-0.1(-0.1)=0.51 $$ $$ b_{\mathrm{new}}=0-0.1(-0.75)=0.075 $$ **Final answer:** gradients are −1.25, −0.1 and −0.75; updated parameters are u = 0.325, v = 0.51 and b = 0.075. ### Evaluate the updated model Restart from the same h₀ = 0 and use all updated parameters together: $$ h_{1,\mathrm{new}}=0.325(1)+0.51(0)+0.075=0.4 $$ $$ h_{2,\mathrm{new}}=0.325(2)+0.51(0.4)+0.075=0.929 $$ $$ L_{\mathrm{new}}=\frac12(0.929-1)^2=0.0025205 $$ The loss decreases in this particular update. That is consistent with the computed gradient, but a decrease is not guaranteed for every choice of learning rate. ### Extension to a tanh RNN For tanh, the derivative at each state transition also contains 1−hₜ². A trainable output layer contributes its own local derivatives. If losses occur at several time steps, add the direct output-loss contribution and the future-state contribution before applying the local activation derivative. Truncated BPTT limits the number of time steps traversed backward; it does not create separate copies of the recurrent parameters. ### Second worked example: nonlinear tanh BPTT **Question.** A scalar RNN uses hₜ = tanh(uxₜ + vhₜ₋₁ + b). Given u = 0.4, v = 0.3, b = 0.1, h₀ = 0 and inputs x₁ = 1, x₂ = −1, its final output is h₂ and target is 0.2. Use only the final-time loss L = ½(h₂ − 0.2)². Calculate both states, all three shared-parameter gradients and one simultaneous update at η = 0.1. ### Given Shared input weight u = 0.4; shared recurrent weight v = 0.3; shared bias b = 0.1. Initial state h₀ = 0; x₁ = 1, x₂ = −1. Hidden activation = tanh; final target = 0.2; learning rate η = 0.1. There is no separate output layer and no loss at time 1. ### To find Forward states h₁ and h₂, final loss, gradients with respect to u, v and b, and all three updated parameters. ### Formula — which one and why Backpropagation must pass through each tanh derivative. Because u, v and b are shared, add their local contributions from both time steps before updating them. $$ z_t=ux_t+vh_{t-1}+b,\qquad h_t=\tanh(z_t) $$ $$ \delta_2=(h_2-y)(1-h_2^2),\qquad \delta_1=v\delta_2(1-h_1^2) $$ $$ \frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2 $$ $$ \frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1,\qquad \frac{\partial L}{\partial b}=\delta_1+\delta_2 $$ Here δₜ means ∂L/∂zₜ, not ∂L/∂hₜ. The factor v in δ₁ is the **old recurrent weight 0.3**. The derivative of tanh is 1 − hₜ², using that step's activated state. ### Solution — step by step ### Step 1: calculate the first state z₁ = 0.4(1) + 0.3(0) + 0.1 = 0.5. h₁ = tanh(0.5) = **0.462117157**. ### Step 2: carry that state into time 2 z₂ = 0.4(−1) + 0.3(0.462117157) + 0.1 = **−0.161364853**. h₂ = tanh(−0.161364853) = **−0.159978716**. ### Step 3: calculate the final loss and final-time delta Prediction error h₂ − y = −0.159978716 − 0.2 = **−0.359978716**. L = ½(−0.359978716)² = **0.064792338**. δ₂ = (−0.359978716)[1 − (−0.159978716)²] = **−0.350765712**. ### Step 4: backpropagate through the recurrent connection and first tanh There is no direct loss at time 1. Its gradient arrives entirely through time 2: δ₁ = 0.3(−0.350765712)[1 − (0.462117157)²] = **−0.082757670**. ### Step 5: sum each shared parameter's contributions over time | Parameter | Time contributions | Total gradient | | --- | --- | --- | | u | (−0.082757670)(1) + (−0.350765712)(−1) | 0.268008043 | | v | (−0.082757670)(0) + (−0.350765712)(0.462117157) | −0.162094854 | | b | −0.082757670 + (−0.350765712) | −0.433523382 | For v, the sending state is hₜ₋₁: h₀ for time 1, h₁ for time 2. The negative second input makes its contribution to the u gradient positive. ### Step 6: update all parameters using the same old forward pass u\_new = 0.4 − 0.1(0.268008043) = **0.373199196**. v\_new = 0.3 − 0.1(−0.162094854) = **0.316209485**. b\_new = 0.1 − 0.1(−0.433523382) = **0.143352338**. ### Step 7: interpret the update with a fresh forward pass Using the new parameters from h₀ = 0 gives h₁,new = tanh(0.373199196 + 0.143352338) = **0.475034096**. h₂,new = tanh(−0.373199196 + 0.316209485(0.475034096) + 0.143352338) = **−0.079468645**. The new loss is ½(−0.079468645 − 0.2)² = **0.039051362**. **Final answer:** gradients (u,v,b) = **(0.268008043, −0.162094854, −0.433523382)** and updated parameters = **(0.373199196, 0.316209485, 0.143352338)**. This step moves the final prediction toward 0.2 and reduces loss from 0.064792338 to 0.039051362. Both time steps contributed to learning the same parameters. --- ## TEE · Vanishing and exploding gradients Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s8 **Definition.** Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment. ### Explanation and working In BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity. ![Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.](https://deep-learning-study-room.vercel.app/assets/figures/v_vanishing.svg) Diagram: Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.. Labels: Repeated multiplication changes gradient size | Repeated multiplication changes gradient size | These scalar examples illustrate one part of the BPTT mechanism. | Repeated factors below 1 | 0.5 × 0.5 × … × 0.5 | Ten factors: 0.0009765625 | Earlier learning signal is tiny. | Repeated large factors | 2 × 2 × … × 2 | Ten factors: 1,024 | Updates can become unstable. | Gradient clipping limits explosion. It does not rebuild a gradient that has | already vanished. Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates. Tap the diagram to enlarge. **Numerical intuition:** 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs. Useful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory. ### Why the effect grows with sequence length For a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians: $$ \frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(1-h_t^2)W_{hh} $$ $$ \frac{\partial h_T}{\partial h_t}=\prod_{k=T}^{t+1}\frac{\partial h_k}{\partial h_{k-1}} $$ The product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone. ### Gradient norm clipping: a complete numerical Norm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient. $$ g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right) $$ If g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation. **Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.** ### Given Gradient g = [6,8]ᵀ; norm threshold τ = 5. Old weights w = [1,1]ᵀ; η = 0.1. ### To find Find the clipped gradient and updated weights. ### Formula — which one and why **Gradient norm clipping.** If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction. g = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate. $$ \lVert g\rVert_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min\left(1,\frac\tau{\lVert g\rVert_2}\right) $$ $$ w_{\mathrm{new}}=w-\eta g_{\mathrm{clip}} $$ **Yaad rakhne ka cue:** Bada gradient? Sab components ko ek hi factor se chhota karo. ### Solution — step by step **Step 1: calculate the original norm.** $$ \lVert g\rVert_2=\sqrt{6^2+8^2}=\sqrt{36+64}=10 $$ **Step 2: find the scale.** Since 10 > 5, scale the vector by 5/10 = 0.5. $$ g_{\mathrm{clipped}}=0.5\begin{bmatrix}6\\8\end{bmatrix}=\begin{bmatrix}3\\4\end{bmatrix} $$ **Step 3: verify the clipped norm.** $$ \sqrt{3^2+4^2}=\sqrt{9+16}=5 $$ **Step 4: update each parameter with the clipped gradient.** $$ w_{1,\mathrm{new}}=1-0.1(3)=0.7 $$ $$ w_{2,\mathrm{new}}=1-0.1(4)=0.6 $$ **Final answer:** clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture. ### Gradient norm clipping: [3, 4] **Question. Clip gradient g = [3, 4] to maximum L2 norm 10. Use global norm clipping.** ### Given g = [3, 4]; threshold τ = 10. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 3^2+4^2=25 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{25}=5 $$ **Step 3: choose the shared scale.** $$ c=\min(1,10/5)=1 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=3(1)=3 $$ $$ g_2^{\mathrm{clip}}=4(1)=4 $$ **Final answer:** Clipped gradient = [3, 4]. ### Gradient norm clipping: [0, 12] **Question. Clip gradient g = [0, 12] to maximum L2 norm 3. Use global norm clipping.** ### Given g = [0, 12]; threshold τ = 3. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 0^2+12^2=144 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{144}=12 $$ **Step 3: choose the shared scale.** $$ c=\min(1,3/12)=0.25 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=0(0.25)=0 $$ $$ g_2^{\mathrm{clip}}=12(0.25)=3 $$ **Final answer:** Clipped gradient = [0.0, 3.0]. --- ## TEE · Model answer: backpropagation through time Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s9 ## Question **Question.** Explain backpropagation through time and the vanishing/exploding gradient problems in a recurrent neural network. ## Answer — step by step Backpropagation through time, or BPTT, applies backpropagation to an RNN unrolled across a sequence. The forward pass computes the state and any output at each time step. The backward pass follows how later losses depend on earlier states and shared parameters. ![The drawn copies reuse one parameter set. Their backward contributions add into one gradient for each shared parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_bptt.svg) Diagram: The drawn copies reuse one parameter set. Their backward contributions add into one gradient for each shared parameter.. Labels: One shared weight receives contributions from many times | One shared weight receives contributions from many times | Unrolling draws repeated uses of one cell, not independent copies of its | weights. | Time 1 | Same weights W | Time 2 | Same weights W | Time 3 | Same weights W | Gradients flow backward through the state dependencies | Add shared-weight contributions | Total gradient = contribution 1 + contribution 2 + | contribution 3 | After summing the required contributions, update each shared parameter once. The drawn copies reuse one parameter set. Their backward contributions add into one gradient for each shared parameter. Tap the diagram to enlarge. ### Working procedure 1. Start from the supplied initial state and compute h₁, h₂ and all subsequent states in order. 2. Calculate the loss at the output positions specified by the task. 3. Propagate loss derivatives backward through the output computations and recurrent state transitions. 4. Add contributions from all time steps that use the same parameter. 5. Apply the optimizer update to the shared parameter after its gradient contributions have been accumulated. ### Why gradients can vanish or explode The chain rule repeatedly multiplies derivatives through time. A simple illustration is multiplying by 0.5 ten times, giving about 0.000977, versus multiplying by 2 ten times, giving 1,024. Real RNNs involve matrices and activation derivatives, so these scalars illustrate the mechanism rather than define a universal threshold. Vanishing gradients make distant earlier inputs receive very weak learning signals. Exploding gradients can produce extremely large updates and unstable training. Gradient clipping limits gradient magnitude before an optimizer update and is useful for exploding gradients. It does not by itself restore already vanished information. LSTM/GRU gating, suitable initialization and architecture choices can improve long-range learning. Truncated BPTT limits how far the backward computation runs for computational reasons; it does not mean that each unrolled box has independent parameters. --- ## TEE · LSTM a memory path with three gates Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s10 **Definition.** Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell. ### Explanation and working Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference. ![Follow the cell state from left to right then identify forget input and output gates](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: Follow the cell state from left to right then identify forget input and output gates. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. Follow the cell state from left to right then identify forget input and output gates Tap the diagram to enlarge. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c) $$ $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\qquad h_t=o_t\odot\tanh(C_t) $$ The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state. --- ## TEE · Reading an LSTM equation one operation at a time Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s11 ### 1. Two memories on the page do not mean the same value The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other. ![Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_split.svg) Diagram: Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.. Labels: Calculate cell memory before the hidden output | Calculate cell memory before the hidden output | Example: old cell 0.8, forget gate 0.25, input gate 0.5, candidate −0.4. | Retain old content | 0.25 × 0.8 = 0.2 | Admit new content | 0.5 × (−0.4) = −0.2 | New cell | 0.2 + (−0.2) = 0 | Hidden output: 0.7 × tanh(0) = 0 | The output gate is 0.7 here. Apply it to tanh of the NEW cell, after the cell | update. Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output. Tap the diagram to enlarge. ### 2. Gate values are multipliers For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention. An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive. ### 3. Add old and new contributions With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel. $$ c_t=f_tc_{t-1}+i_t\widetilde c_t $$ This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update. ### 4. Calculate the hidden state afterward Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0. $$ h_t=o_t\tanh(c_t) $$ If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c. ### 5. Supplied gates and calculated gates are different numerical types When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition. --- ## TEE · LSTM complete numerical with supplied gates Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s12 **Question.** An LSTM cell has input gate iₜ = 0.7, forget gate fₜ = 0.4, output gate oₜ = 0.8, candidate cell content C̃ₜ = 0.9 and previous cell state Cₜ₋₁ = 0.5. Calculate the new cell state Cₜ and hidden state hₜ. ### Given Activated input gate i = 0.7; forget gate f = 0.4; output gate o = 0.8. Candidate = 0.9; old cell = 0.5. ### To find Find the new cell C and hidden state h. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Equations and supplied-value convention The question supplies **activated gate outputs** and the candidate content. Do not apply sigmoid or tanh to those supplied values again. First update the cell, then calculate the hidden state: $$ C_t=f_tC_{t-1}+i_t\widetilde C_t $$ $$ h_t=o_t\tanh(C_t) $$ The first equation combines retained old content and admitted candidate content. The second equation is a separate transformation of the updated cell. ![The diagram uses the supplied gate values. The output gate is applied after the cell has been updated and passed through tanh.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_given.svg) Diagram: The diagram uses the supplied gate values. The output gate is applied after the cell has been updated and passed through tanh.. Labels: When the LSTM gate values are already supplied | When the LSTM gate values are already supplied | Example: f = 0.4, old C = 0.5, i = 0.7, candidate = 0.9, o = 0.8. | Old part | 0.4 × 0.5 = | 0.2 | New part | 0.7 × 0.9 = | 0.63 | New cell | 0.2 + 0.63 = | 0.83 | Hidden output | 0.8 × | tanh(0.83) ≈ | 0.544381 | The final hidden output includes tanh; it is not just 0.8 × 0.83. The diagram uses the supplied gate values. The output gate is applied after the cell has been updated and passed through tanh. Tap the diagram to enlarge. **Step 1: calculate the retained old-cell contribution.** $$ f_tC_{t-1}=0.4(0.5)=0.2 $$ **Step 2: calculate the admitted candidate contribution.** $$ i_t\widetilde C_t=0.7(0.9)=0.63 $$ **Step 3: add the two contributions to obtain the new cell.** $$ C_t=0.2+0.63=0.83 $$ **Step 4: activate the updated cell.** $$ \tanh(C_t)=\tanh(0.83)\approx0.680476006 $$ **Step 5: multiply by the output gate.** $$ h_t=0.8\tanh(0.83)\approx0.544380805 $$ **Final answer:** Cₜ = 0.83 and hₜ ≈ 0.544381. Multiplying the output gate directly by Cₜ would give 0.664 and omit the required tanh operation. ### Related vector calculation: retention alone **Question.** Given forget gate f = [0.1, 0.9] and previous cell C = [2, 1], calculate the retained old-cell contribution. ### Given Forget gate = [0.1,0.9]; old cell = [2,1]. Input-gate and candidate values are not supplied. ### To find Find only the retained old-cell contribution. ### Formula — which one and why **LSTM retained old memory.** Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell. f = activated forget gate; C\_old = previous cell state; ⊙ means multiply matching entries. $$ C_{\mathrm{retained}}=f\odot C_{\mathrm{old}} $$ **Yaad rakhne ka cue:** Forget gate × old cell = bacha hua purana hissa. ### Solution — step by step **Step 1: retain the first old-cell component.** $$ 0.1(2)=0.2 $$ **Step 2: retain the second old-cell component.** $$ 0.9(1)=0.9 $$ **Step 3: assemble the two retained components.** Keep their original order. This gives the following column: $$ f\odot C=\begin{bmatrix}0.1(2)\\0.9(1)\end{bmatrix}=\begin{bmatrix}0.2\\0.9\end{bmatrix} $$ **Answer:** [0.2, 0.9]. This is the retained contribution, not necessarily the final new cell. The candidate contribution must also be known to calculate the full cell update. --- ## TEE · LSTM numerical when gates must first be calculated Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s13 **Question.** At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state. ### Given x = 1; old hidden state = 0; old cell = 0.2. Forget/input/output preactivations are all 0. Candidate preactivation = 0.5. ### To find Activate all gates and candidate, then find C and h. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Distinguish preactivations from gate outputs The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate. $$ \sigma(a)=\frac1{1+e^{-a}} $$ $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ ![A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_gates_first.svg) Diagram: A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.. Labels: When the question supplies gate raw scores | When the question supplies gate raw scores | A gate raw score is not yet a gate value. | Gate scores | f, i, o scores | = 0 | Apply sigmoid | Each gate | becomes 0.5 | Candidate | score | 0.5 → tanh → | 0.462117 | Then update | Compute cell, | then hidden | output | Sigmoid(0) = 0.5. Do not replace a zero raw score with a closed gate. A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update. Tap the diagram to enlarge. **Step 1: activate the three gates.** $$ f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5 $$ **Step 2: activate the candidate preactivation.** $$ \widetilde C_t=\tanh(0.5)\approx0.462117157 $$ **Step 3: calculate the old and new cell contributions separately.** $$ f_tC_{t-1}=0.5(0.2)=0.1 $$ $$ i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579 $$ **Step 4: calculate the updated cell.** $$ C_t=0.1+0.5\tanh(0.5)\approx0.331058579 $$ **Step 5: calculate the hidden output from the updated cell.** $$ h_t=0.5\tanh(C_t)\approx0.159735818 $$ **Final answer:** fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely. ### LSTM gates from raw score 0 **Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 0. Old cell C\_old = 1; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(0)=\frac1{1+e^{-0}}\approx0.500000000 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(0)\approx0.000000000 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.462117157 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579 $$ **Final answer:** f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579. ### LSTM gates from raw score 1 **Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 1. Old cell C\_old = -0.5; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(1)=\frac1{1+e^{-1}}\approx0.731058579 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(1)\approx0.761594156 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.188942847 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289 $$ **Final answer:** f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289. --- ## TEE · GRU two gates and one state Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s14 **Definition.** A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state. ### Gate equations and the convention used here The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state: $$ r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z) $$ The candidate equation used in these notes applies reset before the recurrent candidate transformation: $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h) $$ These notes define z as the fraction of **new candidate** admitted: $$ h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture. ![First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention. Tap the diagram to enlarge. ### Complete numerical with supplied gate values **Question.** A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ. ### Given Old hidden state = 0.4; r = 0.5; z = 0.25. x = 1; W = 0.2; U = 0.5; b = 0. Here z weights the candidate in the final mixture. ### To find Find candidate and new hidden state. ### Formula — which one and why **GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate. r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state. $$ \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b) $$ $$ h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h $$ **Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control. ### Solution — step by step **Step 1: apply the reset gate to the previous hidden state.** $$ r_th_{t-1}=0.5(0.4)=0.2 $$ The gates are already supplied as outputs, so no further sigmoid calculation is needed. **Step 2: calculate the candidate preactivation.** $$ a_{\mathrm{candidate}}=W_hx_t+U_h(r_th_{t-1})+b_h $$ $$ a_{\mathrm{candidate}}=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh to obtain the candidate.** $$ \widetilde h_t=\tanh(0.3)\approx0.291312612 $$ **Step 4: calculate the two mixture contributions.** $$ (1-z_t)h_{t-1}=(1-0.25)(0.4)=0.3 $$ $$ z_t\widetilde h_t=0.25\tanh(0.3)\approx0.072828153 $$ **Step 5: add the contributions.** $$ h_t=0.3+0.25\tanh(0.3)\approx0.372828153 $$ **Final answer:** candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture. ### If the question uses the other update-gate convention Some texts define z as the fraction of the **old state retained**: $$ h_t=z_th_{t-1}+(1-z_t)\widetilde h_t $$ Under that explicitly different convention, the same numerical value z = 0.25 gives: $$ h_t=0.25(0.4)+0.75\tanh(0.3)\approx0.318484459 $$ This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values. ### Relationship to LSTM A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task. ### Retain-old GRU with gate 0 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = 0.4; z = 0; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: retain the old-state fraction.** $$ 0(0.4)=0 $$ **Step 5: calculate the candidate fraction.** $$ (1-0)(0.291312612)\approx0.291312612 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612 $$ **Final answer:** Candidate ≈ 0.291312612; new state ≈ 0.291312612. ### Retain-old GRU with gate 0.8 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = -0.2; z = 0.8; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(-0.2)=-0.1 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(-0.1)+0=0.15 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.15)\approx0.148885034 $$ **Step 4: retain the old-state fraction.** $$ 0.8(-0.2)=-0.16 $$ **Step 5: calculate the candidate fraction.** $$ (1-0.8)(0.148885034)\approx0.029777007 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993 $$ **Final answer:** Candidate ≈ 0.148885034; new state ≈ -0.130222993. --- ## TEE · Comparing the three recurrent models Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s15 | Feature | Vanilla RNN | LSTM | GRU | | --- | --- | --- | --- | | Main state | h | h and C | h | | Gates | None in vanilla cell | Forget input output | Reset update | | Long dependencies | Often difficult to learn | Designed to help retain useful long context | Designed to help retain useful long context | | Parameters at equal dimensions | Usually fewest | Usually most of these three | Usually between vanilla RNN and LSTM | | Typical choice | Simple short sequences | Complex long-context sequences | Efficient gated sequence baseline | With input size d and hidden size H, a basic single-bias vanilla recurrent layer has H(d+H+1) parameters; a standard LSTM has 4H(d+H+1); a basic GRU has 3H(d+H+1). These omit output layers; library versions with extra bias vectors differ. The factor four in LSTM counts three gates plus one candidate transform. ![Compare what is carried through time and how many gate or candidate transforms are used.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_comparison.svg) Diagram: Compare what is carried through time and how many gate or candidate transforms are used.. Labels: What each recurrent cell carries forward | What each recurrent cell carries forward | Compare three different cell designs. These are alternatives, not consecutive | layers. | Vanilla RNN | One hidden state h | No memory gates | LSTM | Cell C and hidden h | Forget / input / | output gates | GRU | One hidden state h | Reset / update gates | At equal dimensions, a standard LSTM uses four affine transforms; a basic GRU | uses three. Compare what is carried through time and how many gate or candidate transforms are used. Tap the diagram to enlarge. --- ## TEE · Model answer: LSTM and GRU memory Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s16 ## Question **Question.** Compare a vanilla RNN, an LSTM and a GRU. Explain the roles of the LSTM gates and state the GRU convention used in this course. ## Answer — step by step A vanilla RNN repeatedly transforms the current input and previous hidden state. LSTMs and GRUs add learned gates that control the retention and replacement of information. Gates usually use sigmoid, so their entries lie between 0 and 1; gate multiplication controls how much of a corresponding value is passed. ![LSTM carries a cell and hidden state; GRU carries a hidden state with two gate roles.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_comparison.svg) Diagram: LSTM carries a cell and hidden state; GRU carries a hidden state with two gate roles.. Labels: What each recurrent cell carries forward | What each recurrent cell carries forward | Compare three different cell designs. These are alternatives, not consecutive | layers. | Vanilla RNN | One hidden state h | No memory gates | LSTM | Cell C and hidden h | Forget / input / | output gates | GRU | One hidden state h | Reset / update gates | At equal dimensions, a standard LSTM uses four affine transforms; a basic GRU | uses three. LSTM carries a cell and hidden state; GRU carries a hidden state with two gate roles. Tap the diagram to enlarge. ### LSTM An LSTM carries a cell state c and a hidden state h. Its forget gate f controls the old cell contribution. Its input gate i controls admission of a new candidate c-tilde. Its output gate o controls the exposed hidden state. $$ c_t=f_t\odot c_{t-1}+i_t\odot\widetilde c_t $$ $$ h_t=o_t\odot\tanh(c_t) $$ The candidate is commonly formed using tanh, so it can be negative. Gate multiplication is elementwise. The updated cell is not the same as the hidden output; the second equation is another required step. ### GRU A GRU carries one hidden state rather than a separate LSTM-style cell state. A reset gate r controls how previous hidden information contributes to candidate formation. An update gate z mixes the old state and candidate. This guide uses: $$ h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ Thus z near 1 favors the new candidate under this convention. Some sources exchange z and 1 − z; always follow the equation printed in the question. | Model | Carried state | Main gates | Typical parameter implication for the same dimensions | | --- | --- | --- | --- | | Vanilla RNN | Hidden state | None of these memory gates | Fewest affine transformations | | LSTM | Cell and hidden states | Forget, input and output gates | Four affine transformations including candidate | | GRU | Hidden state | Reset and update gates | Three affine transformations including candidate | Gated models improve control over information flow but do not guarantee perfect indefinite memory. Their benefit must be evaluated for the sequence task. --- ## TEE · Sequence mappings and applications Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s17/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s17 **Definition.** A sequence-model configuration specifies how many inputs and outputs are processed and how their time indices relate. One-to-many, many-to-one and many-to-many configurations correspond to different prediction tasks. ### Explanation and working ![The number and timing of inputs and outputs determine the sequence mapping](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: The number and timing of inputs and outputs determine the sequence mapping. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. The number and timing of inputs and outputs determine the sequence mapping Tap the diagram to enlarge. | Mapping | Meaning | Example and output | | --- | --- | --- | | One to many | One input conditions several outputs | Image captioning: image to a word sequence | | Many to one | Sequence summarized as one output | Review sentiment: words to positive neutral or negative | | Many to many aligned | Output associated with each input time | Video frame labeling or per-word tagging | | Many to many unaligned | Output sequence length may differ | Encoder-decoder translation | Music generation can use earlier notes to predict the next note repeatedly. Next-word prediction uses the prefix to produce a probability distribution over a vocabulary. A word with the largest predicted probability is a simple choice, but the precise prediction cannot be calculated without the learned model's weights. **Sentence explanation:** In “The dogs that bark near my house are loud,” the model should retain the plural subject across the intervening words when choosing “are”. Gates can retain subject information and admit later context. This is an interpretation of what a trained model may learn, not a claim that an individual gate is guaranteed to represent grammar. **Sentiment model answer:** tokenize the review, map tokens to numerical embeddings, process them with an RNN/LSTM/GRU, use a final state or pooled sequence representation, then a dense layer and softmax over three sentiment classes. Train with categorical cross-entropy on labeled reviews; evaluate on held-out data. A bidirectional model is not needed for this answer and is not explicitly listed in the current syllabus. --- ## TEE · Recurrent networks: the complete relationship Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s18/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s18 A vanilla RNN repeatedly combines the new input and previous hidden state with shared weights. Backpropagation through time accumulates contributions from every use of those shared weights. LSTM adds an explicit cell state with forget, input and output gates. GRU mixes an old hidden state and a candidate using reset and update gates. ![Read from the ordinary recurrent calculation to the two gated memory designs.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_comparison.svg) Diagram: Read from the ordinary recurrent calculation to the two gated memory designs.. Labels: What each recurrent cell carries forward | What each recurrent cell carries forward | Compare three different cell designs. These are alternatives, not consecutive | layers. | Vanilla RNN | One hidden state h | No memory gates | LSTM | Cell C and hidden h | Forget / input / | output gates | GRU | One hidden state h | Reset / update gates | At equal dimensions, a standard LSTM uses four affine transforms; a basic GRU | uses three. Read from the ordinary recurrent calculation to the two gated memory designs. Tap the diagram to enlarge. For every recurrent calculation, compute the gate or hidden preactivation first, apply its stated activation, then combine the resulting values. Elementwise gate multiplication is different from matrix multiplication. The stated GRU update convention determines whether z retains the old state or selects the candidate. --- ## TEE · Gate logic without memorizing a story Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c9-s19/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/9/tee-c9-s19 In an LSTM, let the previous cell be 2, forget gate 0.5, input gate 0.25, candidate −0.4 and output gate 1. Calculate the retained part, admitted part, new cell and hidden output. ![The input gate multiplies candidate content. The output gate is applied after updating and activating the cell.](https://deep-learning-study-room.vercel.app/assets/figures/v_gate_order.svg) Diagram: The input gate multiplies candidate content. The output gate is applied after updating and activating the cell.. Labels: Keep the order of gate calculations straight | Keep the order of gate calculations straight | The cell equation specifies what each gate multiplies. | Forget path | Old cell × | forget gate | Input path | Candidate × | input gate | Cell update | Add those two | parts | Hidden output | tanh(new cell) | × output gate | The input gate multiplies the candidate, not simply the raw input x. The input gate multiplies candidate content. The output gate is applied after updating and activating the cell. Tap the diagram to enlarge. **Step 1 retained content:** 0.5 × 2 = 1. **Step 2 admitted content:** 0.25 × (−0.4) = −0.1. A candidate can be negative even though the gate itself is between zero and one. **Step 3 cell:** 1 + (−0.1) = 0.9. **Step 4 hidden:** output gate × tanh(cell) = 1 × tanh(0.9) ≈ 0.716298. The hidden output is not simply 0.9. Explain this distinction once before attempting another LSTM numerical. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-intro Sources S03, S01 and the three Teams photographs supplied later. MT1 establishes prerequisites; it does not prove that earlier concepts cannot reappear in MT2. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · MT1A Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s0 ## Question **Question (2 marks).** Define deep learning and explain its relationship with AI and machine learning. ## Answer — step by step Artificial intelligence is the broad goal of building systems that perform tasks associated with intelligence. Machine learning is a subset that learns patterns from data. Deep learning is a subset of machine learning using neural networks with multiple processing layers to learn representations. Thus DL ⊂ ML ⊂ AI. Example: a CNN learning image features is deep learning; a decision tree trained on tabular data is machine learning without being deep learning. --- ## TEE · MT1A Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s1 ## Question **Question (2 marks).** List three applications of deep learning and explain any one. ## Answer — step by step Three applications are image recognition, speech recognition and language translation. In image recognition, a CNN applies filters to learn local edges and textures, combines them into more complex features, and uses an output layer to assign class probabilities. It learns these features from labeled examples instead of requiring every image feature to be manually designed. --- ## TEE · MT1A Q2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s2 ## Question **Question (3 marks).** A sigmoid neuron has inputs [2,1,3], weights [0.4,−0.2,0.5] and bias 0.1. Compute its output. ## Answer — step by step ### Given Inputs x₁ = 2,x₂ = 1,x₃ = 3. Matching weights w₁ = 0.4,w₂ = −0.2,w₃ = 0.5. Bias b = 0.1; activation = sigmoid. ### To find Calculate net input z and activated output a. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. --- ## TEE · MT1A Q3 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s3 ## Question **Question (3 marks).** For z = [−1,0,1], calculate sigmoid, softmax and Leaky ReLU with α = 0.01 (an explicit assumption because the paper does not give the slope). ## Answer — step by step ### Given Scores z = [−1,0,1]. Calculate sigmoid, softmax and Leaky ReLU. The paper omits the Leaky slope: α = 0.01 is the stated practice assumption. ### To find Calculate every component under each of the three activations. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step Sigmoid is applied separately: σ(−1) = $\frac{1}{1+e^{1}}$ = 0.268941421; σ(0) = 1/2 = 0.5; σ(1) = $\frac{1}{1+e^{-1}}$ = 0.731058579. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-2, -1, 0]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.135335283, 0.367879441, 1]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.090030573, 0.244728471, 0.665240956]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Leaky ReLU multiplies only negative values by α: [0.01 × (−1),0,1] = **[−0.01,0,1]**. Softmax normalizes the entire vector; independent sigmoid values need not sum to 1. --- ## TEE · MT1A Q4 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s4 ## Question **Question (3 marks).** Input x = [2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1]; W₂ is shown below; b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_bbb5f79ed7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.5 | W₂ | 0.8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [2]. W₁ is printed with the question. b₁ = [0.1]. W₂ is printed with the question. b₂ = [0.2]. Hidden and output activations are ReLU. Target y = [1]. ### To find Find all preactivations, hidden activations and final outputs. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.5 \times 2) + (0.1) = 1.1 $$ $$ h_{1} = \max(0,1.1) = 1.1 $$ **Output neuron 1:** $$ z_{o_1} = (0.8 \times 1.1) + (0.2) = 1.08 $$ $$ o_{1} = \max(0,1.08) = 1.08 $$ **Step 2 loss convention.** The question specifies half squared loss. No learning rate is supplied, so report gradients, not invented weight updates. Prediction minus target = [0.08]. Squared errors = [0.0064]. Half-sum squared loss = 0.0032; mean squared error = 0.0064. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (1.08-1) \times 1 = 0.08 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**1.1**h1 × Receiving delta**0.08**δ at o1 = Connection gradient**0.088**h1 → o1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(0.08)\times(1.1)\approx 0.088 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.08** × Old outgoing weight**0.8** = Path contribution**0.064** ↓ Add every path contribution**0.064**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.8)\times(0.08)\approx 0.064 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx 0.064 $$ $$ \delta_{h_1} = 0.064 \times 1 = 0.064 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**2**x1 × Receiving delta**0.064**δ at h1 = Connection gradient**0.128**x1 → h1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.064)\times(2)\approx 0.128 $$ **Requested answers:** dL/dw₂ = **0.088** and dL/db₂ = **0.08**. Hidden gradients above explain the dependency and are additional practice. --- ## TEE · MT1B Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s5 ## Question **Question (2 marks).** State two motivations for using deep learning. ## Answer — step by step First, deep networks can learn useful representations directly from complex data such as images and speech, reducing dependence on manually engineered features. Second, multiple nonlinear layers can model complicated patterns and relationships when suitable data, computation and training are available. Example: a CNN can learn edges, parts and object features at successive levels. --- ## TEE · MT1B Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s6 ## Question **Question (2 marks).** Differentiate machine learning and deep learning with respect to feature extraction and data requirements. ## Answer — step by step Traditional machine-learning pipelines often rely on manually selected or engineered features; deep networks often learn feature representations jointly with the task. Training a large deep network from scratch commonly needs more data than simpler models. However, pretrained deep features can be useful even with a small target dataset. These are practical tendencies, not absolute definitions. --- ## TEE · MT1B Q2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s7 ## Question **Question (3 marks).** Show how a perceptron with w₁ = w₂ = 0.6 and bias −0.8 implements AND. ## Answer — step by step ### Given Binary inputs 00,01,10,11. w₁ = w₂ = 0.6; b = −0.8; output 1 when net ≥ 0. ### To find Calculate all four net values and outputs to show AND. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step ### Input (0, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0+0=0 $$ **Step 4: add the bias once.** $$ z=0+(-0.8)=-0.8 $$ **Step 5: apply the stated threshold.** z = -0.8; output = 0 because output is 1 exactly when z ≥ 0. ### Input (0, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0+0.6=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0.6+0=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0.6+0.6=1.2 $$ **Step 4: add the bias once.** $$ z=1.2+(-0.8)=0.4 $$ **Step 5: apply the stated threshold.** z = 0.4; output = 1 because output is 1 exactly when z ≥ 0. **Final answer:** outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND. --- ## TEE · MT1B Q3 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s8 ## Question **Question (3 marks).** Calculate sigmoid, ReLU and tanh for z = [−2,0,2]. ## Answer — step by step ### Given Input scores = [−2,0,2]. Required activations: sigmoid, ReLU and tanh. ### To find Calculate all three activation outputs for each score. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a². z = value entering tanh; a = its output. A prime means derivative, or local rate of change. $$ a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}} $$ $$ \frac{da}{dz}=1-a^2 $$ **Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square. ### Solution — step by step ### Preactivation z = -2 **Step 1: negate z for sigmoid.** $$ -z=2 $$ **Step 2: evaluate the exponential.** $$ e^{-z}\approx7.389056099 $$ **Step 3: add one to form the denominator.** $$ 1+7.389056099=8.389056099 $$ **Step 4: take the reciprocal for sigmoid.** $$ \sigma(-2)=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: compare z with zero for ReLU.** $$ \mathrm{ReLU}(-2)=\max(0,-2)=0 $$ **Step 6: calculate the tanh numerator.** $$ e^{-2}-e^{2}\approx0.135335283-7.389056099=-7.253720816 $$ **Step 7: calculate the tanh denominator.** $$ e^{-2}+e^{2}\approx0.135335283+7.389056099=7.524391382 $$ **Step 8: divide for tanh.** $$ \tanh(-2)\approx\frac{-7.253720816}{7.524391382}\approx-0.964027580 $$ ### Preactivation z = 0 **Step 1: negate z for sigmoid.** $$ -z=0 $$ **Step 2: evaluate the exponential.** $$ e^{-z}\approx1.000000000 $$ **Step 3: add one to form the denominator.** $$ 1+1.000000000=2.000000000 $$ **Step 4: take the reciprocal for sigmoid.** $$ \sigma(0)=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: compare z with zero for ReLU.** $$ \mathrm{ReLU}(0)=\max(0,0)=0 $$ **Step 6: calculate the tanh numerator.** $$ e^{0}-e^{0}\approx1.000000000-1.000000000=0.000000000 $$ **Step 7: calculate the tanh denominator.** $$ e^{0}+e^{0}\approx1.000000000+1.000000000=2.000000000 $$ **Step 8: divide for tanh.** $$ \tanh(0)\approx\frac{0.000000000}{2.000000000}\approx0.000000000 $$ ### Preactivation z = 2 **Step 1: negate z for sigmoid.** $$ -z=-2 $$ **Step 2: evaluate the exponential.** $$ e^{-z}\approx0.135335283 $$ **Step 3: add one to form the denominator.** $$ 1+0.135335283=1.135335283 $$ **Step 4: take the reciprocal for sigmoid.** $$ \sigma(2)=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: compare z with zero for ReLU.** $$ \mathrm{ReLU}(2)=\max(0,2)=2 $$ **Step 6: calculate the tanh numerator.** $$ e^{2}-e^{-2}\approx7.389056099-0.135335283=7.253720816 $$ **Step 7: calculate the tanh denominator.** $$ e^{2}+e^{-2}\approx7.389056099+0.135335283=7.524391382 $$ **Step 8: divide for tanh.** $$ \tanh(2)\approx\frac{7.253720816}{7.524391382}\approx0.964027580 $$ **Final answer:** sigmoid = [0.119202922, 0.5, 0.880797078]; ReLU = [0, 0, 2]; tanh = [−0.964027580, 0, 0.964027580]. --- ## TEE · MT1B Q4 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s9 ## Question **Question (3 marks).** Input x = [3]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. Use L = ½(o−y)². Calculate the gradients of the output weight and output bias. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ec03263237.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.4 | W₂ | 0.6 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [3]. W₁ is printed with the question. b₁ = [0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are sigmoid. Target y = [1]. ### To find Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.4 \times 3) + (0.2) = 1.4 $$ $$ h_{1}=\frac{1}{1+e^{-(1.4)}}=0.802183889 $$ **Output neuron 1:** $$ z_{o_1} = (0.6 \times 0.802183889) + (0.1) = 0.581310333 $$ $$ o_{1}=\frac{1}{1+e^{-(0.581310333)}}=0.641368858 $$ **Step 2 loss convention.** Use the half squared loss stated in this question. No learning rate is supplied. Prediction minus target = [-0.358631142]. Squared errors = [0.128616296]. Half-sum squared loss = 0.064308148; mean squared error = 0.128616296. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.641368858-1) \times 0.230014846 = -0.082490487 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.802183889**h1 × Receiving delta**−0.082490487**δ at o1 = Connection gradient**−0.066172539**h1 → o1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.082490487)\times(0.802183889)\approx -0.066172539 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.082490487** × Old outgoing weight**0.6** = Path contribution**−0.049494292** ↓ Add every path contribution**−0.049494292**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.6)\times(-0.082490487)\approx -0.049494292 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.049494292 $$ $$ \delta_{h_1} = -0.049494292 \times 0.158684897 = -0.007853997 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**3**x1 × Receiving delta**−0.007853997**δ at h1 = Connection gradient**−0.02356199**x1 → h1 The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.007853997)\times(3)\approx -0.02356199 $$ **Requested answers:** dL/dw₂ = **−0.066172539**; dL/db₂ = **−0.082490487**. Negative gradients mean ordinary gradient descent would increase those parameters. --- ## TEE · Teacher vectorized ReLU example Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s10 ![The teacher vectorized example with its exact connections](https://deep-learning-study-room.vercel.app/assets/figures/net_handwritten.svg) Diagram: The teacher vectorized example with its exact connections. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=2 | x2=3 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.11 + x2 × 0.21; bias 0 | h2 incoming: x1 × 0.12 + x2 × 0.08; bias 0 | o1 incoming: h1 × 0.14 + h2 × 0.15; bias 0 | Use these OLD connections for all gradients in the same pass. The teacher vectorized example with its exact connections Tap the diagram to enlarge. **Question.** Input x = [2, 3]. Each row below lists the incoming weights of one receiving neuron. W₁ = $\begin{bmatrix}0.11 & 0.21\\0.12 & 0.08\end{bmatrix}$ ; b₁ = [0, 0]; W₂ = $\begin{bmatrix}0.14 & 0.15\end{bmatrix}$ ; b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes. ### Given Input x = [2, 3]. W₁ = $\begin{bmatrix}0.11 & 0.21\\0.12 & 0.08\end{bmatrix}$ . b₁ = [0, 0]. W₂ = $\begin{bmatrix}0.14 & 0.15\end{bmatrix}$ . b₂ = [0]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.05. Use half squared loss and perform forward and backward passes. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.11 \times 2) + (0.21 \times 3) + (0) = 0.85 $$ $$ h_{1} = \max(0,0.85) = 0.85 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.12 \times 2) + (0.08 \times 3) + (0) = 0.48 $$ $$ h_{2} = \max(0,0.48) = 0.48 $$ **Output neuron 1:** $$ z_{o_1} = (0.14 \times 0.85) + (0.15 \times 0.48) + (0) = 0.191 $$ $$ o_{1} = \max(0,0.191) = 0.191 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.809]. Squared errors = [0.654481]. Half-sum squared loss = 0.3272405; mean squared error = 0.654481. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.191-1) \times 1 = -0.809 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.85**h1 × Receiving delta**−0.809**δ at o1 = Connection gradient**−0.68765**h1 → o1 Now update this weight Old weight**0.14** → New weight**0.1743825** Subtract η × gradient = **−0.0343825**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.809)\times(0.85)\approx -0.68765 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.14-0.05(-0.68765)\approx 0.1743825 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.48**h2 × Receiving delta**−0.809**δ at o1 = Connection gradient**−0.38832**h2 → o1 Now update this weight Old weight**0.15** → New weight**0.169416** Subtract η × gradient = **−0.019416**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.809)\times(0.48)\approx -0.38832 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.15-0.05(-0.38832)\approx 0.169416 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.809** × Old outgoing weight**0.14** = Path contribution**−0.11326** ↓ Add every path contribution**−0.11326**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.14)\times(-0.809)\approx -0.11326 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.11326 $$ $$ \delta_{h_1} = -0.11326 \times 1 = -0.11326 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.809** × Old outgoing weight**0.15** = Path contribution**−0.12135** ↓ Add every path contribution**−0.12135**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.15)\times(-0.809)\approx -0.12135 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.12135 $$ $$ \delta_{h_2} = -0.12135 \times 1 = -0.12135 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**2**x1 × Receiving delta**−0.11326**δ at h1 = Connection gradient**−0.22652**x1 → h1 Now update this weight Old weight**0.11** → New weight**0.121326** Subtract η × gradient = **−0.011326**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.11326)\times(2)\approx -0.22652 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.11-0.05(-0.22652)\approx 0.121326 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**3**x2 × Receiving delta**−0.11326**δ at h1 = Connection gradient**−0.33978**x2 → h1 Now update this weight Old weight**0.21** → New weight**0.226989** Subtract η × gradient = **−0.016989**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.11326)\times(3)\approx -0.33978 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.21-0.05(-0.33978)\approx 0.226989 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**2**x1 × Receiving delta**−0.12135**δ at h2 = Connection gradient**−0.2427**x1 → h2 Now update this weight Old weight**0.12** → New weight**0.132135** Subtract η × gradient = **−0.012135**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.12135)\times(2)\approx -0.2427 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.12-0.05(-0.2427)\approx 0.132135 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**3**x2 × Receiving delta**−0.12135**δ at h2 = Connection gradient**−0.36405**x2 → h2 Now update this weight Old weight**0.08** → New weight**0.0982025** Subtract η × gradient = **−0.0182025**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.12135)\times(3)\approx -0.36405 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.08-0.05(-0.36405)\approx 0.0982025 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0-0.05(-0.113260000)=0.005663000 $$ $$ b_{h_2,\mathrm{new}}=0-0.05(-0.121350000)=0.006067500 $$ $$ b_{o_1,\mathrm{new}}=0-0.05(-0.809000000)=0.040450000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Correction to the handwritten working:** the hidden deltas are negative (−0.11326 and −0.12135). A positive value at that step is a sign error. Keep output error as prediction minus target throughout. --- ## TEE · Teams photos full sigmoid example Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c10-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/10/tee-c10-s11 The three photographs show this 2 → 2 → 2 network and its chain rule. The photo label for w₃ is read as 0.25, consistent with the visible output/error values and the matching worked example in S05. Learning rate is not legible/specified in these photos; η = 0.5 below is an explicit practice assumption. **Question.** Input x = [0.05, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ = $\begin{bmatrix}0.15 & 0.2\\0.25 & 0.3\end{bmatrix}$ ; b₁ = [0.35, 0.35]; W₂ = $\begin{bmatrix}0.4 & 0.45\\0.5 & 0.55\end{bmatrix}$ ; b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. Compute all outputs, total half squared loss, all eight weight gradients and one simultaneous weight update. Map weights in order: w₁ = x₁ → h₁, w₂ = x₂ → h₁, w₃ = x₁ → h₂, w₄ = x₂ → h₂; w₅ = h₁ → o₁, w₆ = h₂ → o₁, w₇ = h₁ → o₂, w₈ = h₂ → o₂. ### Given Input x = [0.05, 0.1]. W₁ = $\begin{bmatrix}0.15 & 0.2\\0.25 & 0.3\end{bmatrix}$ . b₁ = [0.35, 0.35]. W₂ = $\begin{bmatrix}0.4 & 0.45\\0.5 & 0.55\end{bmatrix}$ . b₂ = [0.6, 0.6]. Hidden and output activations are sigmoid. Target y = [0.01, 0.99]. Learning rate η = 0.5. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![Every arrow in the teacher example is labeled so the two backward paths can be traced](https://deep-learning-study-room.vercel.app/assets/figures/teams.svg) Diagram: Every arrow in the teacher example is labeled so the two backward paths can be traced. Labels: INPUTS | HIDDEN | OUTPUTS | x1 | x2 | h1 | h2 | o1 | o2 | 0.05 | 0.10 | w1 = .15 | w2 = .20 | w3 = .25 | w4 = .30 | w5 = .40 | w6 = .45 | w7 = .50 | w8 = .55 | target .01 | target .99 | Hidden biases = .35 Output biases = .60 | Sigmoid at both learned layers. Half summed squared loss. Every arrow in the teacher example is labeled so the two backward paths can be traced Tap the diagram to enlarge. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.15 \times 0.05) + (0.2 \times 0.1) + (0.35) = 0.3775 $$ $$ h_{1}=\frac{1}{1+e^{-(0.3775)}}=0.593269992 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.25 \times 0.05) + (0.3 \times 0.1) + (0.35) = 0.3925 $$ $$ h_{2}=\frac{1}{1+e^{-(0.3925)}}=0.596884378 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.593269992) + (0.45 \times 0.596884378) + (0.6) = 1.105905967 $$ $$ o_{1}=\frac{1}{1+e^{-(1.105905967)}}=0.75136507 $$ **Output neuron 2:** $$ z_{o_2} = (0.5 \times 0.593269992) + (0.55 \times 0.596884378) + (0.6) = 1.224921404 $$ $$ o_{2}=\frac{1}{1+e^{-(1.224921404)}}=0.772928465 $$ **Step 2 loss convention.** The photos explicitly use ½(t₁−o₁)² + ½(t₂−o₂)². With exactly two output components, this equals their mean squared error, 0.298371109. The unhalved sum is 0.596742218. Prediction minus target = [0.74136507, -0.217071535]. Squared errors = [0.549622166, 0.047120051]. Half-sum squared loss = 0.298371109; mean squared error = 0.298371109. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.75136507-0.01) \times 0.186815602 = 0.138498562 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.772928465-0.99) \times 0.175510053 = -0.038098237 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.593269992**h1 × Receiving delta**0.138498562**δ at o1 = Connection gradient**0.082167041**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.35891648** Subtract η × gradient = **0.04108352**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(0.138498562)\times(0.593269992)\approx 0.082167041 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.5(0.082167041)\approx 0.35891648 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.596884378**h2 × Receiving delta**0.138498562**δ at o1 = Connection gradient**0.082667628**h2 → o1 Now update this weight Old weight**0.45** → New weight**0.408666186** Subtract η × gradient = **0.041333814**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(0.138498562)\times(0.596884378)\approx 0.082667628 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.45-0.5(0.082667628)\approx 0.408666186 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.593269992**h1 × Receiving delta**−0.038098237**δ at o2 = Connection gradient**−0.02260254**h1 → o2 Now update this weight Old weight**0.5** → New weight**0.51130127** Subtract η × gradient = **−0.01130127**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(-0.038098237)\times(0.593269992)\approx -0.02260254 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.5-0.5(-0.02260254)\approx 0.51130127 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.596884378**h2 × Receiving delta**−0.038098237**δ at o2 = Connection gradient**−0.022740242**h2 → o2 Now update this weight Old weight**0.55** → New weight**0.561370121** Subtract η × gradient = **−0.011370121**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(-0.038098237)\times(0.596884378)\approx -0.022740242 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.55-0.5(-0.022740242)\approx 0.561370121 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.138498562** × Old outgoing weight**0.4** = Path contribution**0.055399425** Output 2 Output delta**−0.038098237** × Old outgoing weight**0.5** = Path contribution**−0.019049118** ↓ Add every path contribution**0.036350306**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(0.138498562)\approx 0.055399425 $$ **Contribution from output 2.** $$ C_2=(0.5)\times(-0.038098237)\approx -0.019049118 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.036350306 $$ $$ \delta_{h_1} = 0.036350306 \times 0.241300709 = 0.008771355 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.138498562** × Old outgoing weight**0.45** = Path contribution**0.062324353** Output 2 Output delta**−0.038098237** × Old outgoing weight**0.55** = Path contribution**−0.02095403** ↓ Add every path contribution**0.041370323**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.45)\times(0.138498562)\approx 0.062324353 $$ **Contribution from output 2.** $$ C_2=(0.55)\times(-0.038098237)\approx -0.02095403 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.041370323 $$ $$ \delta_{h_2} = 0.041370323 \times 0.240613417 = 0.009954255 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.05**x1 × Receiving delta**0.008771355**δ at h1 = Connection gradient**0.000438568**x1 → h1 Now update this weight Old weight**0.15** → New weight**0.149780716** Subtract η × gradient = **0.000219284**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.008771355)\times(0.05)\approx 0.000438568 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.15-0.5(0.000438568)\approx 0.149780716 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.1**x2 × Receiving delta**0.008771355**δ at h1 = Connection gradient**0.000877135**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199561432** Subtract η × gradient = **0.000438568**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.008771355)\times(0.1)\approx 0.000877135 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.5(0.000877135)\approx 0.199561432 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.05**x1 × Receiving delta**0.009954255**δ at h2 = Connection gradient**0.000497713**x1 → h2 Now update this weight Old weight**0.25** → New weight**0.249751144** Subtract η × gradient = **0.000248856**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.009954255)\times(0.05)\approx 0.000497713 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.25-0.5(0.000497713)\approx 0.249751144 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.1**x2 × Receiving delta**0.009954255**δ at h2 = Connection gradient**0.000995425**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.299502287** Subtract η × gradient = **0.000497712**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.009954255)\times(0.1)\approx 0.000995425 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.5(0.000995425)\approx 0.299502287 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.35-0.5(0.008771355)=0.345614322 $$ $$ b_{h_2,\mathrm{new}}=0.35-0.5(0.009954255)=0.345022872 $$ $$ b_{o_1,\mathrm{new}}=0.6-0.5(0.138498562)=0.530750719 $$ $$ b_{o_2,\mathrm{new}}=0.6-0.5(-0.038098237)=0.619049118 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Why the photos add two terms for h₁:** dL/dh₁ = δo₁w₅ + δo₂w₇ = 0.138498562 × 0.4 + (−0.038098237) × 0.5 = 0.055399425−0.019049118 = 0.036350306. Now multiply by h₁(1−h₁) = 0.241300709 to obtain δh₁ = 0.008771355; multiply by x₁ = 0.05 to obtain dL/dw₁ = 0.000438568. Adding the two output paths is essential. **Memory sequence:** error → output slope → outgoing weight → add paths → hidden slope → input. If you can explain why each factor appears, changed numbers or extra output neurons are much less confusing. **Bias convention:** the photo draws one bias value per layer. If these are literally tied scalars rather than equal initial per-neuron biases, sum the relevant deltas: hidden shared b becomes 0.35−0.5(0.008771355+0.009954255) = 0.340637195; output shared b becomes 0.6−0.5(0.138498562−0.038098237) = 0.549799837. If only weights are requested, do not invent a bias update requirement. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-intro F25 = S12, 3 December 2025, with S13 as a comparison answer key. S18 is another copy of this same paper. All questions are included; its historical choice rule does not remove questions from this practice bank. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · F25 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s0 ## Question **Question (5 marks).** Give four ML versus DL differences with examples. Choose and justify ML or DL for (1) 2,000 short product reviews, (2) labeled financial fraud records, (3) captions for 100,000 labeled natural-scene images. ## Answer — step by step Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. **Choices:** (1) Start with traditional ML using text features because the labeled dataset is small; pretrained text models are a viable extension. (2) Start with a tabular ML classifier such as a tree ensemble; labeled structured features do not automatically require DL. (3) DL is appropriate for learning visual features and generating a word sequence; a CNN encoder with a sequence decoder is a syllabus-aligned design. Compare held-out metrics rather than claiming one method must always win. --- ## TEE · F25 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s1 ## Question **Question (5 marks).** For automated leaf-disease image classification, discuss factors affecting loss selection. For classes Normal, Mild, Severe with target [0,1,0] and predicted probabilities [0.1,0.8,0.1], compute and interpret categorical cross-entropy. ## Answer — step by step ### Given Classes = Normal,Mild,Severe; target = [0,1,0]. Predicted probabilities = [0.1,0.8,0.1]; use natural logarithms. ### To find Calculate categorical cross-entropy and interpret the result. ### Formula — which one and why **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step Select a loss according to the task and output meaning: real-valued regression often uses MSE/MAE, binary classification uses binary cross-entropy, and one-of-K classification uses categorical cross-entropy. Consider class imbalance, noisy labels/outliers, whether outputs are calibrated probabilities, and differentiability for optimization. **Step 1: expand the categorical cross-entropy sum.** $$ L=-[0\ln(0.1)+1\ln(0.8)+0\ln(0.1)] $$ **Step 2: keep the term for the true class.** $$ L=-\ln(0.8) $$ **Step 3: calculate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 4: negate the logarithm.** $$ L\approx0.223143551 $$ **Final answer:** loss ≈ 0.223144 nats. The true class received probability 0.8, so the loss is relatively small; probability 1 would give loss zero. --- ## TEE · F25 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s2 ## Question **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. --- ## TEE · F25 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s3 ## Question **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. --- ## TEE · F25 Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s4 ## Question **Question (10 marks).** A 2 → 2 → 1 sigmoid network has all four input-to-hidden weights 0.5, both hidden-to-output weights 0.25, hidden and output biases 0, and target 1. Perform forward and backward passes to update the x₁ → h₁ weight, h₁ → output weight and output bias at η = 0.5. The paper does not state the numerical inputs or loss. ![F25 Q2a connections with explicitly assumed inputs](https://deep-learning-study-room.vercel.app/assets/figures/net_f25.svg) Diagram: F25 Q2a connections with explicitly assumed inputs. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=1 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.5 + x2 × 0.5; bias 0 | h2 incoming: x1 × 0.5 + x2 × 0.5; bias 0 | o1 incoming: h1 × 0.25 + h2 × 0.25; bias 0 | Use these OLD connections for all gradients in the same pass. F25 Q2a connections with explicitly assumed inputs Tap the diagram to enlarge. ## Answer — step by step ### Given Architecture 2 → 2 → 1; every neuron uses sigmoid. All four input-to-hidden weights = 0.5; both hidden-to-output weights = 0.25. All biases = 0; target = 1; η = 0.5. The paper omits inputs and loss: this worked solution explicitly assumes x = [1,1] and half squared loss. ### To find Calculate the forward pass, then gradients and updates for x₁→h₁, h₁→output and the output bias. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Assumptions:** use x = [1,1] and half squared loss. The paper permits suitable assumptions and its key also introduces assumed inputs. There is no unique numerical result without them. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.5 \times 1) + (0.5 \times 1) + (0) = 1 $$ $$ h_{1}=\frac{1}{1+e^{-(1)}}=0.731058579 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.5 \times 1) + (0.5 \times 1) + (0) = 1 $$ $$ h_{2}=\frac{1}{1+e^{-(1)}}=0.731058579 $$ **Output neuron 1:** $$ z_{o_1} = (0.25 \times 0.731058579) + (0.25 \times 0.731058579) + (0) = 0.365529289 $$ $$ o_{1}=\frac{1}{1+e^{-(0.365529289)}}=0.590378257 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.409621743]. Squared errors = [0.167789972]. Half-sum squared loss = 0.083894986; mean squared error = 0.167789972. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.590378257-1) \times 0.241831771 = -0.099059551 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.731058579**h1 × Receiving delta**−0.099059551**δ at o1 = Connection gradient**−0.072418335**h1 → o1 Now update this weight Old weight**0.25** → New weight**0.286209167** Subtract η × gradient = **−0.036209168**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.099059551)\times(0.731058579)\approx -0.072418335 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.25-0.5(-0.072418335)\approx 0.286209167 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.731058579**h2 × Receiving delta**−0.099059551**δ at o1 = Connection gradient**−0.072418335**h2 → o1 Now update this weight Old weight**0.25** → New weight**0.286209167** Subtract η × gradient = **−0.036209168**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.099059551)\times(0.731058579)\approx -0.072418335 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.25-0.5(-0.072418335)\approx 0.286209167 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.099059551** × Old outgoing weight**0.25** = Path contribution**−0.024764888** ↓ Add every path contribution**−0.024764888**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.25)\times(-0.099059551)\approx -0.024764888 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.024764888 $$ $$ \delta_{h_1} = -0.024764888 \times 0.196611933 = -0.004869072 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.099059551** × Old outgoing weight**0.25** = Path contribution**−0.024764888** ↓ Add every path contribution**−0.024764888**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.25)\times(-0.099059551)\approx -0.024764888 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.024764888 $$ $$ \delta_{h_2} = -0.024764888 \times 0.196611933 = -0.004869072 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.004869072**δ at h1 = Connection gradient**−0.004869072**x1 → h1 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**1**x2 × Receiving delta**−0.004869072**δ at h1 = Connection gradient**−0.004869072**x2 → h1 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.004869072**δ at h2 = Connection gradient**−0.004869072**x1 → h2 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**1**x2 × Receiving delta**−0.004869072**δ at h2 = Connection gradient**−0.004869072**x2 → h2 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0-0.5(-0.004869072)=0.002434536 $$ $$ b_{h_2,\mathrm{new}}=0-0.5(-0.004869072)=0.002434536 $$ $$ b_{o_1,\mathrm{new}}=0-0.5(-0.099059551)=0.049529775 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Requested values:** input weight **0.502434536**, output weight **0.286209167**, output bias **0.049529775**. Other supplied inputs produce different correct conditional results. --- ## TEE · F25 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s5 ## Question **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs. --- ## TEE · F25 Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s6 ## Question **Question (10 marks).** Train one epoch of a single perceptron with η = 1, initial weights and bias 0, threshold 0, on [−1,1,−1] → 0, [−1,1,1] → 1, [1,−1,1] → 1, [−1,1,1] → 1. Then map biological-neuron components to artificial-neuron components. ## Answer — step by step ### Given Initial weights and bias = 0; η = 1; threshold = 0. Samples in order: [−1,1,−1]→0; [−1,1,1]→1; [1,−1,1]→1; [−1,1,1]→1. Use output 1 when net > 0 and 0 otherwise, matching the supplied key; one epoch. ### To find Process each sample in order, carrying updated weights forward, and report final weights/bias. ### Formula — which one and why **Perceptron learning rule.** Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters. t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question. $$ z=\sum_iw_ix_i+b $$ $$ w_i^{\mathrm{new}}=w_i+\eta(t-o)x_i $$ $$ b^{\mathrm{new}}=b+\eta(t-o) $$ **Yaad rakhne ka cue:** Prediction nikalo; target−prediction se correction nikalo. ### Solution — step by step **State the threshold rule.** Use binary output 1 if z>0 and 0 otherwise, matching the associated key. Update Δw = η(t−o)x and Δb = η(t−o), with η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly. ### Epoch 1, sample 1 **Step 1: copy this sample and target.** x = [-1, 1, -1], target = 0. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, -1] = [0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [0, 0, 0], new b = 0. ### Epoch 1, sample 2 **Step 1: copy this sample and target.** x = [-1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (0 × 1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [-1, 1, 1] = [-1, 1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [-1, 1, 1], new b = 1. ### Epoch 1, sample 3 **Step 1: copy this sample and target.** x = [1, -1, 1], target = 1. **Step 2: calculate the current net.** z = (-1 × 1) + (1 × -1) + (1 × 1) + (1) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [1, -1, 1] = [1, -1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [0, 0, 2], new b = 2. ### Epoch 1, sample 4 **Step 1: copy this sample and target.** x = [-1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (2 × 1) + (2) = 4 **Step 3: apply the threshold.** output = 1. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, 1] = [0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [0, 0, 2], new b = 2. **Final after exactly the requested epochs:** w = **[0, 0, 2]**, b = **2**. A finite number of requested epochs does not itself prove convergence or correct classification of every sample. Biological dendrites receive signals, synapses influence connection strength, the cell body integrates information, and an axon carries an output signal. In the artificial analogy, inputs represent received signals, weights represent connection strengths, the weighted sum plus bias performs aggregation, and an activation function produces an output. The bias shifts the firing threshold. This is a simplified computational analogy, not a complete biological model. ![The artificial neuron multiplies each input by a weight then adds bias and activates](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: The artificial neuron multiplies each input by a weight then adds bias and activates. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate The artificial neuron multiplies each input by a weight then adds bias and activates Tap the diagram to enlarge. --- ## TEE · F25 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s7 ## Question **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. --- ## TEE · F25 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s8 ## Question **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters. ## Answer — step by step ### Given N = 1,000; mini-batch size = 100. J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9. Use signed velocity. ### To find Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Step 1: count full-batch updates.** All 1,000 samples contribute to one update: 1 per epoch. **Step 2: count stochastic updates.** One update per sample: 1,000 per epoch. **Step 3: count mini-batches.** $$ 1000/100=10 $$ Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ. ### Momentum iteration 1 **Step 1: evaluate the gradient at the current parameter.** $$ g_1=1 $$ **Step 2: retain the previous displacement.** $$ 0.9(0)=0 $$ **Step 3: scale the current gradient.** $$ 0.1(1)=0.1 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_1=0-0.1=-0.1 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_1=1+(-0.1)=0.9 $$ ### Momentum iteration 2 **Step 1: evaluate the gradient at the current parameter.** $$ g_2=0.9 $$ **Step 2: retain the previous displacement.** $$ 0.9(-0.1)=-0.09 $$ **Step 3: scale the current gradient.** $$ 0.1(0.9)=0.09 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_2=-0.09-0.09=-0.18 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_2=0.9+(-0.18)=0.72 $$ **Hyperparameters:** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update. --- ## TEE · F25 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s9 ## Question **Question (10 marks).** Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1]. ## Answer — step by step ### Given Forget-gate outputs f = [0.1,0.9]; old cell = [2,1]. No input gate or candidate is supplied. ### To find Explain forgetting/retention and calculate the retained old-cell part only. ### Formula — which one and why **LSTM retained old memory.** Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell. f = activated forget gate; C\_old = previous cell state; ⊙ means multiply matching entries. $$ C_{\mathrm{retained}}=f\odot C_{\mathrm{old}} $$ **Yaad rakhne ka cue:** Forget gate × old cell = bacha hua purana hissa. ### Solution — step by step Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph. 1. Initialize the hidden state and run the forward recurrence from the first input to the last. 2. Compute the specified output loss, at the final step or across several steps. 3. Start from the last loss and propagate derivatives backwards through output and hidden operations. 4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states. 5. Sum all uses of each shared weight before applying the optimizer update. $$ \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T} $$ The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step. ![BPTT follows this unrolled network backwards and sums gradients of shared weights](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: BPTT follows this unrolled network backwards and sums gradients of shared weights. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge. The plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path. **Step 1: retain the first old-cell component.** $$ 0.1(2)=0.2 $$ **Step 2: retain the second old-cell component.** $$ 0.9(1)=0.9 $$ **Step 3: assemble the retained old-cell contribution.** $$ f\odot C_{\mathrm{old}}=[0.2,0.9] $$ **Final answer:** [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated. --- ## TEE · F25 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s10 ## Question **Question (10 marks).** State the purpose of each LSTM gate. For “Artificial Intelligence has revolutionized every domain. The work force will have to update themselves to remain relevant,” explain a context change. Calculate C and h given i = 0.7, f = 0.4, o = 0.8, candidate = 0.9, Cprevious = 0.5. Why is LSTM more effective than a simple RNN for language modeling? ## Answer — step by step ### Given Activated gate values: i = 0.7,f = 0.4,o = 0.8. Candidate = 0.9; old cell = 0.5. ### To find Explain the context change in the question; calculate the new cell and hidden state. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. The second sentence shifts attention toward the workforce while retaining useful prior AI context. Input gates can admit the new subject; forget gates can reduce obsolete details. **Step 1: retain old memory.** $$ fC_{old}=0.4(0.5)=0.2 $$ **Step 2: admit candidate content.** $$ i\widetilde C=0.7(0.9)=0.63 $$ **Step 3: add the two contributions.** $$ C=0.2+0.63=0.83 $$ **Step 4: apply tanh to the updated cell.** $$ \tanh(0.83)\approx0.680476006 $$ **Step 5: multiply by the output gate.** $$ h=0.8(0.680476006)\approx0.544380805 $$ **Final answer:** C = 0.83 and h ≈ 0.544380805. The supplied gates are already activated; do not apply sigmoid again. --- ## TEE · F25 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s11 ## Question **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD. ## Answer — step by step AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise: $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change. Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864. Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average. --- ## TEE · F25 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s12 ## Question **Question (10 marks).** Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. Given matrices Input A Input A | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Stride 2; P = 0 (no padding). ## Answer — step by step ### Given Input 128 × 128 × 3. Conv K = 3, filters = 16, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 3, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Dense 240 units. Dense 120 units. Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. ### To find Find every layer shape, its parameter count and the network total. Find every convolution output cell. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**128 × 128 × 3**height × width × channels → Convolution**16 filters · 3 × 3 · S = 1 · P = 0** → Output shape**126 × 126 × 16**height × width × channels Weights**432** + Biases**16** = Trainable parameters**448** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **128 × 128 × 3**. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{128+2(0)-3}{1}\right\rfloor+1=126 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **126 × 126 × 16**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times3\times16=432 $$ $$ N_{\mathrm{biases}}=16 $$ $$ P_1=432+16=448 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**126 × 126 × 16**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**63 × 63 × 16**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **126 × 126 × 16**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{126-2}{2}\right\rfloor+1=63 $$ Pooling acts independently on each channel. The output is **63 × 63 × 16**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**63 × 63 × 16**height × width × channels → Convolution**32 filters · 3 × 3 · S = 1 · P = 0** → Output shape**61 × 61 × 32**height × width × channels Weights**4,608** + Biases**32** = Trainable parameters**4,640** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **63 × 63 × 16**. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{63+2(0)-3}{1}\right\rfloor+1=61 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **61 × 61 × 32**. **3. Count trainable values.** Each filter receives all 16 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times16\times32=4{,}608 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_3=4{,}608+32=4{,}640 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**61 × 61 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**30 × 30 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **61 × 61 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{61-2}{2}\right\rfloor+1=30 $$ Pooling acts independently on each channel. The output is **30 × 30 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 5 — flatten Layer 5: what changes and what is learned Input shape**30 × 30 × 32**height × width × channels → Flatten**Rearrange activations** → Output shape**28,800**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **30 × 30 × 32**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=30\times30\times32=28{,}800 $$ Output: a vector with **28,800 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 6 — fully connected layer Layer 6: what changes and what is learned Input shape**28,800**input features → Dense layer**240 neurons** → Output shape**240**output features Weights**6,912,000** + Biases**240** = Trainable parameters**6,912,240** Every output neuron receives every input feature, with one bias per output neuron. Input: **28,800 features**. Output: **240 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=28{,}800\times240=6{,}912{,}000 $$ $$ N_{\mathrm{biases}}=240 $$ $$ P_6=6{,}912{,}000+240=6{,}912{,}240 $$ ### Layer 7 — fully connected layer Layer 7: what changes and what is learned Input shape**240**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**28,800** + Biases**120** = Trainable parameters**28,920** Every output neuron receives every input feature, with one bias per output neuron. Input: **240 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=240\times120=28{,}800 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_7=28{,}800+120=28{,}920 $$ ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**120**input features → Dense layer**5 neurons** → Output shape**5**output features Weights**600** + Biases**5** = Trainable parameters**605** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **5 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times5=600 $$ $$ N_{\mathrm{biases}}=5 $$ $$ P_8=600+5=605 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{128+2 \times 0-3}{1}\right\rfloor+1$ | 126 x 126 x 16 | 432 | 16 | 448 | | 2 pool | $\left\lfloor\frac{126-2}{2}\right\rfloor+1$ | 63 x 63 x 16 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{63+2 \times 0-3}{1}\right\rfloor+1$ | 61 x 61 x 32 | 4,608 | 32 | 4,640 | | 4 pool | $\left\lfloor\frac{61-2}{2}\right\rfloor+1$ | 30 x 30 x 32 | 0 | 0 | 0 | | 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 | | 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 | | 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 | | 8 dense | 120 → 5 | 5 | 600 | 5 | 605 | **Parameter substitutions, in layer order:** Layer 1: (3 x 3 x 3 + 1) x 16 = 448. Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640. Layer 6: (28800 + 1) x 240 = 6,912,240. Layer 7: (240 + 1) x 120 = 28,920. Layer 8: (120 + 1) x 5 = 605. **Total:** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. **Separate convolution calculation:** **Step 1 shape:** $\left\lfloor\frac{5+2 \times 0-3}{2}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −2 | | 4 | 0 | −4 | | 3 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−2) = 1** Row 2**4 + 0 + (−4) = 0** Row 3**3 + 0 + (−1) = 2** **1 + 0 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(1)\times(0)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(2)\times(-2)=0 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(0)+(2)=3 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 2 | 0 | −1 | | 4 | 0 | 0 | | 1 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 + (−1) = 1** Row 2**4 + 0 + 0 = 4** Row 3**1 + 0 + (−1) = 0** **1 + 4 + 0 = 5** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(2)\times(0)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(0)\times(-2)=4 $$ **Window row 3.** $$ R_3=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(1)+(4)+(0)=5 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −1 | | 0 | 0 | 0 | | 1 | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−1) = 2** Row 2**0 + 0 + 0 = 0** Row 3**1 + 0 + 0 = 1** **2 + 0 + 1 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Window row 2.** $$ R_2=(0)\times(2)+(3)\times(0)+(0)\times(-2)=0 $$ **Window row 3.** $$ R_3=(1)\times(1)+(1)\times(0)+(0)\times(-1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(2)+(0)+(1)=3 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 1 | 0 | −1 | | 0 | 0 | −4 | | 0 | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 + (−1) = 0** Row 2**0 + 0 + (−4) = −4** Row 3**0 + 0 + (−3) = −3** **0 + (−4) + (−3) = −7** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0)\times(2)+(1)\times(0)+(2)\times(-2)=-4 $$ **Window row 3.** $$ R_3=(0)\times(1)+(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(0)+(-4)+(-3)=-7 $$ **Final output:** $\begin{bmatrix}3 & 5\\3 & -7\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. --- ## TEE · F25 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s13 ## Question **Question (10 marks).** Explain activations and the role of nonlinearity. Compare sigmoid, tanh, ReLU and Leaky ReLU including advantages and disadvantages. ## Answer — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. --- ## TEE · F25 Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s14 ## Question **Question (10 marks).** Explain feedforward information flow, weights, biases and activations, then calculate the output. Input x = [1, 0]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_412d332265.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 | W₂ | 0.5 | 0.6 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 0]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. ### To find Find all preactivations, hidden activations and the final output. Explain the role of weights, biases and activations in this forward pass. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step A feedforward network transforms inputs layer by layer with no recurrent feedback. Weights scale contributions; bias shifts the preactivation; activation supplies the nonlinear transformation. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.4 \times 0) + (0.1) = 0.3 $$ $$ h_{1} = \max(0,0.3) = 0.3 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (0.7 \times 0) + (0.2) = 0.5 $$ $$ h_{2} = \max(0,0.5) = 0.5 $$ **Output neuron 1:** $$ z_{o_1} = (0.5 \times 0.3) + (0.6 \times 0.5) + (0.1) = 0.55 $$ $$ o_{1} = \max(0,0.55) = 0.55 $$ **Matrix-orientation note:** this solution uses column input and rows as receiving neurons, giving **0.55**. The historical key uses the transpose interpretation for W₁ and obtains 0.61. If the problem does not define its orientation, state your convention; do not silently swap it during calculation. Under the alternative: hidden values = [0.3,0.6], output = 0.5 × 0.3+0.6 × 0.6+0.1 = 0.61. --- ## TEE · F25 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c11-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/11/tee-c11-s15 ## Question **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages. ## Answer — step by step A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification. ![One filter is reused at successive spatial positions](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: One filter is reused at successive spatial positions. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. One filter is reused at successive spatial positions Tap the diagram to enlarge. Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities. Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation. For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-intro R26 = S09, 10 February 2026. S08 is the associated answer key; S10/S11 are duplicate-content copies in another program folder. One subpart asks a topic not explicitly listed in the current syllabus and is flagged below. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · R26 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s0 ## Question **Question (5 marks).** Describe two situations favoring DL over traditional ML. Choose ML or DL for house prices from 1,000 numerical records, real-time facial-emotion video recognition, and translation from millions of sentence pairs. ## Answer — step by step Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. **Choices:** house prices: start with tabular ML because the features are structured and data are limited. Video emotion: DL can learn visual spatial features and temporal patterns. Large parallel-text translation: DL can learn representations and sequence dependencies. Two situations particularly suited to learned deep representations are complex image recognition and large-scale speech/language tasks; compare validation results instead of assuming an unconditional advantage. --- ## TEE · R26 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s1 ## Question **Question (5 marks).** Explain how activations affect gradients, convergence and performance. State preferred contexts for sigmoid, tanh, ReLU and Leaky ReLU. ## Answer — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. --- ## TEE · R26 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s2 ## Question **Question (5 marks).** Explain L1 and L2 regularization. Apply 50% dropout to outputs [0.8,−0.4,0.6,0.2] using an explicitly assumed mask. ## Answer — step by step ### Given Activations = [0.8,−0.4,0.6,0.2]; dropout rate = 50%. The worked example assumes mask [1,0,1,0] and uses inverted dropout. No numerical weights or λ are supplied for the regularization explanation. ### To find Explain L1/L2 and calculate post-dropout activations under the stated mask. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** This answer uses λ/2 times the sum of squared weights. Differentiating a square supplies a factor of 2, which cancels the one-half, so the added gradient is λw. Use this same convention in the explanation below. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\frac{\lambda}{2}\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+\lambda w_i $$ **Yaad rakhne ka cue:** L2 mein half diya hai: square ka 2 cancel, gradient mein lambda × weight. **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. ### Numerical with one explicitly chosen mask **Step 1: convert dropout probability to keep probability.** $$ q=1-0.5=0.5 $$ **Step 2: state a possible mask.** Choose [1,0,1,0]; 1 means keep and 0 means drop. The question allows an assumed mask; others give different realizations. **Step 3: multiply each activation by its mask entry.** $$ [0.8(1),(-0.4)(0),0.6(1),0.2(0)]=[0.8,0,0.6,0] $$ **Step 4: divide the kept values by q for inverted dropout.** $$ 0.8/0.5=1.6 $$ $$ 0.6/0.5=1.2 $$ **Step 5: assemble the training output.** $$ a_{\mathrm{train}}=[1.6,0,1.2,0] $$ **Final answer:** simple masking gives [0.8,0,0.6,0]; inverted training dropout gives [1.6,0,1.2,0]. At inference inverted dropout is disabled. --- ## TEE · R26 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s3 ## Question **Question (5 marks).** Explain an RNN for positive, negative or neutral e-commerce review sentiment. Why are recurrence and word order relevant? ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. For this task select a many-to-one output: tokenize and embed the full review, update the recurrent state across its words, and classify the final sequence representation with a three-unit softmax. Train using one-hot sentiment labels and categorical cross-entropy. “Not worth buying” and “worth buying” illustrate why negation and order matter. --- ## TEE · R26 Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s4 ## Question **Question (10 marks).** Use the sigmoid network Input x = [0.25, 0.7]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [1, 1]; W₂ is shown below; b₂ = [1, 1]. Hidden and output activations are sigmoid. Target y = [0.7, 0.3]. Learning rate η = 1. Here w₁ = x₁ → h₁, w₅ = h₁ → o₁ and w₇ = h₁ → o₂. Run forward and backward passes and update w₁, w₅, w₇. The paper gives no explicit loss formula. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_23b4df4e5b.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.15 | 0.28 | 0.3 | W₂ | 0.1 | 0.24 | 0.12 | 0.21 Given matrices for this question Tap the diagram to enlarge. ![R26 Q2a weights and both target outputs](https://deep-learning-study-room.vercel.app/assets/figures/net_r26.svg) Diagram: R26 Q2a weights and both target outputs. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.25 | x2=0.7 | h1 | h2 | o1 | target 0.7 | o2 | target 0.3 | h1 incoming: x1 × 0.2 + x2 × 0.15; bias 1 | h2 incoming: x1 × 0.28 + x2 × 0.3; bias 1 | o1 incoming: h1 × 0.1 + h2 × 0.24; bias 1 | o2 incoming: h1 × 0.12 + h2 × 0.21; bias 1 | Use these OLD connections for all gradients in the same pass. R26 Q2a weights and both target outputs Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.25,0.7]; W₁ and W₂ are printed with the question. Hidden biases = [1,1]; output biases = [1,1]; all activations sigmoid. Target = [0.7,0.3]; η = 1. The paper omits the loss; this solution explicitly assumes half summed squared loss. ### To find Calculate forward values and update only w₁ = x₁→h₁, w₅ = h₁→o₁ and w₇ = h₁→o₂. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 0.25) + (0.15 \times 0.7) + (1) = 1.155 $$ $$ h_{1}=\frac{1}{1+e^{-(1.155)}}=0.760423002 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.28 \times 0.25) + (0.3 \times 0.7) + (1) = 1.28 $$ $$ h_{2}=\frac{1}{1+e^{-(1.28)}}=0.782449776 $$ **Output neuron 1:** $$ z_{o_1} = (0.1 \times 0.760423002) + (0.24 \times 0.782449776) + (1) = 1.263830247 $$ $$ o_{1}=\frac{1}{1+e^{-(1.263830247)}}=0.779684759 $$ **Output neuron 2:** $$ z_{o_2} = (0.12 \times 0.760423002) + (0.21 \times 0.782449776) + (1) = 1.255565213 $$ $$ o_{2}=\frac{1}{1+e^{-(1.255565213)}}=0.778261739 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [0.079684759, 0.478261739]. Squared errors = [0.006349661, 0.228734291]. Half-sum squared loss = 0.117541976; mean squared error = 0.117541976. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.779684759-0.7) \times 0.171776436 = 0.013687964 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.778261739-0.3) \times 0.172570404 = 0.082533822 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.760423002**h1 × Receiving delta**0.013687964**δ at o1 = Connection gradient**0.010408643**h1 → o1 Now update this weight Old weight**0.1** → New weight**0.089591357** Subtract η × gradient = **0.010408643**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(0.013687964)\times(0.760423002)\approx 0.010408643 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.1-1(0.010408643)\approx 0.089591357 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.782449776**h2 × Receiving delta**0.013687964**δ at o1 = Connection gradient**0.010710144**h2 → o1 Now update this weight Old weight**0.24** → New weight**0.229289856** Subtract η × gradient = **0.010710144**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(0.013687964)\times(0.782449776)\approx 0.010710144 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.24-1(0.010710144)\approx 0.229289856 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.760423002**h1 × Receiving delta**0.082533822**δ at o2 = Connection gradient**0.062760617**h1 → o2 Now update this weight Old weight**0.12** → New weight**0.057239383** Subtract η × gradient = **0.062760617**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.082533822)\times(0.760423002)\approx 0.062760617 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.12-1(0.062760617)\approx 0.057239383 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.782449776**h2 × Receiving delta**0.082533822**δ at o2 = Connection gradient**0.06457857**h2 → o2 Now update this weight Old weight**0.21** → New weight**0.14542143** Subtract η × gradient = **0.06457857**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.082533822)\times(0.782449776)\approx 0.06457857 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.21-1(0.06457857)\approx 0.14542143 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.013687964** × Old outgoing weight**0.1** = Path contribution**0.001368796** Output 2 Output delta**0.082533822** × Old outgoing weight**0.12** = Path contribution**0.009904059** ↓ Add every path contribution**0.011272855**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.1)\times(0.013687964)\approx 0.001368796 $$ **Contribution from output 2.** $$ C_2=(0.12)\times(0.082533822)\approx 0.009904059 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.011272855 $$ $$ \delta_{h_1} = 0.011272855 \times 0.18217986 = 0.002053687 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**0.013687964** × Old outgoing weight**0.24** = Path contribution**0.003285111** Output 2 Output delta**0.082533822** × Old outgoing weight**0.21** = Path contribution**0.017332103** ↓ Add every path contribution**0.020617214**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.24)\times(0.013687964)\approx 0.003285111 $$ **Contribution from output 2.** $$ C_2=(0.21)\times(0.082533822)\approx 0.017332103 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.020617214 $$ $$ \delta_{h_2} = 0.020617214 \times 0.170222124 = 0.003509506 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.25**x1 × Receiving delta**0.002053687**δ at h1 = Connection gradient**0.000513422**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.199486578** Subtract η × gradient = **0.000513422**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.002053687)\times(0.25)\approx 0.000513422 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-1(0.000513422)\approx 0.199486578 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.7**x2 × Receiving delta**0.002053687**δ at h1 = Connection gradient**0.001437581**x2 → h1 Now update this weight Old weight**0.15** → New weight**0.148562419** Subtract η × gradient = **0.001437581**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.002053687)\times(0.7)\approx 0.001437581 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.15-1(0.001437581)\approx 0.148562419 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.25**x1 × Receiving delta**0.003509506**δ at h2 = Connection gradient**0.000877376**x1 → h2 Now update this weight Old weight**0.28** → New weight**0.279122624** Subtract η × gradient = **0.000877376**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.003509506)\times(0.25)\approx 0.000877376 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.28-1(0.000877376)\approx 0.279122624 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.7**x2 × Receiving delta**0.003509506**δ at h2 = Connection gradient**0.002456654**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.297543346** Subtract η × gradient = **0.002456654**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.003509506)\times(0.7)\approx 0.002456654 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-1(0.002456654)\approx 0.297543346 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=1-1(0.002053687)=0.997946313 $$ $$ b_{h_2,\mathrm{new}}=1-1(0.003509506)=0.996490494 $$ $$ b_{o_1,\mathrm{new}}=1-1(0.013687964)=0.986312036 $$ $$ b_{o_2,\mathrm{new}}=1-1(0.082533822)=0.917466178 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Requested updates:** w₁ = **0.199486578**, w₅ = **0.089591357**, w₇ = **0.057239383**. For two outputs, this chosen half-sum equals their MSE; an unaveraged unhalved sum would double the gradients. --- ## TEE · R26 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s5 ## Question **Question (10 marks).** Explain AlexNet regularization and training improvements; explain Inception multiscale features and 1 × 1 convolutions; recommend an architecture for large-scale image classification with justification. ## Answer — step by step AlexNet uses a sequence of convolutional layers followed by fully connected classification layers. Its influential design used ReLU activations, max pooling, dropout in the fully connected part, data augmentation and GPU training. ReLU's positive derivative supports learning compared with heavily saturated sigmoid hidden units. Augmentation supplies varied training views; dropout reduces co-adaptation; GPU computation makes a large network practical to train. ![The common classroom AlexNet layout contains five convolutions and three dense layers](https://deep-learning-study-room.vercel.app/assets/figures/alexnet.svg) Diagram: The common classroom AlexNet layout contains five convolutions and three dense layers. Labels: Input | 227×227×3 | Conv1 | 11×11, 96 | Pool | 3×3, s2 | Conv2 | 5×5, 256 | Pool | 3×3, s2 | Conv3 | 3×3, 384 | Conv4 | 3×3, 384 | Conv5 | 3×3, 256 | Pool + flat | 9216 | Dense | 4096 | Dense | 4096 | Output | 1000 | Read rows in order. Apply nonlinearities between learned layers. The common classroom AlexNet layout contains five convolutions and three dense layers Tap the diagram to enlarge. The common 227-pixel classroom layout is: Conv 11 stride 4 with 96 filters; pool 3 stride 2; Conv 5 padding 2 with 256 filters; pool 3 stride 2; three Conv 3 padding 1 layers with 384,384,256 filters; pool 3 stride 2; dense 4096; dense 4096; output 1000. Nonlinearities are applied between learned layers. Exact padding, grouping and input conventions must follow the question: the original model used channel groups in some layers, so parameter counts differ from a fully connected-channel simplification. AlexNet follows a mainly sequential feature-extraction path. An Inception module processes several spatial scales in parallel and concatenates them, often using 1 × 1 bottlenecks to reduce cost. Neither architecture is universally best; the task, resource budget and validation performance determine the choice. An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. **Recommendation:** an Inception-family model is a defensible choice when varied object scales and computation efficiency matter, because parallel scales with bottlenecks provide diverse features at controlled cost. Compare held-out accuracy, latency and memory with available alternatives. This is a justified choice, not a claim that Inception always beats AlexNet on every dataset. --- ## TEE · R26 Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s6 ## Question **Question (10 marks).** Train a perceptron for one epoch, η = 1, initial weights and bias 0, threshold 0. Samples: [−1,1,−1] → −1; [1,1,1] → 1; [−1,−1,1] → −1; [−1,1,1] → −1. Discuss limitations with examples. ## Answer — step by step ### Given Train a perceptron for one epoch, η = 1, initial weights and bias 0, threshold 0. Samples: [−1,1,−1] → −1. [1,1,1] → 1. [−1,−1,1] → −1. [−1,1,1] → −1. ### To find Process samples in order for the stated epochs and report the final weights and bias. ### Formula — which one and why **Target-based perceptron correction.** This worked convention changes parameters only on a wrong prediction. Use the target itself as the signed multiplier; it differs from the binary error rule. t is the bipolar target; η is the learning rate. Output is +1 above zero, −1 below zero and 0 at exactly zero. $$ \Delta w=\eta t x,\quad\Delta b=\eta t\quad\text{on a mismatch} $$ **Yaad rakhne ka cue:** Is convention mein galat hua toh target × input se correction. ### Solution — step by step **State the threshold rule.** Use output +1 if z>0, −1 if z<0, and 0 at z = 0. On a mismatch use Δw = η tx and Δb = ηt; on a match make no update. η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly. ### Epoch 1, sample 1 **Step 1: copy this sample and target.** x = [-1, 1, -1], target = -1. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = -1 **Step 5: calculate all weight changes.** Δw = -1 × [-1, 1, -1] = [1, -1, 1] **Step 6: calculate the bias change.** Δb = -1. **Step 7: carry the updated parameters forward.** New w = [1, -1, 1], new b = -1. ### Epoch 1, sample 2 **Step 1: copy this sample and target.** x = [1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (1 × 1) + (-1 × 1) + (1 × 1) + (-1) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [1, 1, 1] = [1, 1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [2, 0, 2], new b = 0. ### Epoch 1, sample 3 **Step 1: copy this sample and target.** x = [-1, -1, 1], target = -1. **Step 2: calculate the current net.** z = (2 × -1) + (0 × -1) + (2 × 1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = -1 **Step 5: calculate all weight changes.** Δw = -1 × [-1, -1, 1] = [1, 1, -1] **Step 6: calculate the bias change.** Δb = -1. **Step 7: carry the updated parameters forward.** New w = [3, 1, 1], new b = -1. ### Epoch 1, sample 4 **Step 1: copy this sample and target.** x = [-1, 1, 1], target = -1. **Step 2: calculate the current net.** z = (3 × -1) + (1 × 1) + (1 × 1) + (-1) = -2 **Step 3: apply the threshold.** output = -1. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, 1] = [0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [3, 1, 1], new b = -1. **Final after exactly the requested epochs:** w = **[3, 1, 1]**, b = **-1**. A finite number of requested epochs does not itself prove convergence or correct classification of every sample. A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator. ![Opposite corners for XOR cannot be split by one straight line](https://deep-learning-study-room.vercel.app/assets/figures/xor.svg) Diagram: Opposite corners for XOR cannot be split by one straight line. Labels: x1 | x2 | 0 | 1 | 1 | 0 | XOR labels occupy opposite corners | 0 | 1 | 0 | 1 | One straight decision boundary cannot separate the two classes. Opposite corners for XOR cannot be split by one straight line Tap the diagram to enlarge. For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1. For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR. A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map. **Key correction:** recomputing each net with the latest weights gives net values 0,0,0,−2 and final w = [3,1,1], b = −1 under the stated bipolar rule. The supplied answer-key table contains inconsistent intermediate arithmetic. --- ## TEE · R26 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s7 ## Question **Question (10 marks).** How should CNN filter counts be selected? Explain effects on feature extraction, dimensionality, computation/memory, overfitting and hierarchical learning. ## Answer — step by step Choose filter counts as architecture hyperparameters using task complexity, available data, validation performance and resource limits. There is no universal correct count. 1. More filters can represent a wider variety of learned features at a layer; redundant filters do not automatically improve accuracy. 2. Cout filters produce Cout output channels; spatial height/width still follow kernel, padding and stride. 3. A K × K layer uses K²CinCout weights plus Cout biases. More output channels increase current activations and the input-channel cost of the next convolution. 4. Extra capacity can overfit limited data, so use validation, augmentation, transfer learning or regularization. Too few channels may underfit. 5. Later layers often use more channels after spatial downsampling, allowing complex feature combinations at a smaller spatial size. This is a common design choice, not a compulsory rule. Example: a 3 × 3 layer with 16 input channels has (9 × 16+1) × 32 = 4,640 parameters for 32 filters, versus 9,280 for 64 filters. Output-channel count doubles while the same spatial-size formula applies. --- ## TEE · R26 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s8 ## Question **Question (10 marks).** Compare batch GD and SGD for loss oscillations with a graph, memory and large datasets. Discuss learning rate extremes. For J(θ) = 0.7 θ³, θ₀ = 1, η = 0.1, perform two iterations. ## Answer — step by step ### Given J(θ) = 0.7θ³; θ₀ = 1; η = 0.1. ### To find Calculate the derivative and perform two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable. ![Illustrative full batch and stochastic traces distinguish smoothness from progress](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative full batch and stochastic traces distinguish smoothness from progress. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative full batch and stochastic traces distinguish smoothness from progress Tap the diagram to enlarge. Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory. Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories. Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD. **Step 1: differentiate the objective before substituting.** $$ J(\theta)=0.7\theta^3 $$ $$ g(\theta)=0.7(3)\theta^2=2.1\theta^2 $$ **Step 2: evaluate gradient at the original parameter.** $$ g_1=2.1(1)^2=2.1 $$ **Step 3: scale the first gradient.** $$ 0.1(2.1)=0.21 $$ **Step 4: subtract to obtain the first update.** $$ \theta_1=1-0.21=0.79 $$ **Step 5: square the changed parameter.** $$ 0.79^2=0.6241 $$ **Step 6: recalculate the gradient.** $$ g_2=2.1(0.6241)=1.31061 $$ **Step 7: scale the second gradient.** $$ 0.1(1.31061)=0.131061 $$ **Step 8: subtract from the first updated parameter.** $$ \theta_2=0.79-0.131061=0.658939 $$ **Final answer:** θ₁ = 0.79; θ₂ = 0.658939. No individual samples are supplied, so these are the stated scalar gradient updates. The cubic is unbounded below; two steps are not a convergence proof. --- ## TEE · R26 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s9 ## Question **Question.** Compare LSTM and GRU gates and architecture. Explain the potential computational efficiency of GRU and how its gates can reduce the effect of irrelevant tokens. **Syllabus-aligned extract:** source R26 Q4b, parts i–iii. Marks for these retained parts are not separately stated. ## Answer — step by step A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate: $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ ![Reset controls candidate context and update controls the old new mixture](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: Reset controls candidate context and update controls the old new mixture. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. Reset controls candidate context and update controls the old new mixture Tap the diagram to enlarge. The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate. Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN. --- ## TEE · R26 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s10 ## Question **Question (10 marks).** Explain feedforward limitations for sequential context and how recurrence helps. Calculate h using tanh for Wxh is shown below, Whh is shown below, x = [0,1], hprevious = [0,1], zero bias. Explain next-word probabilities. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_deac39f14c.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Wxh | 0.6 | 0.4 | 0.3 | 0.7 | Whh | 0.5 | 0.3 | 0.2 | 0.4 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Wxh and Whh are printed in the question; use their receiving-neuron rows. Current input = [0,1]; previous hidden state = [0,1]; bias = 0. Activation = tanh. ### To find Calculate both entries of the new hidden state. Output-layer weights are not supplied. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. **Step 1: multiply the input vector into row 1.** $$ 0.6(0)+0.4(1)=0+0.4=0.4 $$ **Step 2: multiply the input vector into row 2.** $$ 0.3(0)+0.7(1)=0+0.7=0.7 $$ **Step 3: multiply previous state into recurrent row 1.** $$ 0.5(0)+0.3(1)=0+0.3=0.3 $$ **Step 4: multiply previous state into recurrent row 2.** $$ 0.2(0)+0.4(1)=0+0.4=0.4 $$ **Step 5: combine contributions for hidden unit 1.** $$ a_1=0.4+0.3+0=0.7 $$ **Step 6: combine contributions for hidden unit 2.** $$ a_2=0.7+0.4+0=1.1 $$ **Step 7: apply tanh separately to each complete sum.** $$ h_1=\tanh(0.7)\approx0.604367777 $$ $$ h_2=\tanh(1.1)\approx0.800499022 $$ **Final answer:** h ≈ [0.604367777,0.800499022]. The absent vocabulary output weights prevent calculating a numerical next-word probability list. --- ## TEE · R26 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s11 ## Question **Question (10 marks).** Explain RMSProp, its update rule and accumulated squared gradients. Give advantages and disadvantages relative to SGD. ## Answer — step by step RMSProp keeps an exponentially weighted average of squared gradients, instead of AdaGrad's ever-growing sum. For each parameter, initialize s₀ = 0 and use: $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ ρ controls how strongly earlier squared gradients are retained. A large average magnitude produces a larger denominator and a smaller step. A small average magnitude produces a relatively larger step. Squaring happens before averaging; averaging signed gradients instead would allow cancellation and would not be RMSProp. Example: θ₀ = 1, η = 0.1, ρ = 0.9, s₀ = 0, g₁ = 2. Then s₁ = 0.9 × 0+0.1 × 4 = 0.4. θ₁ = 1−0.2/√0.4 = 0.683772234 when ε is negligible. If g₂ = 1, s₂ = 0.9 × 0.4+0.1 × 1 = 0.46 and θ₂ = 0.683772234−0.1/√0.46 = 0.536330278. Benefits include adapting to different coordinate scales and avoiding AdaGrad's permanent accumulation of all past squared gradients. Costs include extra state memory and tuning η, ρ and ε; noisy or badly chosen settings can still produce poor training. Unlike standard SGD it rescales each coordinate; unlike Adam it does not include a separate first-moment estimate in this basic form. --- ## TEE · R26 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s12 ## Question **Question (10 marks).** For the binary CNN below, calculate output dimensions and trainable parameters after every layer. Input 256 × 256 × 3. → Conv K = 5, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 128, stride = 1, padding = 1. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 84 units. → Dense 32 units. → Dense 1 units. The printed output size is unspecified; assume one sigmoid output. Unspecified convolution padding is taken as valid. ## Answer — step by step ### Given Input 256 × 256 × 3. Conv K = 5, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 64, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 128, stride = 1, padding = 1. Pool K = 2, stride = 2, padding = 0. Dense 84 units. Dense 32 units. Dense 1 units. assume one sigmoid output. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**256 × 256 × 3**height × width × channels → Convolution**32 filters · 5 × 5 · S = 1 · P = 0** → Output shape**252 × 252 × 32**height × width × channels Weights**2,400** + Biases**32** = Trainable parameters**2,432** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **256 × 256 × 3**. Use 32 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{256+2(0)-5}{1}\right\rfloor+1=252 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **252 × 252 × 32**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times3\times32=2{,}400 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_1=2{,}400+32=2{,}432 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**252 × 252 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**126 × 126 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **252 × 252 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{252-2}{2}\right\rfloor+1=126 $$ Pooling acts independently on each channel. The output is **126 × 126 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**126 × 126 × 32**height × width × channels → Convolution**64 filters · 5 × 5 · S = 1 · P = 0** → Output shape**122 × 122 × 64**height × width × channels Weights**51,200** + Biases**64** = Trainable parameters**51,264** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **126 × 126 × 32**. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{126+2(0)-5}{1}\right\rfloor+1=122 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **122 × 122 × 64**. **3. Count trainable values.** Each filter receives all 32 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times32\times64=51{,}200 $$ $$ N_{\mathrm{biases}}=64 $$ $$ P_3=51{,}200+64=51{,}264 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**122 × 122 × 64**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**61 × 61 × 64**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **122 × 122 × 64**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{122-2}{2}\right\rfloor+1=61 $$ Pooling acts independently on each channel. The output is **61 × 61 × 64**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**61 × 61 × 64**height × width × channels → Convolution**128 filters · 5 × 5 · S = 1 · P = 1** → Output shape**59 × 59 × 128**height × width × channels Weights**204,800** + Biases**128** = Trainable parameters**204,928** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **61 × 61 × 64**. Use 128 filters of spatial size 5 × 5, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{61+2(1)-5}{1}\right\rfloor+1=59 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **59 × 59 × 128**. **3. Count trainable values.** Each filter receives all 64 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times64\times128=204{,}800 $$ $$ N_{\mathrm{biases}}=128 $$ $$ P_5=204{,}800+128=204{,}928 $$ ### Layer 6 — pooling Layer 6: what changes and what is learned Input shape**59 × 59 × 128**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**29 × 29 × 128**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **59 × 59 × 128**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{59-2}{2}\right\rfloor+1=29 $$ Pooling acts independently on each channel. The output is **29 × 29 × 128**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 28.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 7 — flatten Layer 7: what changes and what is learned Input shape**29 × 29 × 128**height × width × channels → Flatten**Rearrange activations** → Output shape**107,648**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **29 × 29 × 128**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=29\times29\times128=107{,}648 $$ Output: a vector with **107,648 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**107,648**input features → Dense layer**84 neurons** → Output shape**84**output features Weights**9,042,432** + Biases**84** = Trainable parameters**9,042,516** Every output neuron receives every input feature, with one bias per output neuron. Input: **107,648 features**. Output: **84 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=107{,}648\times84=9{,}042{,}432 $$ $$ N_{\mathrm{biases}}=84 $$ $$ P_8=9{,}042{,}432+84=9{,}042{,}516 $$ ### Layer 9 — fully connected layer Layer 9: what changes and what is learned Input shape**84**input features → Dense layer**32 neurons** → Output shape**32**output features Weights**2,688** + Biases**32** = Trainable parameters**2,720** Every output neuron receives every input feature, with one bias per output neuron. Input: **84 features**. Output: **32 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=84\times32=2{,}688 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_9=2{,}688+32=2{,}720 $$ ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**32**input features → Dense layer**1 neurons** → Output shape**1**output features Weights**32** + Biases**1** = Trainable parameters**33** Every output neuron receives every input feature, with one bias per output neuron. Input: **32 features**. Output: **1 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=32\times1=32 $$ $$ N_{\mathrm{biases}}=1 $$ $$ P_10=32+1=33 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{256+2 \times 0-5}{1}\right\rfloor+1$ | 252 x 252 x 32 | 2,400 | 32 | 2,432 | | 2 pool | $\left\lfloor\frac{252-2}{2}\right\rfloor+1$ | 126 x 126 x 32 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{126+2 \times 0-5}{1}\right\rfloor+1$ | 122 x 122 x 64 | 51,200 | 64 | 51,264 | | 4 pool | $\left\lfloor\frac{122-2}{2}\right\rfloor+1$ | 61 x 61 x 64 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{61+2 \times 1-5}{1}\right\rfloor+1$ | 59 x 59 x 128 | 204,800 | 128 | 204,928 | | 6 pool | $\left\lfloor\frac{59-2}{2}\right\rfloor+1$ | 29 x 29 x 128 | 0 | 0 | 0 | | 7 flat | 29 × 29 × 128 | 107648 | 0 | 0 | 0 | | 8 dense | 107648 → 84 | 84 | 9,042,432 | 84 | 9,042,516 | | 9 dense | 84 → 32 | 32 | 2,688 | 32 | 2,720 | | 10 dense | 32 → 1 | 1 | 32 | 1 | 33 | **Parameter substitutions, in layer order:** Layer 1: (5 x 5 x 3 + 1) x 32 = 2,432. Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264. Layer 5: (5 x 5 x 64 + 1) x 128 = 204,928. Layer 8: (107648 + 1) x 84 = 9,042,516. Layer 9: (84 + 1) x 32 = 2,720. Layer 10: (32 + 1) x 1 = 33. **Total:** 2,432 + 51,264 + 204,928 + 9,042,516 + 2,720 + 33 = **9,303,893 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. **Correction:** Conv 3 has (5 × 5 × 64+1) × 128 = **204,928** parameters, not 73,856 as printed in the key. Correct total is **9,303,893**. If binary classification is instead implemented with two softmax outputs, the last dense layer has(32+1) × 2 = 66 parameters and the total becomes 9,303,926; explicitly state the output convention. --- ## TEE · R26 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s13 ## Question **Question (10 marks).** For a 10-layer network using sigmoid: compute f(6), compute its derivative at 6, explain its effect on backpropagation, and calculate softmax for [2.3,4.5,3.2]. ## Answer — step by step ### Given Sigmoid is evaluated at z = 6 in a ten-layer network. Softmax scores = [2.3,4.5,3.2]. ### To find Calculate sigmoid and its derivative at 6, explain the gradient effect and calculate all three softmax probabilities. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. ### Solution — step by step f(6) = $\frac{1}{1+e^{-6}}$ = **0.997527377**. Derivative = f(6)[1−f(6)] = 0.997527377 × 0.002472623 = **0.002466509**. This is a saturated sigmoid with a very small slope. Repeatedly multiplying small activation derivatives and weight factors can weaken earlier-layer gradients; the exact gradient also depends on weights and all preactivations, so it is not simply this derivative raised to 10 unless those conditions are assumed. **Step 1 stabilize:** subtract the largest logit 4.5 from every value: [-2.2, 0, -1.3]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.110803158, 1, 0.272531793]. **Step 3 denominator:** sum = 1.383334951. **Step 4 divide each value by this same sum:** probabilities = **[0.080098575, 0.722890721, 0.197010704]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. --- ## TEE · R26 Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s14 ## Question **Question (10 marks).** Compare single-layer and multilayer feedforward networks and explain hidden nonlinearity. Compute the output for Input x = [0.5, 0.2, 0.1]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.5]. Hidden and output activations are sigmoid. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_2fad289da9.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.4 | 0.2 | 0.3 | 0.5 | 0.7 | W₂ | 0.6 | 0.9 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.5, 0.2, 0.1]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.5]. Hidden and output activations are sigmoid. ### To find Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator. ![Opposite corners for XOR cannot be split by one straight line](https://deep-learning-study-room.vercel.app/assets/figures/xor.svg) Diagram: Opposite corners for XOR cannot be split by one straight line. Labels: x1 | x2 | 0 | 1 | 1 | 0 | XOR labels occupy opposite corners | 0 | 1 | 0 | 1 | One straight decision boundary cannot separate the two classes. Opposite corners for XOR cannot be split by one straight line Tap the diagram to enlarge. For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1. For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR. A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.5) + (0.4 \times 0.2) + (0.2 \times 0.1) + (0.1) = 0.25 $$ $$ h_{1}=\frac{1}{1+e^{-(0.25)}}=0.562176501 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 0.5) + (0.5 \times 0.2) + (0.7 \times 0.1) + (0.2) = 0.52 $$ $$ h_{2}=\frac{1}{1+e^{-(0.52)}}=0.627147766 $$ **Output neuron 1:** $$ z_{o_1} = (0.6 \times 0.562176501) + (0.9 \times 0.627147766) + (0.5) = 1.40173889 $$ $$ o_{1}=\frac{1}{1+e^{-(1.40173889)}}=0.802459679 $$ --- ## TEE · R26 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c12-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/12/tee-c12-s15 ## Question **Question (10 marks).** Explain transfer learning with CNNs and how to adapt a pretrained model to a new image classification task, including an example model. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. **Concrete example:** replace the 1000-class head of a pretrained ResNet with a 10-class vegetable head. Train the new head with the base frozen, then optionally fine-tune late residual blocks at a smaller rate. Input preprocessing must match the pretrained model. The 10 output probabilities sum to 1 for mutually exclusive classes. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-intro F23 = S17, 2 December 2023. Scanned formulas and diagrams were checked visually rather than trusting OCR alone. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · F23 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s0 ## Question **Question (5 marks).** State the features and limitations of deep learning. ## Answer — step by step Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. --- ## TEE · F23 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s1 ## Question **Question (5 marks).** Identify the hyperparameters to tune when training an ANN and explain their significance. ## Answer — step by step Hyperparameters are chosen outside the ordinary weight-gradient updates. Learning rate controls step size; number of layers and units controls capacity; activation determines nonlinear behavior; batch size controls the data used per update and memory demand; epochs control training duration. Regularization strength, dropout rate and early-stopping patience affect generalization. Optimizer-specific choices include momentum β or Adam β₁, β₂,ε. Tune using validation data while keeping the final test set separate. Weights and biases are learned parameters; they are not hyperparameters merely because their initial values must be selected. --- ## TEE · F23 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s2 ## Question **Question (5 marks).** Draw a time-unrolled RNN and briefly explain training by backpropagation through time. ## Answer — step by step Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph. 1. Initialize the hidden state and run the forward recurrence from the first input to the last. 2. Compute the specified output loss, at the final step or across several steps. 3. Start from the last loss and propagate derivatives backwards through output and hidden operations. 4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states. 5. Sum all uses of each shared weight before applying the optimizer update. $$ \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T} $$ The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step. ![BPTT follows this unrolled network backwards and sums gradients of shared weights](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: BPTT follows this unrolled network backwards and sums gradients of shared weights. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge. --- ## TEE · F23 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s3 ## Question **Question (5 marks).** Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Kernel K Kernel K | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Compute the convolution using the input A and kernel K shown below using padding 0 and stride 1. ### To find Find every convolution output cell. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-3}{1}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 10 | 9 | 5 | | 5 | 14 | 6 | | 3 | 3 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**10 + 9 + 5 = 24** Row 2**5 + 14 + 6 = 25** Row 3**3 + 3 + 2 = 8** **24 + 25 + 8 = 57** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 1, column 1. **Window row 1.** $$ R_1=(10)\times(1)+(9)\times(1)+(5)\times(1)=24 $$ **Window row 2.** $$ R_2=(5)\times(1)+(7)\times(2)+(6)\times(1)=25 $$ **Window row 3.** $$ R_3=(3)\times(1)+(3)\times(1)+(2)\times(1)=8 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(24)+(25)+(8)=57 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **2–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 9 | 5 | 6 | | 7 | 12 | 7 | | 3 | 2 | 8 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**9 + 5 + 6 = 20** Row 2**7 + 12 + 7 = 26** Row 3**3 + 2 + 8 = 13** **20 + 26 + 13 = 59** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 1, column 2. **Window row 1.** $$ R_1=(9)\times(1)+(5)\times(1)+(6)\times(1)=20 $$ **Window row 2.** $$ R_2=(7)\times(1)+(6)\times(2)+(7)\times(1)=26 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(1)+(8)\times(1)=13 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(20)+(26)+(13)=59 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–4**, columns **1–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 5 | 7 | 6 | | 3 | 6 | 2 | | 3 | 2 | 6 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**5 + 7 + 6 = 18** Row 2**3 + 6 + 2 = 11** Row 3**3 + 2 + 6 = 11** **18 + 11 + 11 = 40** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 2, column 1. **Window row 1.** $$ R_1=(5)\times(1)+(7)\times(1)+(6)\times(1)=18 $$ **Window row 2.** $$ R_2=(3)\times(1)+(3)\times(2)+(2)\times(1)=11 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(1)+(6)\times(1)=11 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(18)+(11)+(11)=40 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–4**, columns **2–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 10 | 9 | 5 | 6 | | 5 | 7 | 6 | 7 | | 3 | 3 | 2 | 8 | | 3 | 2 | 6 | 7 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 1 | 1 | | 1 | 2 | 1 | | 1 | 1 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 7 | 6 | 7 | | 3 | 4 | 8 | | 2 | 6 | 7 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**7 + 6 + 7 = 20** Row 2**3 + 4 + 8 = 15** Row 3**2 + 6 + 7 = 15** **20 + 15 + 15 = 50** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 57 | 59 | | 40 | 50 | Output row 2, column 2. **Window row 1.** $$ R_1=(7)\times(1)+(6)\times(1)+(7)\times(1)=20 $$ **Window row 2.** $$ R_2=(3)\times(1)+(2)\times(2)+(8)\times(1)=15 $$ **Window row 3.** $$ R_3=(2)\times(1)+(6)\times(1)+(7)\times(1)=15 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(20)+(15)+(15)=50 $$ **Final output:** $\begin{bmatrix}57 & 59\\40 & 50\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. --- ## TEE · F23 Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s4 ## Question **Question (10 marks).** A feedforward network has inputs 1,2; hidden nodes 3,4; outputs 5,6. Weights: w13 = −2, w23 = 3, w14 = 4, w24 = −1, w35 = 1, w45 = −1, w36 = −1, w46 = 1; all biases 0. All nodes 3–6 use step(z) = 1 if z≥0 else 0. Find(y5, y6) for inputs(0,0), (1,0), (0,1), (1,1). ## Answer — step by step ### Given A feedforward network has inputs 1,2. hidden nodes 3,4. outputs 5,6. Weights: w13 = −2, w23 = 3, w14 = 4, w24 = −1, w35 = 1, w45 = −1, w36 = −1, w46 = 1. all biases 0. All nodes 3–6 use step(z) = 1 if z≥0 else 0. Find(y5, y6) for inputs(0,0), (1,0), (0,1), (1,1). ### To find Write every row of the truth table and the requested network output. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step Use exactly the printed step: output 1 if z≥0, otherwise 0. In particular, step(0) = 1. The equations are z₃ = −2x₁+3x₂, z₄ = 4x₁−x₂, z₅ = h₃−h₄, z₆ = −h₃+h₄. **Step 1: Input(0,0).** z₃ = −2 × 0+3 × 0 = 0; h₃ = 1. z₄ = 4 × 0−0 = 0; h₄ = 1. z₅ = 1−1 = 0; y₅ = 1. z₆ = −1+1 = 0; y₆ = 1. Final pair = **(1,1)**. **Step 2: Input(1,0).** z₃ = −2 × 1+3 × 0 = -2; h₃ = 0. z₄ = 4 × 1−0 = 4; h₄ = 1. z₅ = 0−1 = -1; y₅ = 0. z₆ = −0+1 = 1; y₆ = 1. Final pair = **(0,1)**. **Step 3: Input(0,1).** z₃ = −2 × 0+3 × 1 = 3; h₃ = 1. z₄ = 4 × 0−1 = -1; h₄ = 0. z₅ = 1−0 = 1; y₅ = 1. z₆ = −1+0 = -1; y₆ = 0. Final pair = **(1,0)**. **Step 4: Input(1,1).** z₃ = −2 × 1+3 × 1 = 1; h₃ = 1. z₄ = 4 × 1−1 = 3; h₄ = 1. z₅ = 1−1 = 0; y₅ = 1. z₆ = −1+1 = 0; y₆ = 1. Final pair = **(1,1)**. --- ## TEE · F23 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s5 ## Question **Question (10 marks).** (i) Justify the need for activations and explain any two. (ii) Choose ML or DL for facial recognition using 227 × 227 × 1 images and state the advantages. ## Answer — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. **Facial recognition:** a CNN is a suitable DL choice because the input contains spatial visual patterns. It learns local and hierarchical features with shared filters, reducing dependence on manually selected face descriptors. One channel means grayscale. With limited target data, pretrained features and carefully chosen augmentation are preferable starting points; the image dimensions alone do not guarantee enough data for training from scratch. --- ## TEE · F23 Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s6 ## Question **Question (10 marks).** For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers. Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1. Update w46 and w56 using backpropagation. ![F23 Q3a connections with the supplied biases](https://deep-learning-study-room.vercel.app/assets/figures/net_f23.svg) Diagram: F23 Q3a connections with the supplied biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=0 | x3=1 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.4 + x3 × -0.5; bias -0.4 | h2 incoming: x1 × -0.3 + x2 × 0.1 + x3 × 0.2; bias 0.2 | o1 incoming: h1 × -0.3 + h2 × -0.2; bias 0.1 | Use these OLD connections for all gradients in the same pass. F23 Q3a connections with the supplied biases Tap the diagram to enlarge. ## Answer — step by step ### Given For x = [1,0,1], target 1, η = 0.9, use sigmoid at hidden and output layers. Weights w14 = .2, w15 = −.3, w24 = .4, w25 = .1, w34 = −.5, w35 = .2, w46 = −.3, w56 = −.2, biases at 4,5,6 are−.4,.2,.1. Update w46 and w56 using backpropagation. ### To find Find all preactivations, hidden activations and final outputs. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.4 \times 0) + (-0.5 \times 1) + (-0.4) = -0.7 $$ $$ h_{1}=\frac{1}{1+e^{-(-0.7)}}=0.331812228 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.3 \times 1) + (0.1 \times 0) + (0.2 \times 1) + (0.2) = 0.1 $$ $$ h_{2}=\frac{1}{1+e^{-(0.1)}}=0.524979187 $$ **Output neuron 1:** $$ z_{o_1} = (-0.3 \times 0.331812228) + (-0.2 \times 0.524979187) + (0.1) = -0.104539506 $$ $$ o_{1}=\frac{1}{1+e^{-(-0.104539506)}}=0.473888899 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.526111101]. Squared errors = [0.276792891]. Half-sum squared loss = 0.138396445; mean squared error = 0.276792891. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.473888899-1) \times 0.24931821 = -0.131169078 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.331812228**h1 × Receiving delta**−0.131169078**δ at o1 = Connection gradient**−0.043523504**h1 → o1 Now update this weight Old weight**−0.3** → New weight**−0.260828846** Subtract η × gradient = **−0.039171154**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.131169078)\times(0.331812228)\approx -0.043523504 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=-0.3-0.9(-0.043523504)\approx -0.260828846 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.524979187**h2 × Receiving delta**−0.131169078**δ at o1 = Connection gradient**−0.068861036**h2 → o1 Now update this weight Old weight**−0.2** → New weight**−0.138025068** Subtract η × gradient = **−0.061974932**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.131169078)\times(0.524979187)\approx -0.068861036 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.2-0.9(-0.068861036)\approx -0.138025068 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.131169078** × Old outgoing weight**−0.3** = Path contribution**0.039350723** ↓ Add every path contribution**0.039350723**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.3)\times(-0.131169078)\approx 0.039350723 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx 0.039350723 $$ $$ \delta_{h_1} = 0.039350723 \times 0.221712873 = 0.008724562 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.131169078** × Old outgoing weight**−0.2** = Path contribution**0.026233816** ↓ Add every path contribution**0.026233816**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.2)\times(-0.131169078)\approx 0.026233816 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx 0.026233816 $$ $$ \delta_{h_2} = 0.026233816 \times 0.24937604 = 0.006542085 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**0.008724562**δ at h1 = Connection gradient**0.008724562**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.192147894** Subtract η × gradient = **0.007852106**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.008724562)\times(1)\approx 0.008724562 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.9(0.008724562)\approx 0.192147894 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0**x2 × Receiving delta**0.008724562**δ at h1 = Connection gradient**0**x2 → h1 Now update this weight Old weight**0.4** → New weight**0.4** Subtract η × gradient = **0**. This weight **stays unchanged**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.008724562)\times(0)\approx 0 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.4-0.9(0)\approx 0.4 $$ **x3 → h1:** The calculation on x3 → h1 Sending input**1**x3 × Receiving delta**0.008724562**δ at h1 = Connection gradient**0.008724562**x3 → h1 Now update this weight Old weight**−0.5** → New weight**−0.507852106** Subtract η × gradient = **0.007852106**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,3}}=\delta_{h_1}x_3 $$ $$ g_{1,3}^{(1)}=(0.008724562)\times(1)\approx 0.008724562 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,3,\mathrm{new}}=-0.5-0.9(0.008724562)\approx -0.507852106 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**0.006542085**δ at h2 = Connection gradient**0.006542085**x1 → h2 Now update this weight Old weight**−0.3** → New weight**−0.305887877** Subtract η × gradient = **0.005887876**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.006542085)\times(1)\approx 0.006542085 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.3-0.9(0.006542085)\approx -0.305887877 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0**x2 × Receiving delta**0.006542085**δ at h2 = Connection gradient**0**x2 → h2 Now update this weight Old weight**0.1** → New weight**0.1** Subtract η × gradient = **0**. This weight **stays unchanged**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.006542085)\times(0)\approx 0 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.1-0.9(0)\approx 0.1 $$ **x3 → h2:** The calculation on x3 → h2 Sending input**1**x3 × Receiving delta**0.006542085**δ at h2 = Connection gradient**0.006542085**x3 → h2 Now update this weight Old weight**0.2** → New weight**0.194112123** Subtract η × gradient = **0.005887876**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,3}}=\delta_{h_2}x_3 $$ $$ g_{2,3}^{(1)}=(0.006542085)\times(1)\approx 0.006542085 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,3,\mathrm{new}}=0.2-0.9(0.006542085)\approx 0.194112123 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=-0.4-0.9(0.008724562)=-0.407852106 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.9(0.006542085)=0.194112124 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.9(-0.131169078)=0.218052170 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Required:** w46 = **−0.260828846**, w56 = **−0.138025068** under half squared loss. If a one-output unhalved MSE is explicitly required, double the shown gradients before updating; the paper itself does not state that scaling. --- ## TEE · F23 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s7 ## Question **Question (10 marks).** Explain key advantages of Adam for ANN training and how it differs from SGD. ## Answer — step by step Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g. For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9. Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum. --- ## TEE · F23 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s8 ## Question **Question (10 marks).** Determine feature-map dimensions and parameters at every layer of the pictured AlexNet-style CNN. The normalized architecture is Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units. ## Answer — step by step ### Given The normalized architecture is Input 227 × 227 × 3. Conv K = 11, filters = 96, stride = 4, padding = 0. Pool K = 3, stride = 2, padding = 0. Conv K = 5, filters = 256, stride = 1, padding = 2. Pool K = 3, stride = 2, padding = 0. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 256, stride = 1, padding = 1. Pool K = 3, stride = 2, padding = 0. Dense 4096 units. Dense 4096 units. Dense 1000 units. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**227 × 227 × 3**height × width × channels → Convolution**96 filters · 11 × 11 · S = 4 · P = 0** → Output shape**55 × 55 × 96**height × width × channels Weights**34,848** + Biases**96** = Trainable parameters**34,944** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **227 × 227 × 3**. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-11}{4}\right\rfloor+1=55 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **55 × 55 × 96**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=11\times11\times3\times96=34{,}848 $$ $$ N_{\mathrm{biases}}=96 $$ $$ P_1=34{,}848+96=34{,}944 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**55 × 55 × 96**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**27 × 27 × 96**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **55 × 55 × 96**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{55-3}{2}\right\rfloor+1=27 $$ Pooling acts independently on each channel. The output is **27 × 27 × 96**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**27 × 27 × 96**height × width × channels → Convolution**256 filters · 5 × 5 · S = 1 · P = 2** → Output shape**27 × 27 × 256**height × width × channels Weights**614,400** + Biases**256** = Trainable parameters**614,656** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **27 × 27 × 96**. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{27+2(2)-5}{1}\right\rfloor+1=27 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **27 × 27 × 256**. **3. Count trainable values.** Each filter receives all 96 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times96\times256=614{,}400 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_3=614{,}400+256=614{,}656 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**27 × 27 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**13 × 13 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **27 × 27 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{27-3}{2}\right\rfloor+1=13 $$ Pooling acts independently on each channel. The output is **13 × 13 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**884,736** + Biases**384** = Trainable parameters**885,120** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 256**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 256 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times256\times384=884{,}736 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_5=884{,}736+384=885{,}120 $$ ### Layer 6 — convolution Layer 6: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**1,327,104** + Biases**384** = Trainable parameters**1,327,488** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times384=1{,}327{,}104 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_6=1{,}327{,}104+384=1{,}327{,}488 $$ ### Layer 7 — convolution Layer 7: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**256 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 256**height × width × channels Weights**884,736** + Biases**256** = Trainable parameters**884,992** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 256**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times256=884{,}736 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_7=884{,}736+256=884{,}992 $$ ### Layer 8 — pooling Layer 8: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**6 × 6 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **13 × 13 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{13-3}{2}\right\rfloor+1=6 $$ Pooling acts independently on each channel. The output is **6 × 6 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 9 — flatten Layer 9: what changes and what is learned Input shape**6 × 6 × 256**height × width × channels → Flatten**Rearrange activations** → Output shape**9,216**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **6 × 6 × 256**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=6\times6\times256=9{,}216 $$ Output: a vector with **9,216 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**9,216**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**37,748,736** + Biases**4,096** = Trainable parameters**37,752,832** Every output neuron receives every input feature, with one bias per output neuron. Input: **9,216 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=9{,}216\times4{,}096=37{,}748{,}736 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_10=37{,}748{,}736+4{,}096=37{,}752{,}832 $$ ### Layer 11 — fully connected layer Layer 11: what changes and what is learned Input shape**4,096**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**16,777,216** + Biases**4,096** = Trainable parameters**16,781,312** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times4{,}096=16{,}777{,}216 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_11=16{,}777{,}216+4{,}096=16{,}781{,}312 $$ ### Layer 12 — fully connected layer Layer 12: what changes and what is learned Input shape**4,096**input features → Dense layer**1,000 neurons** → Output shape**1,000**output features Weights**4,096,000** + Biases**1,000** = Trainable parameters**4,097,000** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **1,000 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times1{,}000=4{,}096{,}000 $$ $$ N_{\mathrm{biases}}=1{,}000 $$ $$ P_12=4{,}096{,}000+1{,}000=4{,}097{,}000 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{227+2 \times 0-11}{4}\right\rfloor+1$ | 55 x 55 x 96 | 34,848 | 96 | 34,944 | | 2 pool | $\left\lfloor\frac{55-3}{2}\right\rfloor+1$ | 27 x 27 x 96 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{27+2 \times 2-5}{1}\right\rfloor+1$ | 27 x 27 x 256 | 614,400 | 256 | 614,656 | | 4 pool | $\left\lfloor\frac{27-3}{2}\right\rfloor+1$ | 13 x 13 x 256 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 884,736 | 384 | 885,120 | | 6 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 | | 7 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 256 | 884,736 | 256 | 884,992 | | 8 pool | $\left\lfloor\frac{13-3}{2}\right\rfloor+1$ | 6 x 6 x 256 | 0 | 0 | 0 | | 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 | | 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 | | 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 | | 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 | **Parameter substitutions, in layer order:** Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944. Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656. Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120. Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488. Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992. Layer 10: (9216 + 1) x 4096 = 37,752,832. Layer 11: (4096 + 1) x 4096 = 16,781,312. Layer 12: (4096 + 1) x 1000 = 4,097,000. **Total:** 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = **62,378,344 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. The count treats every listed convolution as connected to all input channels. Original grouped AlexNet implementations have different counts; do not insert grouping that is not shown in the question. --- ## TEE · F23 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s9 ## Question **Question (10 marks).** Design an LSTM autocomplete model for “Maharashtra is a state in the western part of India. The capital of Maharashtra is \_\_\_.” Draw the model and write conditional probabilities at each time step. ## Answer — step by step An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. Architecture: token → embedding → LSTM state → vocabulary softmax → next-token distribution. During training use the actual prefix and its next-word labels with cross-entropy. At inference, feed the available prefix and choose/sample the next token. “Mumbai” is the intended factual completion in this example; a model only predicts it if learned probabilities support it. **One explicit tokenization ignoring punctuation:** 1. P(Maharashtra | START). 2. P(is | Maharashtra). 3. P(a | Maharashtra is). 4. P(state | Maharashtra is a). 5. P(in | Maharashtra is a state). 6. P(the | Maharashtra is a state in). 7. P(western | Maharashtra is a state in the). 8. P(part | Maharashtra is a state in the western). 9. P(of | Maharashtra is a state in the western part). 10. P(India | Maharashtra is a state in the western part of). 11. P(The | Maharashtra is a state in the western part of India). 12. P(capital | Maharashtra is a state in the western part of India The). 13. P(of | Maharashtra is a state in the western part of India The capital). 14. P(Maharashtra | Maharashtra is a state in the western part of India The capital of). 15. P(is | Maharashtra is a state in the western part of India The capital of Maharashtra). 16. P(Mumbai | Maharashtra is a state in the western part of India The capital of Maharashtra is). The joint sequence probability is the product of these conditionals. Numerical probability values cannot be computed from the sentence alone because weights are not supplied. ![Replace the repeated recurrent cell with an LSTM and attach a vocabulary softmax at each step](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Replace the repeated recurrent cell with an LSTM and attach a vocabulary softmax at each step. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Replace the repeated recurrent cell with an LSTM and attach a vocabulary softmax at each step Tap the diagram to enlarge. --- ## TEE · F23 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s10 ## Question **Question (10 marks).** For Inception V 1, justify different kernel sizes in one block, explain 1 × 1 convolutions and sketch one block. ## Answer — step by step An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. --- ## TEE · F23 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s11 ## Question **Question (10 marks).** Explain why a single perceptron cannot classify nonlinearly separable data. Implement XOR and XNOR using suitable perceptrons. ## Answer — step by step ### Given Required logic functions = XOR and XNOR on two binary inputs. Weights and biases are design choices, not given constants. ### To find Explain the failure of a single linear separator, choose suitable hidden/output perceptrons and show all four truth-table rows. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator. ![Opposite corners for XOR cannot be split by one straight line](https://deep-learning-study-room.vercel.app/assets/figures/xor.svg) Diagram: Opposite corners for XOR cannot be split by one straight line. Labels: x1 | x2 | 0 | 1 | 1 | 0 | XOR labels occupy opposite corners | 0 | 1 | 0 | 1 | One straight decision boundary cannot separate the two classes. Opposite corners for XOR cannot be split by one straight line Tap the diagram to enlarge. For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1. For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR. A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map. ### Verify input (0,0) **Step 1: calculate hidden OR preactivation.** $$ z_1=0+0-0.5=-0.5 $$ **Step 2: apply its threshold.** $$ h_1=0 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=0+0-1.5=-1.5 $$ **Step 4: apply its threshold.** $$ h_2=0 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=0-2(0)-0.5=-0.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=0 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-0+2(0)+0.5=0.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=1 $$ ### Verify input (0,1) **Step 1: calculate hidden OR preactivation.** $$ z_1=0+1-0.5=0.5 $$ **Step 2: apply its threshold.** $$ h_1=1 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=0+1-1.5=-0.5 $$ **Step 4: apply its threshold.** $$ h_2=0 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=1-2(0)-0.5=0.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=1 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-1+2(0)+0.5=-0.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=0 $$ ### Verify input (1,0) **Step 1: calculate hidden OR preactivation.** $$ z_1=1+0-0.5=0.5 $$ **Step 2: apply its threshold.** $$ h_1=1 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=1+0-1.5=-0.5 $$ **Step 4: apply its threshold.** $$ h_2=0 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=1-2(0)-0.5=0.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=1 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-1+2(0)+0.5=-0.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=0 $$ ### Verify input (1,1) **Step 1: calculate hidden OR preactivation.** $$ z_1=1+1-0.5=1.5 $$ **Step 2: apply its threshold.** $$ h_1=1 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=1+1-1.5=0.5 $$ **Step 4: apply its threshold.** $$ h_2=1 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=1-2(1)-0.5=-1.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=0 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-1+2(1)+0.5=1.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=1 $$ **Final answer:** XOR = [0,1,1,0]; XNOR = [1,0,0,1] for 00,01,10,11. The hidden nonlinear units make this possible. --- ## TEE · F23 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s12 ## Question **Question (10 marks).** For the pictured LSTM time step, explain the forget, “update” and output gates, develop “update and reset” equations, and discuss long sequence lengths. The source mixes LSTM and GRU terminology. ## Answer — step by step **Terminology correction:** the pictured LSTM uses forget, input and output gates plus a candidate. Its input/candidate path updates the cell. A reset gate belongs to GRU, so no genuine LSTM reset-gate equation exists. The complete consistent LSTM answer is: An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. **For the explicitly named GRU terms in the question:** A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate: $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ ![Reset controls candidate context and update controls the old new mixture](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: Reset controls candidate context and update controls the old new mixture. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. Reset controls candidate context and update controls the old new mixture Tap the diagram to enlarge. The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate. Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN. In an exam, state the mismatch in one sentence, label each architecture clearly and then give its correct equations. Do not memorize a fictitious reset gate for LSTM. --- ## TEE · F23 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s13 ## Question **Question (10 marks).** Explain how ResNet addresses vanishing gradients. Compare a block output with and without a residual connection. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. --- ## TEE · F23 Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s14 ## Question **Question (10 marks).** Design and sketch a sentiment-analysis model whose input is a small paragraph. Explain how its output is generated. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. **Concrete many-to-one design:** tokenize paragraph → embeddings → LSTM sequence processor → final hidden state → dense 3 → softmax for positive/neutral/negative. If the task specifies only two labels, use one sigmoid or two softmax outputs instead. Train with the matching cross-entropy. Keep token order, handle padding/masks consistently and evaluate held-out reviews. ![Sentiment uses the many-to-one pattern](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: Sentiment uses the many-to-one pattern. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. Sentiment uses the many-to-one pattern Tap the diagram to enlarge. --- ## TEE · F23 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c13-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/13/tee-c13-s15 ## Question **Question (10 marks).** Explain the bias-variance tradeoff. Fill effects on bias and variance for weight regularization, more hidden units, dropout and more training data from the same distribution. ## Answer — step by step Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data. ![Validation behavior distinguishes useful fitting from prolonged fitting to training details](https://deep-learning-study-room.vercel.app/assets/figures/overfitting.svg) Diagram: Validation behavior distinguishes useful fitting from prolonged fitting to training details. Labels: Loss | Epochs | Validation | Training | Best validation region | Save the best validation checkpoint; do not choose by test loss. Validation behavior distinguishes useful fitting from prolonged fitting to training details Tap the diagram to enlarge. Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws. Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate. Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting. | Change | Typical bias | Typical variance | | --- | --- | --- | | Stronger weight regularization | Increases | Decreases | | More hidden units | Decreases | Increases | | Dropout during training | Increases | Decreases | | More same-distribution training data | Approximately no change | Decreases | These are the conventional exam trends, assuming other conditions are comparable. They are not guarantees for every training run. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-intro R24 = S14, 13 February 2024. Its source matrix uses an input-to-hidden indexing convention, explicitly handled below. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · R24 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s0 ## Question **Question (5 marks).** What factors drive widespread adoption of deep learning? ## Answer — step by step Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. --- ## TEE · R24 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s1 ## Question **Question (5 marks).** Draw an animal-classification network with 10 inputs, three hidden layers of 5 neurons each and 4 output classes. Every hidden/output neuron has a bias. Count all trainable parameters. ## Answer — step by step ### Given 10 input features → 5 hidden → 5 hidden → 5 hidden → 4 output classes. One bias per hidden/output neuron. ### To find Draw the layer structure and count weights and biases. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step Architecture: 10 → 5 → 5 → 5 → 4. Each non-input neuron has one bias. ![R24 Q1b every layer contains the stated number of neurons](https://deep-learning-study-room.vercel.app/assets/figures/dense_10_5_5_5_4.svg) Diagram: R24 Q1b every layer contains the stated number of neurons. Labels: Input | 10 neurons | Hidden 1 | 5 neurons | Hidden 2 | 5 neurons | Hidden 3 | 5 neurons | Output | 4 neurons | Every receiving neuron has one bias. All adjacent layers are fully connected. R24 Q1b every layer contains the stated number of neurons Tap the diagram to enlarge. ### Layer 10 → 5 **Step 1: count connections.** $$ 10(5)=50 $$ **Step 2: count receiving-neuron biases.** There are 5 receiving neurons, so 5 biases. **Step 3: add weights and biases.** $$ 50+5=55 $$ ### Layer 5 → 5 **Step 1: count connections.** $$ 5(5)=25 $$ **Step 2: count receiving-neuron biases.** There are 5 receiving neurons, so 5 biases. **Step 3: add weights and biases.** $$ 25+5=30 $$ ### Layer 5 → 5 **Step 1: count connections.** $$ 5(5)=25 $$ **Step 2: count receiving-neuron biases.** There are 5 receiving neurons, so 5 biases. **Step 3: add weights and biases.** $$ 25+5=30 $$ ### Layer 5 → 4 **Step 1: count connections.** $$ 5(4)=20 $$ **Step 2: count receiving-neuron biases.** There are 4 receiving neurons, so 4 biases. **Step 3: add weights and biases.** $$ 20+4=24 $$ **Total = 139**. Input values and activations are not trainable parameters. Use hidden nonlinearities such as ReLU and a 4-class softmax. Learned parameters are the connections and biases; the number of layers and units are design choices. --- ## TEE · R24 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s2 ## Question **Question (5 marks).** A deep network strongly overfits its training set and performs poorly on unseen data. Explain remedies. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. Use a representative validation set to choose the remedy. Also inspect leakage, label quality and distribution mismatch. If the final test set was repeatedly used to tune changes, reserve a fresh untouched set for final reporting. --- ## TEE · R24 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s3 ## Question **Question (5 marks).** For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_915e1db14b.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given For feature map Matrix 1 shown below, find max and average pooling with 2 × 2 windows, stride 2. ### To find Find the requested pooled output(s). ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [21, 11, 12, 9] **Step 2: add all selected entries.** 21 + 11 + 12 + 9 = 53 **Step 3: divide by the 4 entries in this window.** 53/4 = **13.25**. Maximum = **21** ### Pooling window (1,2) **Step 1: select the entries.** [15, 10, 10, 11] **Step 2: add all selected entries.** 15 + 10 + 10 + 11 = 46 **Step 3: divide by the 4 entries in this window.** 46/4 = **11.5**. Maximum = **15** ### Pooling window (2,1) **Step 1: select the entries.** [11, 13, 7, 4] **Step 2: add all selected entries.** 11 + 13 + 7 + 4 = 35 **Step 3: divide by the 4 entries in this window.** 35/4 = **8.75**. Maximum = **13** ### Pooling window (2,2) **Step 1: select the entries.** [18, 12, 15, 12] **Step 2: add all selected entries.** 18 + 12 + 15 + 12 = 57 **Step 3: divide by the 4 entries in this window.** 57/4 = **14.25**. Maximum = **18** **Average output:** $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . **Max output:** $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. --- ## TEE · R24 Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s4 ## Question **Question (10 marks).** A 4 → 4 → 1 sigmoid network has input [.4,.3,.5,−.4]. The printed W₁ is shown below, W₂ = [.5,.7,−.6,.2]; hidden bias 0.6, output bias 0.2, target 0.9, η = .5. Compute mean squared error and update the connection h₁ → output. In the graph θ ij connects input i to hidden j. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_28edc44005.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.8 | 0.2 | 0.3 | 0.4 | −0.5 | 0.4 | 0.3 | 0.2 | 0.1 | 0.2 | −0.1 | 0.2 | −0.2 | 0.5 | 0.5 | 0.8 Given matrices for this question Tap the diagram to enlarge. ![R24 Q2a receiving-neuron rows after transposing the source convention](https://deep-learning-study-room.vercel.app/assets/figures/net_r24.svg) Diagram: R24 Q2a receiving-neuron rows after transposing the source convention. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.4 | x2=0.3 | x3=0.5 | x4=-0.4 | h1 | h2 | h3 | h4 | o1 | target 0.9 | h1 incoming: x1 × 0.8 + x2 × -0.5 + x3 × 0.1 + x4 × -0.2; bias 0.6 | h2 incoming: x1 × 0.2 + x2 × 0.4 + x3 × 0.2 + x4 × 0.5; bias 0.6 | h3 incoming: x1 × 0.3 + x2 × 0.3 + x3 × -0.1 + x4 × 0.5; bias 0.6 | h4 incoming: x1 × 0.4 + x2 × 0.2 + x3 × 0.2 + x4 × 0.8; bias 0.6 | o1 incoming: h1 × 0.5 + h2 × 0.7 + h3 × -0.6 + h4 × 0.2; bias 0.2 | Use these OLD connections for all gradients in the same pass. R24 Q2a receiving-neuron rows after transposing the source convention Tap the diagram to enlarge. ## Answer — step by step ### Given A 4 → 4 → 1 sigmoid network has input [.4,.3,.5,−.4]. The printed W₁ is printed with the question, W₂ = [.5,.7,−.6,.2]. hidden bias 0.6, output bias 0.2, target 0.9, η = .5. ### To find Find all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Mean squared loss.** Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean. n = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target. $$ L=\frac1n\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=\frac2n(o_j-y_j) $$ **Yaad rakhne ka cue:** MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Orientation:** the source rows list sending inputs, so use each COLUMN to form a receiving hidden neuron. In our usual column-input convention this means using the transpose of printed W₁. The loss is one-output MSE = (o−0.9)², so its derivative includes 2. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.8 \times 0.4) + (-0.5 \times 0.3) + (0.1 \times 0.5) + (-0.2 \times -0.4) + (0.6) = 0.9 $$ $$ h_{1}=\frac{1}{1+e^{-(0.9)}}=0.710949503 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.2 \times 0.4) + (0.4 \times 0.3) + (0.2 \times 0.5) + (0.5 \times -0.4) + (0.6) = 0.7 $$ $$ h_{2}=\frac{1}{1+e^{-(0.7)}}=0.668187772 $$ **Hidden neuron 3:** $$ z_{h_3} = (0.3 \times 0.4) + (0.3 \times 0.3) + (-0.1 \times 0.5) + (0.5 \times -0.4) + (0.6) = 0.56 $$ $$ h_{3}=\frac{1}{1+e^{-(0.56)}}=0.63645254 $$ **Hidden neuron 4:** $$ z_{h_4} = (0.4 \times 0.4) + (0.2 \times 0.3) + (0.2 \times 0.5) + (0.8 \times -0.4) + (0.6) = 0.6 $$ $$ h_{4}=\frac{1}{1+e^{-(0.6)}}=0.645656306 $$ **Output neuron 1:** $$ \begin{aligned}z_{o_1}={}&0.5(0.710949503)+0.7(0.668187772)\\&-0.6(0.63645254)+0.2(0.645656306)+0.2\\={}&0.770465929\end{aligned} $$ $$ o_{1}=\frac{1}{1+e^{-(0.770465929)}}=0.683621675 $$ **Step 2 loss convention.** Use the explicitly requested one-output MSE, not half squared loss. Prediction minus target = [-0.216378325]. Squared errors = [0.04681958]. Half-sum squared loss = 0.02340979; mean squared error = 0.04681958. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 2 \times (0.683621675-0.9) \times 0.216283081 = -0.093597941 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.710949503**h1 × Receiving delta**−0.093597941**δ at o1 = Connection gradient**−0.06654341**h1 → o1 Now update this weight Old weight**0.5** → New weight**0.533271705** Subtract η × gradient = **−0.033271705**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.093597941)\times(0.710949503)\approx -0.06654341 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.5-0.5(-0.06654341)\approx 0.533271705 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.668187772**h2 × Receiving delta**−0.093597941**δ at o1 = Connection gradient**−0.062541**h2 → o1 Now update this weight Old weight**0.7** → New weight**0.7312705** Subtract η × gradient = **−0.0312705**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.093597941)\times(0.668187772)\approx -0.062541 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.7-0.5(-0.062541)\approx 0.7312705 $$ **h3 → o1:** The calculation on h3 → o1 Sending activation**0.63645254**h3 × Receiving delta**−0.093597941**δ at o1 = Connection gradient**−0.059570648**h3 → o1 Now update this weight Old weight**−0.6** → New weight**−0.570214676** Subtract η × gradient = **−0.029785324**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,3}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,3}}=\delta_{o_1}h_3 $$ $$ g_{1,3}^{(2)}=(-0.093597941)\times(0.63645254)\approx -0.059570648 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,3,\mathrm{new}}=-0.6-0.5(-0.059570648)\approx -0.570214676 $$ **h4 → o1:** The calculation on h4 → o1 Sending activation**0.645656306**h4 × Receiving delta**−0.093597941**δ at o1 = Connection gradient**−0.060432101**h4 → o1 Now update this weight Old weight**0.2** → New weight**0.230216051** Subtract η × gradient = **−0.03021605**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,4}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,4}}=\delta_{o_1}h_4 $$ $$ g_{1,4}^{(2)}=(-0.093597941)\times(0.645656306)\approx -0.060432101 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,4,\mathrm{new}}=0.2-0.5(-0.060432101)\approx 0.230216051 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.093597941** × Old outgoing weight**0.5** = Path contribution**−0.04679897** ↓ Add every path contribution**−0.046798971**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.093597941)\approx -0.04679897 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.046798971 $$ $$ \delta_{h_1} = -0.046798971 \times 0.205500307 = -0.009617203 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.093597941** × Old outgoing weight**0.7** = Path contribution**−0.065518559** ↓ Add every path contribution**−0.065518559**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.7)\times(-0.093597941)\approx -0.065518559 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.065518559 $$ $$ \delta_{h_2} = -0.065518559 \times 0.221712873 = -0.014526308 $$ **Hidden 3 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.093597941** × Old outgoing weight**−0.6** = Path contribution**0.056158765** ↓ Add every path contribution**0.056158765**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.6)\times(-0.093597941)\approx 0.056158765 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx 0.056158765 $$ $$ \delta_{h_3} = 0.056158765 \times 0.231380704 = 0.012994055 $$ **Hidden 4 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.093597941** × Old outgoing weight**0.2** = Path contribution**−0.018719588** ↓ Add every path contribution**−0.018719588**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.093597941)\approx -0.018719588 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.018719588 $$ $$ \delta_{h_4} = -0.018719588 \times 0.22878424 = -0.004282747 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.4**x1 × Receiving delta**−0.009617203**δ at h1 = Connection gradient**−0.003846881**x1 → h1 Now update this weight Old weight**0.8** → New weight**0.801923441** Subtract η × gradient = **−0.00192344**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.009617203)\times(0.4)\approx -0.003846881 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.8-0.5(-0.003846881)\approx 0.801923441 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.3**x2 × Receiving delta**−0.009617203**δ at h1 = Connection gradient**−0.002885161**x2 → h1 Now update this weight Old weight**−0.5** → New weight**−0.49855742** Subtract η × gradient = **−0.00144258**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.009617203)\times(0.3)\approx -0.002885161 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=-0.5-0.5(-0.002885161)\approx -0.49855742 $$ **x3 → h1:** The calculation on x3 → h1 Sending input**0.5**x3 × Receiving delta**−0.009617203**δ at h1 = Connection gradient**−0.004808601**x3 → h1 Now update this weight Old weight**0.1** → New weight**0.102404301** Subtract η × gradient = **−0.0024043**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,3}}=\delta_{h_1}x_3 $$ $$ g_{1,3}^{(1)}=(-0.009617203)\times(0.5)\approx -0.004808601 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,3,\mathrm{new}}=0.1-0.5(-0.004808601)\approx 0.102404301 $$ **x4 → h1:** The calculation on x4 → h1 Sending input**−0.4**x4 × Receiving delta**−0.009617203**δ at h1 = Connection gradient**0.003846881**x4 → h1 Now update this weight Old weight**−0.2** → New weight**−0.201923441** Subtract η × gradient = **0.00192344**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,4}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,4}}=\delta_{h_1}x_4 $$ $$ g_{1,4}^{(1)}=(-0.009617203)\times(-0.4)\approx 0.003846881 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,4,\mathrm{new}}=-0.2-0.5(0.003846881)\approx -0.201923441 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.4**x1 × Receiving delta**−0.014526308**δ at h2 = Connection gradient**−0.005810523**x1 → h2 Now update this weight Old weight**0.2** → New weight**0.202905262** Subtract η × gradient = **−0.002905262**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.014526308)\times(0.4)\approx -0.005810523 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.2-0.5(-0.005810523)\approx 0.202905262 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.3**x2 × Receiving delta**−0.014526308**δ at h2 = Connection gradient**−0.004357892**x2 → h2 Now update this weight Old weight**0.4** → New weight**0.402178946** Subtract η × gradient = **−0.002178946**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.014526308)\times(0.3)\approx -0.004357892 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.4-0.5(-0.004357892)\approx 0.402178946 $$ **x3 → h2:** The calculation on x3 → h2 Sending input**0.5**x3 × Receiving delta**−0.014526308**δ at h2 = Connection gradient**−0.007263154**x3 → h2 Now update this weight Old weight**0.2** → New weight**0.203631577** Subtract η × gradient = **−0.003631577**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,3}}=\delta_{h_2}x_3 $$ $$ g_{2,3}^{(1)}=(-0.014526308)\times(0.5)\approx -0.007263154 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,3,\mathrm{new}}=0.2-0.5(-0.007263154)\approx 0.203631577 $$ **x4 → h2:** The calculation on x4 → h2 Sending input**−0.4**x4 × Receiving delta**−0.014526308**δ at h2 = Connection gradient**0.005810523**x4 → h2 Now update this weight Old weight**0.5** → New weight**0.497094738** Subtract η × gradient = **0.002905262**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,4}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,4}}=\delta_{h_2}x_4 $$ $$ g_{2,4}^{(1)}=(-0.014526308)\times(-0.4)\approx 0.005810523 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,4,\mathrm{new}}=0.5-0.5(0.005810523)\approx 0.497094738 $$ **x1 → h3:** The calculation on x1 → h3 Sending input**0.4**x1 × Receiving delta**0.012994055**δ at h3 = Connection gradient**0.005197622**x1 → h3 Now update this weight Old weight**0.3** → New weight**0.297401189** Subtract η × gradient = **0.002598811**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{3,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{3,1}}=\delta_{h_3}x_1 $$ $$ g_{3,1}^{(1)}=(0.012994055)\times(0.4)\approx 0.005197622 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{3,1,\mathrm{new}}=0.3-0.5(0.005197622)\approx 0.297401189 $$ **x2 → h3:** The calculation on x2 → h3 Sending input**0.3**x2 × Receiving delta**0.012994055**δ at h3 = Connection gradient**0.003898216**x2 → h3 Now update this weight Old weight**0.3** → New weight**0.298050892** Subtract η × gradient = **0.001949108**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{3,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{3,2}}=\delta_{h_3}x_2 $$ $$ g_{3,2}^{(1)}=(0.012994055)\times(0.3)\approx 0.003898216 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{3,2,\mathrm{new}}=0.3-0.5(0.003898216)\approx 0.298050892 $$ **x3 → h3:** The calculation on x3 → h3 Sending input**0.5**x3 × Receiving delta**0.012994055**δ at h3 = Connection gradient**0.006497027**x3 → h3 Now update this weight Old weight**−0.1** → New weight**−0.103248514** Subtract η × gradient = **0.003248514**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{3,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{3,3}}=\delta_{h_3}x_3 $$ $$ g_{3,3}^{(1)}=(0.012994055)\times(0.5)\approx 0.006497027 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{3,3,\mathrm{new}}=-0.1-0.5(0.006497027)\approx -0.103248514 $$ **x4 → h3:** The calculation on x4 → h3 Sending input**−0.4**x4 × Receiving delta**0.012994055**δ at h3 = Connection gradient**−0.005197622**x4 → h3 Now update this weight Old weight**0.5** → New weight**0.502598811** Subtract η × gradient = **−0.002598811**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{3,4}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{3,4}}=\delta_{h_3}x_4 $$ $$ g_{3,4}^{(1)}=(0.012994055)\times(-0.4)\approx -0.005197622 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{3,4,\mathrm{new}}=0.5-0.5(-0.005197622)\approx 0.502598811 $$ **x1 → h4:** The calculation on x1 → h4 Sending input**0.4**x1 × Receiving delta**−0.004282747**δ at h4 = Connection gradient**−0.001713099**x1 → h4 Now update this weight Old weight**0.4** → New weight**0.400856549** Subtract η × gradient = **−0.00085655**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{4,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{4,1}}=\delta_{h_4}x_1 $$ $$ g_{4,1}^{(1)}=(-0.004282747)\times(0.4)\approx -0.001713099 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{4,1,\mathrm{new}}=0.4-0.5(-0.001713099)\approx 0.400856549 $$ **x2 → h4:** The calculation on x2 → h4 Sending input**0.3**x2 × Receiving delta**−0.004282747**δ at h4 = Connection gradient**−0.001284824**x2 → h4 Now update this weight Old weight**0.2** → New weight**0.200642412** Subtract η × gradient = **−0.000642412**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{4,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{4,2}}=\delta_{h_4}x_2 $$ $$ g_{4,2}^{(1)}=(-0.004282747)\times(0.3)\approx -0.001284824 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{4,2,\mathrm{new}}=0.2-0.5(-0.001284824)\approx 0.200642412 $$ **x3 → h4:** The calculation on x3 → h4 Sending input**0.5**x3 × Receiving delta**−0.004282747**δ at h4 = Connection gradient**−0.002141373**x3 → h4 Now update this weight Old weight**0.2** → New weight**0.201070687** Subtract η × gradient = **−0.001070686**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{4,3}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{4,3}}=\delta_{h_4}x_3 $$ $$ g_{4,3}^{(1)}=(-0.004282747)\times(0.5)\approx -0.002141373 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{4,3,\mathrm{new}}=0.2-0.5(-0.002141373)\approx 0.201070687 $$ **x4 → h4:** The calculation on x4 → h4 Sending input**−0.4**x4 × Receiving delta**−0.004282747**δ at h4 = Connection gradient**0.001713099**x4 → h4 Now update this weight Old weight**0.8** → New weight**0.799143451** Subtract η × gradient = **0.00085655**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{4,4}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{4,4}}=\delta_{h_4}x_4 $$ $$ g_{4,4}^{(1)}=(-0.004282747)\times(-0.4)\approx 0.001713099 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{4,4,\mathrm{new}}=0.8-0.5(0.001713099)\approx 0.799143451 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.6-0.5(-0.009617203)=0.604808601 $$ $$ b_{h_2,\mathrm{new}}=0.6-0.5(-0.014526308)=0.607263154 $$ $$ b_{h_3,\mathrm{new}}=0.6-0.5(0.012994055)=0.593502972 $$ $$ b_{h_4,\mathrm{new}}=0.6-0.5(-0.004282747)=0.602141373 $$ $$ b_{o_1,\mathrm{new}}=0.2-0.5(-0.093597941)=0.246798971 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Required result:** output **0.683621675**, MSE **0.046819580**, h₁ → output new weight **0.533271705**. If instead each printed row is assumed to receive inputs, the conditional alternative is output 0.681090691, MSE 0.047921285 and updated weight 0.534478309. Explain the arrow convention rather than mixing these results. --- ## TEE · R24 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s5 ## Question **Question.** Compare standard momentum and Nesterov accelerated gradient using their gradient-evaluation points and update equations. Explain what may happen when momentum is extremely close to 1, for example 0.9999. **Syllabus-aligned extract:** source R24 Q2b. The current-syllabus optimizer comparison and high-momentum discussion are retained; marks for this extract are not separately stated. ## Answer — step by step Use signed velocity. Standard momentum evaluates g at current θ: vnew = βv−ηg(θ), θ new = θ+vnew. Nesterov evaluates at a look-ahead point: θ look = θ+βv; vnew = βv−ηg(θ look); θ new = θ+vnew. Thus the gradient can correct a direction after anticipating part of the momentum move. Nesterov often reduces overshoot in suitable settings, but it is not universally faster for every neural network. With β = 0.9999, old velocity decays very slowly. The method can respond sluggishly to direction changes, overshoot or oscillate, particularly with a large learning rate. An approximate memory scale 1/(1−β) = 10,000 steps illustrates the long persistence; it is not an exact convergence time. --- ## TEE · R24 Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s6 ## Question **Question (10 marks).** Differentiate sigmoid and ReLU(4). Explain why sigmoid/tanh are often avoided in deep hidden layers(3). Compute softmax of [2.1,4.8,3.5](3). ## Answer — step by step ### Given Softmax scores = [2.1,4.8,3.5]. The numerals (4),(3),(3) in the prompt are marks allocations, not activation inputs. ### To find Differentiate sigmoid/ReLU, explain hidden-layer saturation and calculate softmax. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. **Step 1 stabilize:** subtract the largest logit 4.8 from every value: [-2.7, 0, -1.3]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.067205513, 1, 0.272531793]. **Step 3 denominator:** sum = 1.339737306. **Step 4 divide each value by this same sum:** probabilities = **[0.050163202, 0.746414984, 0.203421814]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. --- ## TEE · R24 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s7 ## Question **Question (10 marks).** Explain why regularization matters, list techniques and elaborate one. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. **Elaborating L2:** add(λ/2)Σw² to the data loss. For weight w = .5, data gradient.2, λ = .1, η = .1, total gradient = .2+.1 × .5 = .25, so wnew = .5−.1 × .25 = .475. Relative to the data-only update.48, the penalty pulls the weight further toward 0. Too large λ can remove useful signal, so tune on validation data. --- ## TEE · R24 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s8 ## Question **Question (10 marks).** Train a perceptron for 2 epochs with η = 1, initial weights and bias 0, threshold 0. Samples in order: [1,1,1,1] → 1; [−1,1,−1,−1] → 1; [1,1,1,−1] → −1; [1,−1,−1,1] → −1. Bias input is 1. ## Answer — step by step ### Given Train a perceptron for 2 epochs with η = 1, initial weights and bias 0, threshold 0. Samples in order: [1,1,1,1] → 1. [−1,1,−1,−1] → 1. [1,1,1,−1] → −1. [1,−1,−1,1] → −1. Bias input is 1. ### To find Process samples in order for the stated epochs and report the final weights and bias. ### Formula — which one and why **Target-based perceptron correction.** This worked convention changes parameters only on a wrong prediction. Use the target itself as the signed multiplier; it differs from the binary error rule. t is the bipolar target; η is the learning rate. Output is +1 above zero, −1 below zero and 0 at exactly zero. $$ \Delta w=\eta t x,\quad\Delta b=\eta t\quad\text{on a mismatch} $$ **Yaad rakhne ka cue:** Is convention mein galat hua toh target × input se correction. ### Solution — step by step **State the threshold rule.** Use output +1 if z>0, −1 if z<0, and 0 at z = 0. On a mismatch use Δw = η tx and Δb = ηt; on a match make no update. η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly. ### Epoch 1, sample 1 **Step 1: copy this sample and target.** x = [1, 1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (0 × 1) + (0 × 1) + (0 × 1) + (0 × 1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [1, 1, 1, 1] = [1, 1, 1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [1, 1, 1, 1], new b = 1. ### Epoch 1, sample 2 **Step 1: copy this sample and target.** x = [-1, 1, -1, -1], target = 1. **Step 2: calculate the current net.** z = (1 × -1) + (1 × 1) + (1 × -1) + (1 × -1) + (1) = -1 **Step 3: apply the threshold.** output = -1. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [-1, 1, -1, -1] = [-1, 1, -1, -1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [0, 2, 0, 0], new b = 2. ### Epoch 1, sample 3 **Step 1: copy this sample and target.** x = [1, 1, 1, -1], target = -1. **Step 2: calculate the current net.** z = (0 × 1) + (2 × 1) + (0 × 1) + (0 × -1) + (2) = 4 **Step 3: apply the threshold.** output = 1. **Step 4: calculate the update multiplier.** Update multiplier = -1 **Step 5: calculate all weight changes.** Δw = -1 × [1, 1, 1, -1] = [-1, -1, -1, 1] **Step 6: calculate the bias change.** Δb = -1. **Step 7: carry the updated parameters forward.** New w = [-1, 1, -1, 1], new b = 1. ### Epoch 1, sample 4 **Step 1: copy this sample and target.** x = [1, -1, -1, 1], target = -1. **Step 2: calculate the current net.** z = (-1 × 1) + (1 × -1) + (-1 × -1) + (1 × 1) + (1) = 1 **Step 3: apply the threshold.** output = 1. **Step 4: calculate the update multiplier.** Update multiplier = -1 **Step 5: calculate all weight changes.** Δw = -1 × [1, -1, -1, 1] = [-1, 1, 1, -1] **Step 6: calculate the bias change.** Δb = -1. **Step 7: carry the updated parameters forward.** New w = [-2, 2, 0, 0], new b = 0. ### Epoch 2, sample 1 **Step 1: copy this sample and target.** x = [1, 1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (-2 × 1) + (2 × 1) + (0 × 1) + (0 × 1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [1, 1, 1, 1] = [1, 1, 1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [-1, 3, 1, 1], new b = 1. ### Epoch 2, sample 2 **Step 1: copy this sample and target.** x = [-1, 1, -1, -1], target = 1. **Step 2: calculate the current net.** z = (-1 × -1) + (3 × 1) + (1 × -1) + (1 × -1) + (1) = 3 **Step 3: apply the threshold.** output = 1. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, -1, -1] = [0, 0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [-1, 3, 1, 1], new b = 1. ### Epoch 2, sample 3 **Step 1: copy this sample and target.** x = [1, 1, 1, -1], target = -1. **Step 2: calculate the current net.** z = (-1 × 1) + (3 × 1) + (1 × 1) + (1 × -1) + (1) = 3 **Step 3: apply the threshold.** output = 1. **Step 4: calculate the update multiplier.** Update multiplier = -1 **Step 5: calculate all weight changes.** Δw = -1 × [1, 1, 1, -1] = [-1, -1, -1, 1] **Step 6: calculate the bias change.** Δb = -1. **Step 7: carry the updated parameters forward.** New w = [-2, 2, 0, 2], new b = 0. ### Epoch 2, sample 4 **Step 1: copy this sample and target.** x = [1, -1, -1, 1], target = -1. **Step 2: calculate the current net.** z = (-2 × 1) + (2 × -1) + (0 × -1) + (2 × 1) + (0) = -2 **Step 3: apply the threshold.** output = -1. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [1, -1, -1, 1] = [0, 0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [-2, 2, 0, 2], new b = 0. **Final after exactly the requested epochs:** w = **[-2, 2, 0, 2]**, b = **0**. A finite number of requested epochs does not itself prove convergence or correct classification of every sample. --- ## TEE · R24 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s9 ## Question **Question (10 marks).** Calculate weights, biases and feature-map sizes of the CNN: Input 227 × 227 × 3. → Conv K = 9, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 5, filters = 64, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 120 units. → Dense 84 units. → Dense 10 units. ## Answer — step by step ### Given Input 227 × 227 × 3. Conv K = 9, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 64, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 5, filters = 64, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Dense 120 units. Dense 84 units. Dense 10 units. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**227 × 227 × 3**height × width × channels → Convolution**32 filters · 9 × 9 · S = 1 · P = 0** → Output shape**219 × 219 × 32**height × width × channels Weights**7,776** + Biases**32** = Trainable parameters**7,808** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **227 × 227 × 3**. Use 32 filters of spatial size 9 × 9, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-9}{1}\right\rfloor+1=219 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **219 × 219 × 32**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=9\times9\times3\times32=7{,}776 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_1=7{,}776+32=7{,}808 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**219 × 219 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**109 × 109 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **219 × 219 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{219-2}{2}\right\rfloor+1=109 $$ Pooling acts independently on each channel. The output is **109 × 109 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 108.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**109 × 109 × 32**height × width × channels → Convolution**64 filters · 5 × 5 · S = 1 · P = 0** → Output shape**105 × 105 × 64**height × width × channels Weights**51,200** + Biases**64** = Trainable parameters**51,264** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **109 × 109 × 32**. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{109+2(0)-5}{1}\right\rfloor+1=105 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **105 × 105 × 64**. **3. Count trainable values.** Each filter receives all 32 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times32\times64=51{,}200 $$ $$ N_{\mathrm{biases}}=64 $$ $$ P_3=51{,}200+64=51{,}264 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**105 × 105 × 64**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**52 × 52 × 64**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **105 × 105 × 64**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{105-2}{2}\right\rfloor+1=52 $$ Pooling acts independently on each channel. The output is **52 × 52 × 64**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 51.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**52 × 52 × 64**height × width × channels → Convolution**64 filters · 5 × 5 · S = 1 · P = 0** → Output shape**48 × 48 × 64**height × width × channels Weights**102,400** + Biases**64** = Trainable parameters**102,464** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **52 × 52 × 64**. Use 64 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{52+2(0)-5}{1}\right\rfloor+1=48 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **48 × 48 × 64**. **3. Count trainable values.** Each filter receives all 64 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times64\times64=102{,}400 $$ $$ N_{\mathrm{biases}}=64 $$ $$ P_5=102{,}400+64=102{,}464 $$ ### Layer 6 — pooling Layer 6: what changes and what is learned Input shape**48 × 48 × 64**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**24 × 24 × 64**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **48 × 48 × 64**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{48-2}{2}\right\rfloor+1=24 $$ Pooling acts independently on each channel. The output is **24 × 24 × 64**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 7 — flatten Layer 7: what changes and what is learned Input shape**24 × 24 × 64**height × width × channels → Flatten**Rearrange activations** → Output shape**36,864**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **24 × 24 × 64**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=24\times24\times64=36{,}864 $$ Output: a vector with **36,864 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**36,864**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**4,423,680** + Biases**120** = Trainable parameters**4,423,800** Every output neuron receives every input feature, with one bias per output neuron. Input: **36,864 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=36{,}864\times120=4{,}423{,}680 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_8=4{,}423{,}680+120=4{,}423{,}800 $$ ### Layer 9 — fully connected layer Layer 9: what changes and what is learned Input shape**120**input features → Dense layer**84 neurons** → Output shape**84**output features Weights**10,080** + Biases**84** = Trainable parameters**10,164** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **84 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times84=10{,}080 $$ $$ N_{\mathrm{biases}}=84 $$ $$ P_9=10{,}080+84=10{,}164 $$ ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**84**input features → Dense layer**10 neurons** → Output shape**10**output features Weights**840** + Biases**10** = Trainable parameters**850** Every output neuron receives every input feature, with one bias per output neuron. Input: **84 features**. Output: **10 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=84\times10=840 $$ $$ N_{\mathrm{biases}}=10 $$ $$ P_10=840+10=850 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{227+2 \times 0-9}{1}\right\rfloor+1$ | 219 x 219 x 32 | 7,776 | 32 | 7,808 | | 2 pool | $\left\lfloor\frac{219-2}{2}\right\rfloor+1$ | 109 x 109 x 32 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{109+2 \times 0-5}{1}\right\rfloor+1$ | 105 x 105 x 64 | 51,200 | 64 | 51,264 | | 4 pool | $\left\lfloor\frac{105-2}{2}\right\rfloor+1$ | 52 x 52 x 64 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{52+2 \times 0-5}{1}\right\rfloor+1$ | 48 x 48 x 64 | 102,400 | 64 | 102,464 | | 6 pool | $\left\lfloor\frac{48-2}{2}\right\rfloor+1$ | 24 x 24 x 64 | 0 | 0 | 0 | | 7 flat | 24 × 24 × 64 | 36864 | 0 | 0 | 0 | | 8 dense | 36864 → 120 | 120 | 4,423,680 | 120 | 4,423,800 | | 9 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 | | 10 dense | 84 → 10 | 10 | 840 | 10 | 850 | **Parameter substitutions, in layer order:** Layer 1: (9 x 9 x 3 + 1) x 32 = 7,808. Layer 3: (5 x 5 x 32 + 1) x 64 = 51,264. Layer 5: (5 x 5 x 64 + 1) x 64 = 102,464. Layer 8: (36864 + 1) x 120 = 4,423,800. Layer 9: (120 + 1) x 84 = 10,164. Layer 10: (84 + 1) x 10 = 850. **Total:** 7,808 + 51,264 + 102,464 + 4,423,800 + 10,164 + 850 = **4,596,350 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. --- ## TEE · R24 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s10 ## Question **Question (10 marks).** A GoogLeNet classifier trains slowly and vanishing gradients are suspected. Suggest an architecture change and explain its important features. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. A ResNet is a suitable architecture to investigate because skip additions provide shorter gradient paths. First verify the diagnosis with gradients and training curves; slow training alone does not establish vanishing gradients. Match projection shortcuts when spatial size/channels change and compare validation performance after the redesign. --- ## TEE · R24 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s11 ## Question **Question (10 marks).** (i) Justify DL for chest-cancer diagnosis from X-ray images(5). (ii) List RNN advantages and disadvantages(5). ## Answer — step by step A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification. ![One filter is reused at successive spatial positions](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: One filter is reused at successive spatial positions. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. One filter is reused at successive spatial positions Tap the diagram to enlarge. Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities. Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation. For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions. **RNN advantages:** ordered context, shared weights over time, variable sequence lengths and flexible input/output mappings. **Limitations:** sequential computation reduces time-step parallelism; vanilla cells have vanishing/exploding gradient issues and limited learned long context; many training choices still require validation. Use CNNs for spatial image patterns and recurrent models when temporal/ordered structure is relevant. --- ## TEE · R24 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s12 ## Question **Question (10 marks).** For a GRU time step, explain the significance of update and reset gates and develop their equations. ## Answer — step by step A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate: $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ ![Reset controls candidate context and update controls the old new mixture](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: Reset controls candidate context and update controls the old new mixture. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. Reset controls candidate context and update controls the old new mixture Tap the diagram to enlarge. The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate. Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN. --- ## TEE · R24 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s13 ## Question **Question (10 marks).** A supermarket vegetable classifier has 10 classes but few examples per class and limited training resources. Explain different ways to use a pretrained model. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small vegetable-image dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. Use independently captured vegetable examples in each split; keep near-duplicates or repeated images of the same item together. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on a held-out vegetable test set and inspect the confusion matrix and per-class errors. Check which vegetables are confused under different lighting or backgrounds. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. For vegetables, replace the source head with 10 softmax outputs. Strategy 1: freeze the base and train only this head. Strategy 2: initialize from the same pretrained model and fine-tune selected later blocks at a small rate after head training. The first is cheaper and less prone to fitting all parameters from limited examples; the second can adapt features more closely to vegetables when validation supports it. --- ## TEE · R24 Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s14 ## Question **Question (10 marks).** Draw and explain LSTM architecture. ## Answer — step by step An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. --- ## TEE · R24 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c14-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/14/tee-c14-s15 ## Question **Question (10 marks).** Explain different RNN input/output sequence types with architecture sketches. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. ![One-to-many many-to-one and two many-to-many arrangements](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: One-to-many many-to-one and two many-to-many arrangements. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. One-to-many many-to-one and two many-to-many arrangements Tap the diagram to enlarge. One-to-many: one image representation generates a caption. Many-to-one: a review produces one sentiment. Aligned many-to-many: each video frame receives a label. Unaligned many-to-many: an encoder reads a source sentence and a decoder generates a translation of possibly different length. Specify which outputs carry losses during training; variable lengths do not require new weights for each position. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-intro F22 = S16, 19 November 2022. The older LeNet calculation is retained as standard CNN arithmetic and because the current course policy includes a LeNet lab. Historical diagram ambiguities are stated rather than silently guessed. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · F22 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s0 ## Question **Question (4 marks).** A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3]. Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding. Compute the output and identify the feature. ## Answer — step by step ### Given A normalized 5 × 5 image has every row [0.5,0.5,0.3,0.3,0.3]. Convolve it with a 3 × 3 filter whose every row is [1,0,−1], stride 2, no padding. ### To find Find every convolution output cell. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{5+2 \times 0-3}{2}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.5 + 0 + (−0.3) = 0.2** Row 2**0.5 + 0 + (−0.3) = 0.2** Row 3**0.5 + 0 + (−0.3) = 0.2** **0.2 + 0.2 + 0.2 = 0.6** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 1, column 1. **Window row 1.** $$ R_1=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 2.** $$ R_2=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 3.** $$ R_3=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(0.2)+(0.2)+(0.2)=0.6 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.3 + 0 + (−0.3) = 0** Row 2**0.3 + 0 + (−0.3) = 0** Row 3**0.3 + 0 + (−0.3) = 0** **0 + 0 + 0 = 0** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 1, column 2. **Window row 1.** $$ R_1=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 3.** $$ R_3=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(0)+(0)+(0)=0 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | | 0.5 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.5 + 0 + (−0.3) = 0.2** Row 2**0.5 + 0 + (−0.3) = 0.2** Row 3**0.5 + 0 + (−0.3) = 0.2** **0.2 + 0.2 + 0.2 = 0.6** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 2, column 1. **Window row 1.** $$ R_1=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 2.** $$ R_2=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Window row 3.** $$ R_3=(0.5)\times(1)+(0.5)\times(0)+(0.3)\times(-1)=0.2 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0.2)+(0.2)+(0.2)=0.6 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | | 0.5 | 0.5 | 0.3 | 0.3 | 0.3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 1 | 0 | −1 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | | 0.3 | 0 | −0.3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0.3 + 0 + (−0.3) = 0** Row 2**0.3 + 0 + (−0.3) = 0** Row 3**0.3 + 0 + (−0.3) = 0** **0 + 0 + 0 = 0** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 0.6 | 0 | | 0.6 | 0 | Output row 2, column 2. **Window row 1.** $$ R_1=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Window row 3.** $$ R_3=(0.3)\times(1)+(0.3)\times(0)+(0.3)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(0)+(0)+(0)=0 $$ **Final output:** $\begin{bmatrix}0.6 & 0\\0.6 & 0\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. The filter compares left and right neighborhoods and responds to a vertical brightness boundary. Inputs are already normalized. No additional output-normalization rule is supplied, so the determined convolution is $\begin{bmatrix}0.6 & 0\\0.6 & 0\end{bmatrix}$ . If an examiner explicitly requests min-max normalization of this result, divide by 0.6 to obtain $\begin{bmatrix}1 & 0\\1 & 0\end{bmatrix}$ ; that is a separate stated operation. --- ## TEE · F22 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s1 ## Question **Question (4 marks).** Weights are 0.55 and 0.65; their gradients are+0.3 and−0.25. With η = 0.1, compute gradient-descent changes and new weights. ## Answer — step by step ### Given Old weights = [0.55,0.65]. Corresponding supplied gradients = [0.3,−0.25]; η = 0.1. ### To find Calculate the signed weight changes and both new weights. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1: calculate the change in weight 1.** $$ \Delta w_1=-0.1(0.3)=-0.03 $$ **Step 2: add that change to old weight 1.** $$ w_1^{\mathrm{new}}=0.55+(-0.03)=0.52 $$ **Step 3: calculate the change in weight 2.** $$ \Delta w_2=-0.1(-0.25)=0.025 $$ **Step 4: add that change to old weight 2.** $$ w_2^{\mathrm{new}}=0.65+0.025=0.675 $$ **Final answer:** changes = [−0.03, +0.025]; new weights = [0.52, 0.675]. A negative gradient increases its weight under descent. --- ## TEE · F22 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s2 ## Question **Question (4 marks).** Compare ML and DL algorithms. ## Answer — step by step Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. --- ## TEE · F22 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s3 ## Question **Question (4 marks).** Compare sigmoid and tanh, emphasizing tanh’s advantage. ## Answer — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. The requested advantage is that tanh is zero-centered, so hidden activations can be positive or negative. Sigmoid activations lie above 0. Both functions saturate and can produce small derivatives; zero-centering does not eliminate the vanishing-gradient problem. --- ## TEE · F22 Q1e Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s4 ## Question **Question (4 marks).** Suggest and sketch an RNN architecture for image captioning. ## Answer — step by step Use a CNN encoder to map the image to a visual representation and a recurrent decoder (RNN/LSTM/GRU) to generate words one at a time. Condition the decoder’s initial state or its inputs on image features. At each step, vocabulary softmax predicts the next word based on the image and earlier words. Train using correct next-word targets and cross-entropy; stop generation at an end token. This is a one-to-many mapping from one image to a caption sequence. ![The one-to-many pattern models image captioning](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: The one-to-many pattern models image captioning. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. The one-to-many pattern models image captioning Tap the diagram to enlarge. --- ## TEE · F22 Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s5 ## Question **Question (10 marks).** Explain GRU advantages over a normal RNN. For “The student who did not submit the assignment was bound to loose marks,” show how memory and update gates can retain the subject before predicting “was”. ## Answer — step by step A GRU uses an update gate and reset gate, with one hidden state and no separate LSTM-style cell state. Under the convention used here, z admits the new candidate: $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ ![Reset controls candidate context and update controls the old new mixture](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: Reset controls candidate context and update controls the old new mixture. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. Reset controls candidate context and update controls the old new mixture Tap the diagram to enlarge. The reset gate controls how much prior hidden information enters the candidate. The update gate balances retention and replacement. A small z retains most of the old state under this formula; another textbook convention swaps z and 1−z. State the formula before interpreting the gate. Compared with an LSTM's input, forget and output gates plus candidate transform, a standard GRU has two gates plus a candidate transform and usually fewer parameters at equal hidden width. This may reduce training cost and the amount of data needed to estimate parameters, but small-dataset superiority is not guaranteed. Validation should decide. To ignore an unhelpful token, a trained GRU can keep the prior state by admitting little candidate content; it still executes the recurrence rather than literally deleting the token. Both GRU and LSTM help long-range learning compared with an ungated vanilla RNN. **Important data limit:** no numerical weights or token vectors are supplied, so actual gate outputs cannot be uniquely calculated. The following is an explicitly idealized one-component illustration: candidate subject feature = 1 at “student”; use hnew = (1−z) hold+z candidate. Exact 0/1 gates are limiting idealizations of sigmoid values. | Time and token | Illustrative z | Singular-subject memory after step | | --- | --- | --- | | 1 The | 0 while retaining this illustrative feature | 0 before subject is read | | 2 student | 1 when admitting subject feature | 1 after subject is read | | 3 who | 0 while retaining this illustrative feature | 1 after subject is read | | 4 did | 0 while retaining this illustrative feature | 1 after subject is read | | 5 not | 0 while retaining this illustrative feature | 1 after subject is read | | 6 submit | 0 while retaining this illustrative feature | 1 after subject is read | | 7 the | 0 while retaining this illustrative feature | 1 after subject is read | | 8 assignment | 0 while retaining this illustrative feature | 1 after subject is read | At “student”: hnew = (1−1) × 0+1 × 1 = 1. At each intervening token in this illustrative component: hnew = (1−0) × 1+0 × candidate = 1. The retained singular subject helps the output layer choose “was”. Other hidden components may still update to represent intervening words; this table is not a computed trace from an actual trained network. --- ## TEE · F22 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s6 ## Question **Question (10 marks).** For a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1. First-stage feature multipliers are [0.3,0.4,0.6]. Next receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3]. Find both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2. Assume zero biases. ## Answer — step by step ### Given For a three-stage sigmoid network, use samples [0.3,0.2,0.4] → 0 and [0.5,0.6,0.8] → 1. First-stage feature multipliers are [0.3,0.4,0.6]. Next receiving rows are [0.5,0.2,0.3], [0.3,0.3,0.3], and output weights [0.3,0.3]. Find both predictions, average MSE and output-weight updates after the first backward pass with η = 0.2. Assume zero biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Mean squared loss.** Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean. n = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target. $$ L=\frac1n\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=\frac2n(o_j-y_j) $$ **Yaad rakhne ka cue:** MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Interpretation.** Follow the supplied diagram: each raw feature is multiplied by 0.3, 0.4 or 0.6, then passed through sigmoid before the next two-neuron layer. Unspecified biases are zero. MSE is averaged over the two samples. The phrase “first backward pass” does not specify batch versus single-sample updating, so both interpretations are identified below. ### Step 1: apply the first-stage multipliers and sigmoid For sample 1, the preactivations are [0.3 × 0.3, 0.2 × 0.4, 0.4 × 0.6] = [0.09, 0.08, 0.24]. Sigmoid gives [0.522484825, 0.519989340, 0.559713649]. For sample 2, they are [0.5 × 0.3, 0.6 × 0.4, 0.8 × 0.6] = [0.15, 0.24, 0.48]. Sigmoid gives [0.537429845, 0.559713649, 0.617747875]. ### Step 2: evaluate the two hidden neurons For each first-stage output a, calculate z₁ = 0.5a₁ + 0.2a₂ + 0.3a₃ and z₂ = 0.3a₁ + 0.3a₂ + 0.3a₃, then apply sigmoid. | Sample | Hidden preactivation 1 | Hidden preactivation 2 | Hidden output 1 | Hidden output 2 | | --- | --- | --- | --- | --- | | 1 | 0.533154375 | 0.480656344 | 0.630218519 | 0.617902849 | | 2 | 0.565982015 | 0.514467411 | 0.637835530 | 0.625853155 | For example, sample 1's first hidden sum is 0.5(0.522484825) + 0.2(0.519989340) + 0.3(0.559713649) = 0.533154375. ### Step 3: evaluate the output neuron The output preactivation is 0.3h₁ + 0.3h₂; apply sigmoid to the complete sum. | Sample | Output preactivation | Prediction | Target | Squared error | | --- | --- | --- | --- | --- | | 1 | 0.374436410 | 0.592530535 | 0 | 0.351092435 | | 2 | 0.379106605 | 0.593657609 | 1 | 0.165114139 | ### Step 4: calculate the average MSE MSE = (0.351092435 + 0.165114139)/2 = **0.258103287**. ### Step 5: differentiate each sample's squared error For one sample, output delta = 2(o − y)o(1 − o); each output-weight gradient is this delta multiplied by its sending hidden activation. Calculate both samples at the original weights before taking their mean. | Sample | Output delta | Gradient for h₁ → output | Gradient for h₂ → output | | --- | --- | --- | --- | | 1 | 0.286118893 | 0.180317425 | 0.176793679 | | 2 | −0.196042530 | −0.125042891 | −0.122693836 | ### Step 6: average gradients and make one batch update The mean gradients are [(0.180317425 − 0.125042891)/2, (0.176793679 − 0.122693836)/2] ≈ [0.027637267, 0.027049922]. First weight: 0.3 − 0.2(0.027637267) = **0.294472547**. Second weight: 0.3 − 0.2(0.027049922) = **0.294590016**. **Final answer under batch MSE:** predictions [0.592530535, 0.593657609], average MSE 0.258103287, updated output weights [0.294472547, 0.294590016]. Averaging the sample gradients is required because the stated objective is their mean loss. ### Alternative: first-sample SGD If “first backward pass” means an update after sample 1 alone, use its gradients without averaging: 0.3 − 0.2(0.180317425) = **0.263936515** and 0.3 − 0.2(0.176793679) = **0.264641264**. This is a different update schedule. State the interpretation and do not combine a batch loss with an unexplained single-sample gradient. --- ## TEE · F22 Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s7 ## Question **Question (10 marks).** Explain bias, variance, underfitting and overfitting. A four-feature product-texture classifier has four grades including reject. Choose output activation/loss. Diagnose training error 15% and validation error 18% and suggest remedies. ## Answer — step by step Bias is error caused by restrictive assumptions or insufficient fit; variance is sensitivity to the particular training sample. A high-bias model may miss useful structure and perform poorly even on training data. A high-variance model can fit training data very well but perform much worse on comparable validation data. ![Validation behavior distinguishes useful fitting from prolonged fitting to training details](https://deep-learning-study-room.vercel.app/assets/figures/overfitting.svg) Diagram: Validation behavior distinguishes useful fitting from prolonged fitting to training details. Labels: Loss | Epochs | Validation | Training | Best validation region | Save the best validation checkpoint; do not choose by test loss. Validation behavior distinguishes useful fitting from prolonged fitting to training details Tap the diagram to enlarge. Increasing model capacity can reduce bias but may increase variance. Stronger regularization usually increases bias and reduces variance. More representative training data often reduces variance without changing the model's approximation bias much. These are typical trends, not unconditional numerical laws. Underfitting remedies include improving features, using a more expressive appropriate model, training sufficiently and reducing excessive regularization. Overfitting remedies include representative extra data, data augmentation, appropriate L1/L2 or dropout, early stopping and sometimes reducing model size. Keep an untouched test set for final evaluation. Repeatedly tuning on test results turns that set into another validation set and requires a fresh test set for an unbiased final estimate. Training and validation errors alone do not determine a diagnosis without task difficulty, evaluation consistency and learning curves. A small gap with both errors high suggests underfitting; a large gap with very low training error suggests overfitting. **Classifier:** four mutually exclusive grades require 4 output scores, softmax probabilities and categorical cross-entropy. A reject class is still a labeled class in this formulation. **Diagnosis:** a modest 3 percentage-point gap with both errors appreciable suggests underfitting/high bias if a much better attainable baseline exists. Confirm learning curves and label quality. Improve relevant features or model capacity, train sufficiently, tune the optimizer and reduce excessive regularization. The percentages alone do not prove the cause. --- ## TEE · F22 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s8 ## Question **Question (10 marks).** Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2; convolution 5 × 5: 16 filters, pooling 2; dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps. ## Answer — step by step ### Given Count trainable parameters of the illustrated LeNet-style digit classifier with convolution 5 × 5: 6 filters, pooling 2. convolution 5 × 5: 16 filters, pooling 2. dense 120, dense 84, softmax 10. Use the consistent 32 × 32 × 1 input that produces the drawn 28,14,10,5 spatial maps. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**32 × 32 × 1**height × width × channels → Convolution**6 filters · 5 × 5 · S = 1 · P = 0** → Output shape**28 × 28 × 6**height × width × channels Weights**150** + Biases**6** = Trainable parameters**156** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **32 × 32 × 1**. Use 6 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{32+2(0)-5}{1}\right\rfloor+1=28 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **28 × 28 × 6**. **3. Count trainable values.** Each filter receives all 1 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times1\times6=150 $$ $$ N_{\mathrm{biases}}=6 $$ $$ P_1=150+6=156 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**28 × 28 × 6**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**14 × 14 × 6**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **28 × 28 × 6**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{28-2}{2}\right\rfloor+1=14 $$ Pooling acts independently on each channel. The output is **14 × 14 × 6**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**14 × 14 × 6**height × width × channels → Convolution**16 filters · 5 × 5 · S = 1 · P = 0** → Output shape**10 × 10 × 16**height × width × channels Weights**2,400** + Biases**16** = Trainable parameters**2,416** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **14 × 14 × 6**. Use 16 filters of spatial size 5 × 5, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{14+2(0)-5}{1}\right\rfloor+1=10 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **10 × 10 × 16**. **3. Count trainable values.** Each filter receives all 6 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times6\times16=2{,}400 $$ $$ N_{\mathrm{biases}}=16 $$ $$ P_3=2{,}400+16=2{,}416 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**10 × 10 × 16**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**5 × 5 × 16**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **10 × 10 × 16**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{10-2}{2}\right\rfloor+1=5 $$ Pooling acts independently on each channel. The output is **5 × 5 × 16**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — flatten Layer 5: what changes and what is learned Input shape**5 × 5 × 16**height × width × channels → Flatten**Rearrange activations** → Output shape**400**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **5 × 5 × 16**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=5\times5\times16=400 $$ Output: a vector with **400 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 6 — fully connected layer Layer 6: what changes and what is learned Input shape**400**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**48,000** + Biases**120** = Trainable parameters**48,120** Every output neuron receives every input feature, with one bias per output neuron. Input: **400 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=400\times120=48{,}000 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_6=48{,}000+120=48{,}120 $$ ### Layer 7 — fully connected layer Layer 7: what changes and what is learned Input shape**120**input features → Dense layer**84 neurons** → Output shape**84**output features Weights**10,080** + Biases**84** = Trainable parameters**10,164** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **84 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times84=10{,}080 $$ $$ N_{\mathrm{biases}}=84 $$ $$ P_7=10{,}080+84=10{,}164 $$ ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**84**input features → Dense layer**10 neurons** → Output shape**10**output features Weights**840** + Biases**10** = Trainable parameters**850** Every output neuron receives every input feature, with one bias per output neuron. Input: **84 features**. Output: **10 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=84\times10=840 $$ $$ N_{\mathrm{biases}}=10 $$ $$ P_8=840+10=850 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{32+2 \times 0-5}{1}\right\rfloor+1$ | 28 x 28 x 6 | 150 | 6 | 156 | | 2 pool | $\left\lfloor\frac{28-2}{2}\right\rfloor+1$ | 14 x 14 x 6 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{14+2 \times 0-5}{1}\right\rfloor+1$ | 10 x 10 x 16 | 2,400 | 16 | 2,416 | | 4 pool | $\left\lfloor\frac{10-2}{2}\right\rfloor+1$ | 5 x 5 x 16 | 0 | 0 | 0 | | 5 flat | 5 × 5 × 16 | 400 | 0 | 0 | 0 | | 6 dense | 400 → 120 | 120 | 48,000 | 120 | 48,120 | | 7 dense | 120 → 84 | 84 | 10,080 | 84 | 10,164 | | 8 dense | 84 → 10 | 10 | 840 | 10 | 850 | **Parameter substitutions, in layer order:** Layer 1: (5 x 5 x 1 + 1) x 6 = 156. Layer 3: (5 x 5 x 6 + 1) x 16 = 2,416. Layer 6: (400 + 1) x 120 = 48,120. Layer 7: (120 + 1) x 84 = 10,164. Layer 8: (84 + 1) x 10 = 850. **Total:** 156 + 2,416 + 48,120 + 10,164 + 850 = **61,706 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. This is the standard classroom full-channel interpretation with nontrainable pooling and a dense 10-class softmax. The historical original LeNet used partial channel connections and other details, so its published parameter count need not equal this diagram-based count. If an input label is read as 28, the stated later 28 map after valid 5 × 5 convolution is inconsistent; record the 32 input assumption rather than silently ignoring the mismatch. --- ## TEE · F22 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s9 ## Question **Question (10 marks).** An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. Compute operations and explain 1 × 1 convolution. ## Answer — step by step ### Given An Inception branch receives 28 × 28 × 192, uses 16 filters of 1 × 1 then 32 filters of 5 × 5 with spatial size preserved. ### To find Find direct and bottleneck computation counts and the requested saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Direct path **Step 2: direct path.** output stays 28 × 28 with 32 channels. **Step 3: direct path.** MACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**. ### Bottleneck path with 16 reduction channels **Step 4: bottleneck path with 16 reduction channels.** 1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs. **Step 5: bottleneck path with 16 reduction channels.** The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs. **Step 6: bottleneck path with 16 reduction channels.** Total = **12,443,648**. ### Saving **Step 7: saving.** 120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r** **Step 8: saving.** there is no unique numeric bottleneck cost until r is chosen. **Step 9: calculate this quantity.** An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. **Step 10: calculate this quantity.** A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. --- ## TEE · F22 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s10 ## Question **Question (10 marks).** Draw a 6-input, two-hidden-layer network with 5 neurons each and 4 classes, trained by AdaGrad. Identify learned parameters and tunable hyperparameters; no input-layer weights or biases. ## Answer — step by step ### Given 6 inputs → hidden 5 → hidden 5 → 4 output classes. AdaGrad is the specified training algorithm; one bias per receiving neuron. ### To find Draw the architecture and count its learned weights/biases. The optimizer name does not change the count. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step Architecture: 6 → 5 → 5 → 4. Each non-input neuron has one bias. ![F22 Q4b the input has six features and the output has four classes](https://deep-learning-study-room.vercel.app/assets/figures/dense_6_5_5_4.svg) Diagram: F22 Q4b the input has six features and the output has four classes. Labels: Input | 6 neurons | Hidden 1 | 5 neurons | Hidden 2 | 5 neurons | Output | 4 neurons | Every receiving neuron has one bias. All adjacent layers are fully connected. F22 Q4b the input has six features and the output has four classes Tap the diagram to enlarge. ### Layer 6 → 5 **Step 1: count connections.** $$ 6(5)=30 $$ **Step 2: count receiving-neuron biases.** There are 5 receiving neurons, so 5 biases. **Step 3: add weights and biases.** $$ 30+5=35 $$ ### Layer 5 → 5 **Step 1: count connections.** $$ 5(5)=25 $$ **Step 2: count receiving-neuron biases.** There are 5 receiving neurons, so 5 biases. **Step 3: add weights and biases.** $$ 25+5=30 $$ ### Layer 5 → 4 **Step 1: count connections.** $$ 5(4)=20 $$ **Step 2: count receiving-neuron biases.** There are 4 receiving neurons, so 4 biases. **Step 3: add weights and biases.** $$ 20+4=24 $$ **Total = 89**. Input values and activations are not trainable parameters. The learned parameters are input-to-hidden, hidden-to-hidden and hidden-to-output weights plus biases at receiving neurons. AdaGrad accumulators are optimizer state, not additional predictive weights. Hyperparameters include hidden widths/depth, learning rate η,ε, batch size, epochs, activations and regularization settings. Use 4-class softmax with categorical cross-entropy. AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise: $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change. Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864. Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average. --- ## TEE · F22 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s11 ## Question **Question (10 marks).** Develop an RNN with hidden states and explain gradient updates using backpropagation. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph. 1. Initialize the hidden state and run the forward recurrence from the first input to the last. 2. Compute the specified output loss, at the final step or across several steps. 3. Start from the last loss and propagate derivatives backwards through output and hidden operations. 4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states. 5. Sum all uses of each shared weight before applying the optimizer update. $$ \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T} $$ The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step. ![BPTT follows this unrolled network backwards and sums gradients of shared weights](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: BPTT follows this unrolled network backwards and sums gradients of shared weights. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge. --- ## TEE · F22 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s12 ## Question **Question (10 marks).** Explain DL optimization challenges and the advantage of momentum over plain mini-batch gradient descent. ## Answer — step by step The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable. ![Illustrative full batch and stochastic traces distinguish smoothness from progress](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative full batch and stochastic traces distinguish smoothness from progress. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative full batch and stochastic traces distinguish smoothness from progress Tap the diagram to enlarge. Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory. Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories. Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD. Momentum uses vnew = βv−ηg and θ new = θ+vnew; consistent gradients accumulate direction, whereas some alternating components cancel. Plain mini-batch descent uses only the current batch gradient. Both can be combined: momentum with mini-batch gradients is a standard coherent choice. --- ## TEE · F22 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s13 ## Question **Question (10 marks).** A CNN image classifier performs poorly because of vanishing gradients. Explain remedies. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. Also inspect saturated sigmoid/tanh hidden units, initialization and gradient magnitudes. Appropriate ReLU-family activations and initialization can help preserve usable gradients. Residual blocks require shape-compatible additions or projections. Verify that the diagnosis is correct; additional training time alone cannot repair gradients that are consistently negligible. --- ## TEE · F22 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s14 ## Question **Question (10 marks).** Design a clothes classifier for 250,000 six-dimensional samples,6 classes and 2 hidden layers. Give train/validation/test split, activations/loss, actions for 15% training 20% validation error, interpret improvement to 1%/1.5%, then address 15% test error. ## Answer — step by step **One explicit design:** 6 → 16 → 8 → 6, ReLU hidden layers,6-unit softmax output, categorical cross-entropy. The hidden widths are chosen assumptions, not supplied numerical requirements. Architecture: 6 → 16 → 8 → 6. Each non-input neuron has one bias. ### Layer 6 → 16 **Step 1: count connections.** $$ 6(16)=96 $$ **Step 2: count receiving-neuron biases.** There are 16 receiving neurons, so 16 biases. **Step 3: add weights and biases.** $$ 96+16=112 $$ ### Layer 16 → 8 **Step 1: count connections.** $$ 16(8)=128 $$ **Step 2: count receiving-neuron biases.** There are 8 receiving neurons, so 8 biases. **Step 3: add weights and biases.** $$ 128+8=136 $$ ### Layer 8 → 6 **Step 1: count connections.** $$ 8(6)=48 $$ **Step 2: count receiving-neuron biases.** There are 6 receiving neurons, so 6 biases. **Step 3: add weights and biases.** $$ 48+6=54 $$ **Total = 302**. Input values and activations are not trainable parameters. **Split:** a stratified 70/15/15 split gives 175,000 training,37,500 validation,37,500 test samples. Fit preprocessing on training data only and keep related duplicates within a single split. **15%/20%:** inspect learning curves and baseline difficulty. Substantial training error suggests capacity/optimization or data-quality issues; the 5 point gap also suggests a generalization issue. Improve training fit, tune regularization and collect useful representative data instead of blindly increasing capacity. **1%/1.5%:** good comparable-set results with a small gap are encouraging; verify that validation has not leaked into training. **15% test:** inspect distribution shift, preprocessing differences, label quality, leakage or repeated validation overfitting. Compare per-class errors and collection conditions. Address the actual cause and collect representative data if necessary. If test results guide retuning, use a fresh untouched final test set. --- ## TEE · F22 Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s15 ## Question **Question (10 marks).** Explain with an example how LSTM handles long sequences in language modeling. ## Answer — step by step An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. Example: “The student, after several difficult assignments and a long discussion, was relieved.” Useful singular-subject information can be retained in selected cell components while intervening words update other components. At the final step a vocabulary softmax predicts the next token using h. Gates close to 1 on the retained path preserve context; numerical gate values require learned weights and cannot be inferred uniquely from the sentence. --- ## TEE · F22 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c15-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/15/tee-c15-s16 ## Question **Question (10 marks).** Justify that single-layer perceptrons cannot classify nonlinearly separable data using an example, and suggest a solution. ## Answer — step by step A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator. ![Opposite corners for XOR cannot be split by one straight line](https://deep-learning-study-room.vercel.app/assets/figures/xor.svg) Diagram: Opposite corners for XOR cannot be split by one straight line. Labels: x1 | x2 | 0 | 1 | 1 | 0 | XOR labels occupy opposite corners | 0 | 1 | 0 | 1 | One straight decision boundary cannot separate the two classes. Opposite corners for XOR cannot be split by one straight line Tap the diagram to enlarge. For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1. For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR. A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-intro R23 = S15, 28 January 2023. Its AlexNet and convolution questions are retained by current Unit 4; sequence questions map to current Unit 5. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · R23 Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s0 ## Question **Question (4 marks).** Suggest and sketch an RNN architecture to synthesize a musical tune. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. Represent notes as discrete tokens or suitable pitch-duration vectors. Train a recurrent decoder to predict the next note from preceding notes. A vocabulary softmax handles discrete note tokens; repeatedly feed generated notes to continue the tune until a stop condition. This is autoregressive sequence generation, typically a many-to-many next-token training pattern. ![Music generation can predict a next note at every recurrent step](https://deep-learning-study-room.vercel.app/assets/figures/sequences.svg) Diagram: Music generation can predict a next note at every recurrent step. Labels: One to many | h | x | y | h | y | h | y | Many to one | h | x | h | x | h | x | y | Many to many aligned | h | x | y | h | x | y | h | x | y | Encoder then decoder | h | x | h | x | h | y | h | y | Schematics show mapping patterns, not fixed sequence lengths. Music generation can predict a next note at every recurrent step Tap the diagram to enlarge. --- ## TEE · R23 Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s1 ## Question **Question (4 marks).** Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below. Give the output and justify pooling. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_847554ea0e.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 2 | 3 | 2 | 5 | 5 | 3 | 4 | 6 | 3 | 7 | 4 | 8 | 5 | 9 | 8 | 5 | 10 | 4 | 2 | 9 | 10 | 3 | 4 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Apply 3 × 3 average pooling with stride 2 to Matrix 1 shown below. ### To find Find the requested pooled output(s). ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step **Shape:** $\left\lfloor\frac{5-3}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 2, 3, 5, 3, 4, 7, 4, 8] **Step 2: add all selected entries.** 1 + 2 + 3 + 5 + 3 + 4 + 7 + 4 + 8 = 37 **Step 3: divide by the 9 entries in this window.** 37/9 = **4.111111111** ### Pooling window (1,2) **Step 1: select the entries.** [3, 2, 5, 4, 6, 3, 8, 5, 9] **Step 2: add all selected entries.** 3 + 2 + 5 + 4 + 6 + 3 + 8 + 5 + 9 = 45 **Step 3: divide by the 9 entries in this window.** 45/9 = **5** ### Pooling window (2,1) **Step 1: select the entries.** [7, 4, 8, 8, 5, 10, 9, 10, 3] **Step 2: add all selected entries.** 7 + 4 + 8 + 8 + 5 + 10 + 9 + 10 + 3 = 64 **Step 3: divide by the 9 entries in this window.** 64/9 = **7.111111111** ### Pooling window (2,2) **Step 1: select the entries.** [8, 5, 9, 10, 4, 2, 3, 4, 3] **Step 2: add all selected entries.** 8 + 5 + 9 + 10 + 4 + 2 + 3 + 4 + 3 = 48 **Step 3: divide by the 9 entries in this window.** 48/9 = **5.333333333** **Average output:** $\begin{bmatrix}4.111111111 & 5\\7.111111111 & 5.333333333\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. --- ## TEE · R23 Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s2 ## Question **Question (4 marks).** State DL advantages and limitations. ## Answer — step by step Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. --- ## TEE · R23 Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s3 ## Question **Question (4 marks).** Evaluate how mini-batch size affects optimization and parallel processing. ## Answer — step by step A mini-batch forms one gradient from B examples. Smaller B usually gives noisier gradients, lower activation memory and more updates per epoch. Larger B averages more examples, often uses vectorized hardware more efficiently up to resource limits, but needs more memory and gives fewer updates per epoch. N examples produce ceil(N/B) updates if the last partial batch is kept. Too-large B does not guarantee faster wall-clock progress or better generalization. Tune batch size with learning rate and resource constraints. Computation within a batch can be parallelized; successive optimizer steps remain dependent on the updated parameters. --- ## TEE · R23 Q1e Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s4 ## Question **Question (4 marks).** Design a perceptron network for inputs 00 → 0,10 → 0,01 → 0,11 → 1; choose suitable weights. ## Answer — step by step ### Given Truth table: 00→0,10→0,01→0,11→1. Weights and bias are to be designed. ### To find Choose a perceptron that implements AND and show its four outputs. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step Choose weights w₁ = w₂ = 1 and bias −1.5. Use output 1 for z ≥ 0. ### Input (0, 0) **Step 1: multiply input 1 by its weight.** $$ 1(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 1(0)=0 $$ **Step 3: add the two products.** $$ s=0+0=0 $$ **Step 4: add the bias once.** $$ z=0+(-1.5)=-1.5 $$ **Step 5: apply the stated threshold.** z = -1.5; output = 0 because output is 1 exactly when z ≥ 0. ### Input (0, 1) **Step 1: multiply input 1 by its weight.** $$ 1(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 1(1)=1 $$ **Step 3: add the two products.** $$ s=0+1=1 $$ **Step 4: add the bias once.** $$ z=1+(-1.5)=-0.5 $$ **Step 5: apply the stated threshold.** z = -0.5; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 0) **Step 1: multiply input 1 by its weight.** $$ 1(1)=1 $$ **Step 2: multiply input 2 by its weight.** $$ 1(0)=0 $$ **Step 3: add the two products.** $$ s=1+0=1 $$ **Step 4: add the bias once.** $$ z=1+(-1.5)=-0.5 $$ **Step 5: apply the stated threshold.** z = -0.5; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 1) **Step 1: multiply input 1 by its weight.** $$ 1(1)=1 $$ **Step 2: multiply input 2 by its weight.** $$ 1(1)=1 $$ **Step 3: add the two products.** $$ s=1+1=2 $$ **Step 4: add the bias once.** $$ z=2+(-1.5)=0.5 $$ **Step 5: apply the stated threshold.** z = 0.5; output = 1 because output is 1 exactly when z ≥ 0. **Final answer:** outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND. A hidden layer is unnecessary for this linearly separable truth table. ![Two inputs with weights one and bias minus one point five implement AND](https://deep-learning-study-room.vercel.app/assets/figures/and_perceptron.svg) Diagram: Two inputs with weights one and bias minus one point five implement AND. Labels: INPUTS | WEIGHTED SUM | DECISION | x1 | x2 | w1 = 1 | w2 = 1 | z = x1 + x2 − 1.5 | step(z) | 1 when z ≥ 0 | 0 otherwise | Bias b = −1.5. Only input [1, 1] crosses the threshold. Two inputs with weights one and bias minus one point five implement AND Tap the diagram to enlarge. --- ## TEE · R23 Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s5 ## Question **Question (10 marks).** A CNN with convolution, pooling, dense and output layers receives 127 × 127 images for 5 classes and trains with Adam. Identify learned parameters and tunable hyperparameters. ## Answer — step by step Convolutional kernel entries and filter biases are learned parameters. Dense weight matrices and biases are learned parameters. Pooling, flattening and ordinary activations have no learned weights. The output has 5 class scores and usually softmax with categorical cross-entropy. Adam’s m, v are running optimizer states, not extra predictive parameters. Hyperparameters include kernel sizes, filter counts, strides, padding, pooling configuration, dense widths, activations, learning rate, Adam β₁/β₂/ε, batch size, epochs and regularization. Architecture choices control representation capacity and resource demand; optimizer settings control update behavior. No filter counts or dense widths are specified, so a unique numerical parameter total cannot be calculated from 127 × 127 and 5 classes alone. Adam combines an exponentially weighted first moment of gradients with a second moment of squared gradients. Both start at zero; early bias correction compensates for this zero initialization. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ Use update count t = 1 for the first update; t = 0 would make the correction denominator zero. m stores direction with momentum-like smoothing; v stores gradient magnitude information; division by its square root adapts coordinate step sizes. Do not multiply the final fraction by another copy of g. For g₁ = 2, β₁ = 0.9 and β₂ = 0.999, m₁ = 0.2 and v₁ = 0.004. Corrected moments are 2 and 4. With θ₀ = 1, η = 0.1 and negligible ε, θ₁ = 1−0.1 × 2/2 = 0.9. Compared with ordinary SGD, Adam often provides a useful adaptive starting point and can cope with uneven or sparse gradient scales. It needs two state arrays per parameter, several hyperparameters and validation of generalization; it is not universally more accurate or guaranteed to converge to the global optimum. --- ## TEE · R23 Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s6 ## Question **Question (10 marks).** Justify ReLU over sigmoid/tanh for deep hidden layers. For predicting brand ratings on a scale of 10, choose output activation and loss. Diagnose 2% training versus 18% validation error and suggest remedies. ## Answer — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. **Rating regression:** for a continuous score, one linear output with MSE is a standard baseline. If strict 0–10 bounds are required, use 10 × sigmoid(z) and the matching regression loss, noting saturation near the endpoints. If labels are categorical rating bins, a categorical formulation would be different; state the continuous-score assumption. **2% versus 18%:** if errors use the same comparable metric and distribution, this large generalization gap strongly suggests overfitting. Use appropriate regularization, early stopping, more representative data and possibly a smaller model; inspect leakage and preprocessing too. Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. --- ## TEE · R23 Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s7 ## Question **Question (10 marks).** Build and train an RNN language model for “Mumbai is the financial capital of Maharashtra.” Draw the architecture and write next-token conditional probabilities at each step. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. Tokenize, build a vocabulary, encode tokens, pair each prefix with its next token, unroll the recurrence, apply vocabulary softmax, sum/average cross-entropy, then use BPTT and an optimizer to update shared weights. Include start/end tokens consistently. **Explicit next-token conditionals ignoring punctuation:** 1. P(Mumbai | START). 2. P(is | Mumbai). 3. P(the | Mumbai is). 4. P(financial | Mumbai is the). 5. P(capital | Mumbai is the financial). 6. P(of | Mumbai is the financial capital). 7. P(Maharashtra | Mumbai is the financial capital of). 8. P(END | Mumbai is the financial capital of Maharashtra). The sequence likelihood is their product; the negative log-likelihood is the sum of negative log conditionals. No numeric probabilities are determined without weights. Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph. 1. Initialize the hidden state and run the forward recurrence from the first input to the last. 2. Compute the specified output loss, at the final step or across several steps. 3. Start from the last loss and propagate derivatives backwards through output and hidden operations. 4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states. 5. Sum all uses of each shared weight before applying the optimizer update. $$ \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T} $$ The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step. ![BPTT follows this unrolled network backwards and sums gradients of shared weights](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: BPTT follows this unrolled network backwards and sums gradients of shared weights. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge. --- ## TEE · R23 Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s8 ## Question **Question (10 marks).** For input [0.5,0.7,0.8], first-stage multipliers [0.3,0.4,0.6], next receiving rows [0.5,0.2,0.3], [0.3,0.3,0.3], output rows [0.3,0.2], [0.2,0.3], use ReLU in the first two stages and sigmoid at outputs. Targets [0,1], zero biases, η = 0.1. Find predictions, MSE and all four hidden-to-output weight updates. ## Answer — step by step ### Given For input [0.5,0.7,0.8], first-stage multipliers [0.3,0.4,0.6], next receiving rows [0.5,0.2,0.3], [0.3,0.3,0.3], output rows [0.3,0.2], [0.2,0.3], use ReLU in the first two stages and sigmoid at outputs. Targets [0,1], zero biases, η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate MSE and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Mean squared loss.** Average the squared scalar prediction errors. The derivative retains the factor 2/n; n must count the outputs included in this mean. n = number of scalar prediction errors in the stated mean; oⱼ = prediction; yⱼ = target. $$ L=\frac1n\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=\frac2n(o_j-y_j) $$ **Yaad rakhne ka cue:** MSE mein derivative 2/n times error; half SSE ka rule blindly mat lagao. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Step 1: evaluate the first ReLU stage The raw feature products are [0.5 × 0.3, 0.7 × 0.4, 0.8 × 0.6] = **[0.15, 0.28, 0.48]**. All are positive, so ReLU leaves them unchanged. Biases are zero. ### Step 2: evaluate the second ReLU stage h₁ = max(0, 0.5(0.15) + 0.2(0.28) + 0.3(0.48)) = max(0, 0.075 + 0.056 + 0.144) = **0.275**. h₂ = max(0, 0.3(0.15) + 0.3(0.28) + 0.3(0.48)) = max(0, 0.045 + 0.084 + 0.144) = **0.273**. ### Step 3: evaluate both sigmoid outputs z₁ = 0.3(0.275) + 0.2(0.273) = 0.1371; o₁ = σ(z₁) = **0.534221413**. z₂ = 0.2(0.275) + 0.3(0.273) = 0.1369; o₂ = σ(z₂) = **0.534171647**. ### Step 4: calculate MSE and the output deltas MSE = [(o₁ − 0)² + (o₂ − 1)²]/2 = **0.251194286**. There are exactly two output components, so the square derivative's factor 2 cancels the division by 2. δ₁ = (o₁ − 0)o₁(1 − o₁) = **0.132929724**. δ₂ = (o₂ − 1)o₂(1 − o₂) = **−0.115913140**. ### Step 5: calculate each output-weight gradient A weight gradient equals its receiving output delta multiplied by its sending hidden activation. Use the same old forward pass for all four connections. | Connection | Gradient calculation | Gradient | | --- | --- | --- | | h₁ → o₁ | 0.132929724 × 0.275 | 0.036555674 | | h₂ → o₁ | 0.132929724 × 0.273 | 0.036289815 | | h₁ → o₂ | −0.115913140 × 0.275 | −0.031876113 | | h₂ → o₂ | −0.115913140 × 0.273 | −0.031644287 | ### Step 6: subtract learning rate times each gradient | Connection | Update at η = 0.1 | New weight | | --- | --- | --- | | h₁ → o₁ | 0.3 − 0.1(0.036555674) | 0.296344433 | | h₂ → o₁ | 0.2 − 0.1(0.036289815) | 0.196371019 | | h₁ → o₂ | 0.2 − 0.1(−0.031876113) | 0.203187611 | | h₂ → o₂ | 0.3 − 0.1(−0.031644287) | 0.303164429 | **Final answer:** predictions [0.534221413, 0.534171647], MSE 0.251194286, and output-weight rows **[0.296344433, 0.196371019]**, **[0.203187611, 0.303164429]**. The positive gradients decrease weights feeding output 1, while the negative gradients increase weights feeding output 2. --- ## TEE · R23 Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s9 ## Question **Question (10 marks).** Design employee face-recognition attendance using a CNN with too few training images. Explain solutions. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. --- ## TEE · R23 Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s10 ## Question **Question (10 marks).** Estimate parameters of the AlexNet diagram. Use Input 227 × 227 × 3. → Conv K = 11, filters = 96, stride = 4, padding = 0. → Pool K = 3, stride = 2, padding = 0. → Conv K = 5, filters = 256, stride = 1, padding = 2. → Pool K = 3, stride = 2, padding = 0. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 384, stride = 1, padding = 1. → Conv K = 3, filters = 256, stride = 1, padding = 1. → Pool K = 3, stride = 2, padding = 0. → Flatten. → Dense 4096 units. → Dense 4096 units. → Dense 1000 units. ## Answer — step by step ### Given Use Input 227 × 227 × 3. Conv K = 11, filters = 96, stride = 4, padding = 0. Pool K = 3, stride = 2, padding = 0. Conv K = 5, filters = 256, stride = 1, padding = 2. Pool K = 3, stride = 2, padding = 0. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 384, stride = 1, padding = 1. Conv K = 3, filters = 256, stride = 1, padding = 1. Pool K = 3, stride = 2, padding = 0. Dense 4096 units. Dense 4096 units. Dense 1000 units. ### To find Find every layer shape, its parameter count and the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**227 × 227 × 3**height × width × channels → Convolution**96 filters · 11 × 11 · S = 4 · P = 0** → Output shape**55 × 55 × 96**height × width × channels Weights**34,848** + Biases**96** = Trainable parameters**34,944** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **227 × 227 × 3**. Use 96 filters of spatial size 11 × 11, stride 4, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{227+2(0)-11}{4}\right\rfloor+1=55 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **55 × 55 × 96**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=11\times11\times3\times96=34{,}848 $$ $$ N_{\mathrm{biases}}=96 $$ $$ P_1=34{,}848+96=34{,}944 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**55 × 55 × 96**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**27 × 27 × 96**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **55 × 55 × 96**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{55-3}{2}\right\rfloor+1=27 $$ Pooling acts independently on each channel. The output is **27 × 27 × 96**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**27 × 27 × 96**height × width × channels → Convolution**256 filters · 5 × 5 · S = 1 · P = 2** → Output shape**27 × 27 × 256**height × width × channels Weights**614,400** + Biases**256** = Trainable parameters**614,656** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **27 × 27 × 96**. Use 256 filters of spatial size 5 × 5, stride 1, and padding 2 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{27+2(2)-5}{1}\right\rfloor+1=27 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **27 × 27 × 256**. **3. Count trainable values.** Each filter receives all 96 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=5\times5\times96\times256=614{,}400 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_3=614{,}400+256=614{,}656 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**27 × 27 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**13 × 13 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **27 × 27 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{27-3}{2}\right\rfloor+1=13 $$ Pooling acts independently on each channel. The output is **13 × 13 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 5 — convolution Layer 5: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**884,736** + Biases**384** = Trainable parameters**885,120** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 256**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 256 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times256\times384=884{,}736 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_5=884{,}736+384=885{,}120 $$ ### Layer 6 — convolution Layer 6: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**384 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 384**height × width × channels Weights**1,327,104** + Biases**384** = Trainable parameters**1,327,488** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 384 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 384**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times384=1{,}327{,}104 $$ $$ N_{\mathrm{biases}}=384 $$ $$ P_6=1{,}327{,}104+384=1{,}327{,}488 $$ ### Layer 7 — convolution Layer 7: what changes and what is learned Input shape**13 × 13 × 384**height × width × channels → Convolution**256 filters · 3 × 3 · S = 1 · P = 1** → Output shape**13 × 13 × 256**height × width × channels Weights**884,736** + Biases**256** = Trainable parameters**884,992** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **13 × 13 × 384**. Use 256 filters of spatial size 3 × 3, stride 1, and padding 1 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{13+2(1)-3}{1}\right\rfloor+1=13 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **13 × 13 × 256**. **3. Count trainable values.** Each filter receives all 384 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times384\times256=884{,}736 $$ $$ N_{\mathrm{biases}}=256 $$ $$ P_7=884{,}736+256=884{,}992 $$ ### Layer 8 — pooling Layer 8: what changes and what is learned Input shape**13 × 13 × 256**height × width × channels → Pooling**3 × 3 window · S = 2** → Output shape**6 × 6 × 256**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **13 × 13 × 256**. Use a 3 × 3 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{13-3}{2}\right\rfloor+1=6 $$ Pooling acts independently on each channel. The output is **6 × 6 × 256**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 9 — flatten Layer 9: what changes and what is learned Input shape**6 × 6 × 256**height × width × channels → Flatten**Rearrange activations** → Output shape**9,216**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **6 × 6 × 256**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=6\times6\times256=9{,}216 $$ Output: a vector with **9,216 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 10 — fully connected layer Layer 10: what changes and what is learned Input shape**9,216**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**37,748,736** + Biases**4,096** = Trainable parameters**37,752,832** Every output neuron receives every input feature, with one bias per output neuron. Input: **9,216 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=9{,}216\times4{,}096=37{,}748{,}736 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_10=37{,}748{,}736+4{,}096=37{,}752{,}832 $$ ### Layer 11 — fully connected layer Layer 11: what changes and what is learned Input shape**4,096**input features → Dense layer**4,096 neurons** → Output shape**4,096**output features Weights**16,777,216** + Biases**4,096** = Trainable parameters**16,781,312** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **4,096 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times4{,}096=16{,}777{,}216 $$ $$ N_{\mathrm{biases}}=4{,}096 $$ $$ P_11=16{,}777{,}216+4{,}096=16{,}781{,}312 $$ ### Layer 12 — fully connected layer Layer 12: what changes and what is learned Input shape**4,096**input features → Dense layer**1,000 neurons** → Output shape**1,000**output features Weights**4,096,000** + Biases**1,000** = Trainable parameters**4,097,000** Every output neuron receives every input feature, with one bias per output neuron. Input: **4,096 features**. Output: **1,000 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=4{,}096\times1{,}000=4{,}096{,}000 $$ $$ N_{\mathrm{biases}}=1{,}000 $$ $$ P_12=4{,}096{,}000+1{,}000=4{,}097{,}000 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{227+2 \times 0-11}{4}\right\rfloor+1$ | 55 x 55 x 96 | 34,848 | 96 | 34,944 | | 2 pool | $\left\lfloor\frac{55-3}{2}\right\rfloor+1$ | 27 x 27 x 96 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{27+2 \times 2-5}{1}\right\rfloor+1$ | 27 x 27 x 256 | 614,400 | 256 | 614,656 | | 4 pool | $\left\lfloor\frac{27-3}{2}\right\rfloor+1$ | 13 x 13 x 256 | 0 | 0 | 0 | | 5 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 884,736 | 384 | 885,120 | | 6 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 384 | 1,327,104 | 384 | 1,327,488 | | 7 conv | $\left\lfloor\frac{13+2 \times 1-3}{1}\right\rfloor+1$ | 13 x 13 x 256 | 884,736 | 256 | 884,992 | | 8 pool | $\left\lfloor\frac{13-3}{2}\right\rfloor+1$ | 6 x 6 x 256 | 0 | 0 | 0 | | 9 flat | 6 × 6 × 256 | 9216 | 0 | 0 | 0 | | 10 dense | 9216 → 4096 | 4096 | 37,748,736 | 4,096 | 37,752,832 | | 11 dense | 4096 → 4096 | 4096 | 16,777,216 | 4,096 | 16,781,312 | | 12 dense | 4096 → 1000 | 1000 | 4,096,000 | 1,000 | 4,097,000 | **Parameter substitutions, in layer order:** Layer 1: (11 x 11 x 3 + 1) x 96 = 34,944. Layer 3: (5 x 5 x 96 + 1) x 256 = 614,656. Layer 5: (3 x 3 x 256 + 1) x 384 = 885,120. Layer 6: (3 x 3 x 384 + 1) x 384 = 1,327,488. Layer 7: (3 x 3 x 384 + 1) x 256 = 884,992. Layer 10: (9216 + 1) x 4096 = 37,752,832. Layer 11: (4096 + 1) x 4096 = 16,781,312. Layer 12: (4096 + 1) x 1000 = 4,097,000. **Total:** 34,944 + 614,656 + 885,120 + 1,327,488 + 884,992 + 37,752,832 + 16,781,312 + 4,097,000 = **62,378,344 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. The diagram labels its final pooled map 6 × 6 × 256; a 3 × 3, stride 2 pool from 13 × 13 produces 6. If a small printed stride label is read as 3, it conflicts with the shown 6 × 6 map. The answer uses the consistent classic stride 2 interpretation. Original grouped AlexNet counts are a different architecture assumption. --- ## TEE · R23 Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s11 ## Question **Question (10 marks).** An RNN does not improve because vanishing gradients prevent useful parameter updates. Suggest a solution and explain required architectural changes. ## Answer — step by step Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph. 1. Initialize the hidden state and run the forward recurrence from the first input to the last. 2. Compute the specified output loss, at the final step or across several steps. 3. Start from the last loss and propagate derivatives backwards through output and hidden operations. 4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states. 5. Sum all uses of each shared weight before applying the optimizer update. $$ \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T} $$ The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step. ![BPTT follows this unrolled network backwards and sums gradients of shared weights](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: BPTT follows this unrolled network backwards and sums gradients of shared weights. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge. Replace the vanilla cell with an LSTM or GRU while preserving a suitable sequence output. An LSTM introduces cell state and learned forget, input, output gates; a GRU introduces reset/update gates and a gated hidden-state mixture. These paths make retaining long context easier. Tune initialization and learning rate and inspect gradients. An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. --- ## TEE · R23 Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s12 ## Question **Question (10 marks).** Justify multiple ANN layers. Design a fraudulent/non-fraudulent transaction classifier for 10,000 samples with 4 features; choose layers/units and count parameters. ## Answer — step by step ### Given 10,000 samples; 4 features per sample; two target classes: fraudulent/non-fraudulent. Hidden layer sizes are design choices, not fixed in the question. ### To find Explain multiple layers, propose an appropriate classifier and count parameters for that explicit design. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step A single perceptron forms a weighted sum and applies a threshold. Its decision boundary is linear: w·x+b = 0. It can learn linearly separable patterns such as AND and OR under suitable learning conditions, but cannot represent XOR with one linear separator. ![Opposite corners for XOR cannot be split by one straight line](https://deep-learning-study-room.vercel.app/assets/figures/xor.svg) Diagram: Opposite corners for XOR cannot be split by one straight line. Labels: x1 | x2 | 0 | 1 | 1 | 0 | XOR labels occupy opposite corners | 0 | 1 | 0 | 1 | One straight decision boundary cannot separate the two classes. Opposite corners for XOR cannot be split by one straight line Tap the diagram to enlarge. For binary AND, w₁ = w₂ = 1 and b = −1.5 give preactivations −1.5, −0.5, −0.5 and 0.5 for 00, 10, 01 and 11. A step that returns 1 at nonnegative inputs produces 0,0,0,1. For XOR, use h₁ = step(x₁+x₂−0.5) for OR and h₂ = step(x₁+x₂−1.5) for AND. Then XOR = step(h₁−2h₂−0.5). For inputs 00,01,10,11, the hidden pairs are (0,0), (1,0), (1,0), (1,1), producing 0,1,1,0. XNOR = step(−h₁+2h₂+0.5), producing 1,0,0,1. This is a multilayer construction, not a claim that a single perceptron solves XOR. A multilayer feedforward network adds hidden nonlinear features before the final boundary. Without hidden nonlinearities, adding linear layers still produces one affine map. **Chosen design:** 4 → 8 → 4 → 1 with ReLU hidden layers, one sigmoid output and binary cross-entropy. Hidden widths are explicit design assumptions. Architecture: 4 → 8 → 4 → 1. Each non-input neuron has one bias. ### Layer 4 → 8 **Step 1: count connections.** $$ 4(8)=32 $$ **Step 2: count receiving-neuron biases.** There are 8 receiving neurons, so 8 biases. **Step 3: add weights and biases.** $$ 32+8=40 $$ ### Layer 8 → 4 **Step 1: count connections.** $$ 8(4)=32 $$ **Step 2: count receiving-neuron biases.** There are 4 receiving neurons, so 4 biases. **Step 3: add weights and biases.** $$ 32+4=36 $$ ### Layer 4 → 1 **Step 1: count connections.** $$ 4(1)=4 $$ **Step 2: count receiving-neuron biases.** There are 1 receiving neurons, so 1 biases. **Step 3: add weights and biases.** $$ 4+1=5 $$ **Total = 81**. Input values and activations are not trainable parameters. Use stratified train/validation/test partitions; handle class imbalance using appropriate training weights and evaluate precision/recall or a suitable decision threshold alongside accuracy. More layers are useful only if they learn needed nonlinear structure; the problem statement does not prescribe one uniquely correct architecture. ### Verify input (0,0) **Step 1: calculate hidden OR preactivation.** $$ z_1=0+0-0.5=-0.5 $$ **Step 2: apply its threshold.** $$ h_1=0 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=0+0-1.5=-1.5 $$ **Step 4: apply its threshold.** $$ h_2=0 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=0-2(0)-0.5=-0.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=0 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-0+2(0)+0.5=0.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=1 $$ ### Verify input (0,1) **Step 1: calculate hidden OR preactivation.** $$ z_1=0+1-0.5=0.5 $$ **Step 2: apply its threshold.** $$ h_1=1 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=0+1-1.5=-0.5 $$ **Step 4: apply its threshold.** $$ h_2=0 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=1-2(0)-0.5=0.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=1 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-1+2(0)+0.5=-0.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=0 $$ ### Verify input (1,0) **Step 1: calculate hidden OR preactivation.** $$ z_1=1+0-0.5=0.5 $$ **Step 2: apply its threshold.** $$ h_1=1 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=1+0-1.5=-0.5 $$ **Step 4: apply its threshold.** $$ h_2=0 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=1-2(0)-0.5=0.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=1 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-1+2(0)+0.5=-0.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=0 $$ ### Verify input (1,1) **Step 1: calculate hidden OR preactivation.** $$ z_1=1+1-0.5=1.5 $$ **Step 2: apply its threshold.** $$ h_1=1 $$ **Step 3: calculate hidden AND preactivation.** $$ z_2=1+1-1.5=0.5 $$ **Step 4: apply its threshold.** $$ h_2=1 $$ **Step 5: calculate XOR output preactivation.** $$ z_{XOR}=1-2(1)-0.5=-1.5 $$ **Step 6: apply the XOR threshold.** $$ y_{XOR}=0 $$ **Step 7: calculate XNOR output preactivation.** $$ z_{XNOR}=-1+2(1)+0.5=1.5 $$ **Step 8: apply the XNOR threshold.** $$ y_{XNOR}=1 $$ **Final answer:** XOR = [0,1,1,0]; XNOR = [1,0,0,1] for 00,01,10,11. The hidden nonlinear units make this possible. --- ## TEE · R23 Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s13 ## Question **Question (10 marks).** Suggest and justify remedies when (1) loss does not minimize after many epochs, (2) gradients are noisy and loss oscillates, (3) loss falls then rises because updates miss a minimum. ## Answer — step by step The learning rate η determines update size: θ new = θ old − ηg. A rate that is too small can make progress extremely slow; one that is too large can overshoot, oscillate or diverge. Nonconvex objectives can contain flat regions, saddle points and local minima. Poor scaling can cause long zig-zag paths; vanishing or exploding gradients can make training ineffective or unstable. ![Illustrative full batch and stochastic traces distinguish smoothness from progress](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative full batch and stochastic traces distinguish smoothness from progress. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative full batch and stochastic traces distinguish smoothness from progress Tap the diagram to enlarge. Batch gradient descent uses all N samples per update, SGD uses one sample, and mini-batch uses B samples. Batch updates are less noisy for a fixed dataset but costly; SGD makes frequent noisy updates; mini-batches balance vectorized computation and stochasticity. Data need not all reside in memory even for a full gradient because gradients can be accumulated, but larger simultaneous batches require more activation memory. Momentum accumulates a running direction, helping movement persist across consistent gradients and damping some oscillation. Mini-batching describes which data forms a gradient; momentum describes how that gradient is used. They can be combined, so they are not competing categories. Diagnose stalled training by checking data, preprocessing, losses and gradients before blindly adding layers. Compare training and validation curves, tune learning rate, inspect saturation and use appropriate initialization or adaptive optimization. A noisy curve with a decreasing trend can be normal for SGD. **Scenario 1:** check correctness and gradient magnitudes, then tune learning rate, activation, initialization or optimizer. **Scenario 2:** consider an appropriate larger mini-batch or momentum; distinguish ordinary stochastic noise from divergence. **Scenario 3:** reduce learning rate or use a schedule; momentum/Nesterov can help but excessive momentum can worsen overshoot. Judge training and validation curves separately. --- ## TEE · R23 Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s14 ## Question **Question (10 marks).** A 28 × 28 × 192 input is convolved with 32 same-padded 5 × 5 filters to produce 28 × 28 × 32. Count computations. What happens if a 1 × 1 convolution precedes it? The reduction-channel count is not supplied. ## Answer — step by step ### Given Input = 28 × 28 × 192; 32 same-padded 5 × 5 filters. A preceding 1 × 1 convolution would reduce to C channels. C is not supplied, so first give the symbolic bottleneck count; 16 channels is only an illustrative assumption. ### To find Calculate direct cost and express bottleneck cost in C; explain when the bottleneck saves computation. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Direct path **Step 2: direct path.** output stays 28 × 28 with 32 channels. **Step 3: direct path.** MACs = 28 × 28 × 192 × 32 × 5 × 5 = **120,422,400**. ### Bottleneck path with 16 reduction channels **Step 4: bottleneck path with 16 reduction channels.** 1 × 1 stage costs 28 × 28 × 192 × 16 = 2,408,448 MACs. **Step 5: bottleneck path with 16 reduction channels.** The 5 × 5 stage costs 28 × 28 × 16 × 32 × 25 = 10,035,200 MACs. **Step 6: bottleneck path with 16 reduction channels.** Total = **12,443,648**. ### Saving **Step 7: saving.** 120,422,400−12,443,648 = 107,978,752 MACs, or 107,978,752/120,422,400 × 100 = **89.666667%**. This counts multiplications with accumulation as MACs and omits biases/nonlinearities. Padding preserves spatial size. If the reduction width is not specified, call it r: total = 784 × 192r + 784 × 25 × 32r = **777,728r** **Step 8: saving.** there is no unique numeric bottleneck cost until r is chosen. The 16-channel bottleneck is an explicitly chosen illustrative value, consistent with a related paper; the symbolic 777,728 r answer is required when r has not been specified. **Step 9: calculate this quantity.** An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. **Step 10: calculate this quantity.** A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. --- ## TEE · R23 Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s15 ## Question **Question (10 marks).** Process with LSTM: “XYZ was very brave soldier. He was a part of the Kargil war and laid down his life fighting against the enemy. The president of India honored patriotic \_\_\_.” Explain the need for its gates. ## Answer — step by step An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. The model can retain the earlier entity XYZ and the soldier context while updating intervening events. At the final prefix a vocabulary softmax predicts a suitable next token; “soldier” is a plausible completion. The exact output is not uniquely determined by the text or architecture alone. The forget gate retains useful entity content, the input gate adds the war/honor context, and the output gate exposes relevant state for prediction. Do not invent numerical gate values when no weights are provided. --- ## TEE · R23 Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c16-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/16/tee-c16-s16 ## Question **Question (10 marks).** Training performance is good but unseen-data performance degrades. Identify the issue and explain ways to improve it. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. A comparable-distribution train/unseen gap suggests overfitting. Also check whether the new data have shifted, preprocessing differs or labels are inconsistent. Select regularization using validation performance, not final-test tuning. Improvement must be demonstrated on representative held-out examples. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-intro These are **authored learning exercises**, not additional PYQs. Their purpose is to practise the methods before attempting the full examination questions in the past exam questions under Solved Questions. Any practice marks describe an exercise's suggested size, not marks or frequency observed in a past paper. For TEE priorities, start with the December 2025 and February 2026 paper question forms; use extra variants to repair a method that is still unclear. These 52 original drills supplement every solved PYQ. They are not predicted exam questions. Use a blank page: copy the question, cover the answer, choose the formula, solve, then compare. After an error, redo the same type with the answer covered later that day and once the next morning. Reading or copying a solution once is not the same as being able to retrieve its method. For TEE, complete every family, including drills 42–45 and 50 on gates and BPTT. A ten-minute block can contain one long numerical or three short ones. Repeat a solved family later with its answer covered before counting it as reliable. --- ## TEE · How to solve a mixed numerical without guessing Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s0 A mixed question is a chain of familiar operations. First circle the requested final quantity. Underline the given activation, loss, padding, stride, threshold, learning rate and any fixed parameters. Draw a small box for each stage. Write the output of one box as the input to the next. Never skip a stage because its title looks unfamiliar. **If the question says output or prediction:** do only the forward chain, unless training is separately requested. For a neuron, multiply and add before activating. For a CNN, determine spatial shapes before flattening. For an RNN, carry the previous state into the next time step. **If it says train or update:** finish the entire old forward pass, calculate the stated loss derivative, move backward, store every gradient, then update together. If there are several output branches, add their effects at the shared hidden unit. For a batch, follow the stated sum or mean loss before updating. **If it mixes dropout and a loss:** apply the given training mask and scaling first; then predict from the resulting activations. The gradient to a dropped path is zero for that mask. If it also specifies L2, add its gradient to the data gradient before the optimizer step. If it also specifies momentum or Adam, carry their stored states as well as the parameter. **If it mixes convolution, pooling and a dense classifier:** output shape → next layer input shape → flatten length → dense scores → activation → loss. Count learned parameters separately from output values. Pooling and flattening have zero learned parameters in these questions. **If it changes a familiar problem:** an extra input adds another product; an extra output adds another backward path; a new channel adds another kernel sum; a new time step adds another shared-state calculation. The method stays the same. The drills below deliberately change signs, targets, loss scales and layer combinations. **Your completion rule:** solve a representative problem today without the answer, repeat it after several hours, then solve a changed-number or mixed version tomorrow. Mark the exact line where your reasoning failed. Correcting that line is what builds reliable recall; reading an open solution does not test it. --- ## TEE · Drill 1 Signs and brackets Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s1 ## Question **Question.** Evaluate 2−0.1 × (−3), then (2−0.1) × (−3). Explain why they differ. ## Answer — step by step ### Given Expressions: 2−0.1×(−3) and (2−0.1)×(−3). ### To find Evaluate both and explain the effect of grouping. ### Formula — which one and why **Order of operations.** Grouping decides which operation is performed first. Evaluate brackets, then powers, then multiplication/division, then addition/subtraction. A minus sign inside brackets belongs to that number. Subtracting a negative changes the operation to addition. $$ a-(-b)=a+b $$ **Yaad rakhne ka cue:** Bracket pehle; phir power; phir multiply/divide; aakhir mein plus-minus. ### Solution — step by step ### Meaning of the calculation A minus sign can mean subtraction or a negative number. Brackets tell us which part belongs together. These two expressions have different brackets, so they describe different calculations. ### Rule used Do brackets first, then multiplication, then addition or subtraction. Subtracting a negative number means adding its positive size. **Step 1: First expression: multiply before subtracting.** $$ 2-0.1\times(-3) $$ First multiply: a positive number times a negative number is negative. $$ 0.1\times(-3)=-0.3 $$ Put this complete result back into the expression. Keep its negative sign inside brackets. $$ 2-(-0.3)=2+0.3=2.3 $$ **Step 2: Second expression: finish the bracket first.** $$ 2-0.1=1.9 $$ Now multiply the whole bracket result by negative 3. $$ 1.9\times(-3)=-5.7 $$ **Final answer:** first expression = 2.3; second expression = −5.7. In a weight update, multiply learning rate by gradient first, then subtract that product from the old weight. --- ## TEE · Drill 2 Half sum versus mean Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s2 ## Question **Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE. ## Answer — step by step ### Given Predictions = [0.2, 0.7, 0.4]. Targets = [0, 1, 1]; number of outputs n = 3. ### To find Find SSE, half SSE, MSE and the prediction gradients requested in the question. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step ### Given values and loss definitions There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i. $$ \mathrm{SSE}=\sum_{i=1}^{n}e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}2 $$ $$ \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}n $$ SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively. ### Step 1: calculate each residual and its square | Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual | | --- | --- | --- | --- | --- | | 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 | | 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 | | 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 | ### Step 2: sum the squared residuals $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel. ### Step 3: apply each requested normalization For half SSE, the denominator is 2: $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245 $$ For MSE, the denominator is the number of pairs, n = 3: $$ \mathrm{MSE}=\frac{0.49}{3}=0.163333333\ldots $$ **Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question. --- ## TEE · Drill 3 Natural log and probability Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s3 ## Question **Question.** Compute −ln(0.9),−ln(0.1), and $e^{-\ln 2}$ . ## Answer — step by step ### Given Probabilities: 0.9 and 0.1. Exponential expression: exp(−ln 2). ### To find Find −ln(0.9), −ln(0.1) and exp(−ln 2). ### Formula — which one and why **Natural logarithms.** The natural logarithm reverses the exponential. A negative log turns a small correct-class probability into a larger positive loss. ln uses base e ≈ 2.71828. exp(a) means e raised to the power a. Keep the minus outside ln when the question says −ln(p). $$ -\ln p=\ln\frac1p $$ $$ e^{\ln a}=a $$ $$ e^{-\ln a}=\frac1a $$ **Yaad rakhne ka cue:** exp banata hai power; ln us power ko wapas nikaalta hai. ### Solution — step by step ### Meaning and formulas The natural logarithm ln tells us the power of e needed to obtain a number. It reverses the exponential operation. A calculator evaluates ln; it is not ordinary division. $$ \ln(e^a)=a,\qquad e^{\ln a}=a\quad(a>0) $$ $$ e^{-a}=\frac1{e^a} $$ ### Step 1: calculate the first logarithm, then its minus sign $$ \ln(0.9)\approx-0.105360516 $$ $$ -\ln(0.9)=-(-0.105360516)=0.105360516 $$ ### Step 2: calculate the second logarithm $$ \ln(0.1)\approx-2.302585093 $$ $$ -\ln(0.1)=-(-2.302585093)=2.302585093 $$ When these are correct-class probabilities, 0.1 represents a worse prediction than 0.9, so its negative-log loss is larger. ### Step 3: simplify the exponential using the inverse rule $$ e^{-\ln2}=\frac1{e^{\ln2}}=\frac12=0.5 $$ **Final answers:** 0.105360516, 2.302585093 and 0.5. Use the ln key; base-10 log gives different first and second answers. --- ## TEE · Drill 4 Dot product and bias Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s4 ## Question **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU. ## Answer — step by step ### Given Inputs x = [2, −1, 0.5]. Weights w = [0.3, 0.4, −0.2]; bias b = 0.1. Activation: ReLU. ### To find Find the weighted sum z and ReLU output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero. $$ z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z) $$ Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output. ### Step 1: multiply the matching pairs $$ w_1x_1=0.3\times2=0.6 $$ $$ w_2x_2=0.4\times(-1)=-0.4 $$ $$ w_3x_3=(-0.2)\times0.5=-0.1 $$ ### Step 2: add all products and add bias once $$ z=0.6+(-0.4)+(-0.1)+0.1 $$ $$ z=0.2-0.1+0.1=0.2 $$ ### Step 3: apply the activation $$ a=\max(0,0.2)=0.2 $$ **Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive. --- ## TEE · Drill 5 Matrix rows receive inputs Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s5 ## Question **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ed1fcbfdf5.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ. Activation: ReLU on each result. ### To find Find Wx+b and the activated output. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formula Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row. $$ z=Wx+b,\qquad a_j=\max(0,z_j) $$ $$ W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix} $$ ### Step 1: calculate receiving neuron 1 from row 1 $$ z_1=(1\times2)+(2\times1)+0.5 $$ $$ z_1=2+2+0.5=4.5 $$ ### Step 2: calculate receiving neuron 2 from row 2 $$ z_2=((-1)\times2)+(3\times1)+(-0.5) $$ $$ z_2=-2+3-0.5=0.5 $$ ### Step 3: apply ReLU to each entry $$ a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5 $$ $$ z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix} $$ The shape is 2 rows by 1 column: one output for each receiving row. --- ## TEE · Drill 6 Chain rule on a tiny loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s6 ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. --- ## TEE · Drill 7 Sigmoid with binary cross entropy Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s7 ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. --- ## TEE · Drill 8 Activation contrast Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s8 ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. --- ## TEE · Drill 9 Softmax and true class loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s9 ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. --- ## TEE · Drill 10 Sigmoid neuron gradient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s10 ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. --- ## TEE · Drill 11 One perceptron learning update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s11 ## Question **Question.** Binary perceptron, w = [0,0], b = 0, x = [1,1], target 1, η = 0.5. Use output 1 if z>0 else 0. Perform one update. ## Answer — step by step ### Given w = [0,0]; b = 0; x = [1,1]; target t = 1. Learning rate η = 0.5. Output 1 only when z > 0; otherwise output 0. ### To find Perform one update and report new weights and bias. ### Formula — which one and why **Perceptron learning rule.** Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters. t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question. $$ z=\sum_iw_ix_i+b $$ $$ w_i^{\mathrm{new}}=w_i+\eta(t-o)x_i $$ $$ b^{\mathrm{new}}=b+\eta(t-o) $$ **Yaad rakhne ka cue:** Prediction nikalo; target−prediction se correction nikalo. ### Solution — step by step ### Meaning and formula A perceptron first computes a weighted sum z and then applies the exact threshold rule in the question. This question uses **strictly greater than zero**, so z = 0 produces output 0. $$ z=w_1x_1+w_2x_2+b,\qquad o=\begin{cases}1&z>0\\0&z\le0\end{cases} $$ The perceptron learning rule uses target minus output, written e below. This is a discrete perceptron rule, not the sigmoid backpropagation derivative. $$ e=t-o,\qquad w_{j,\mathrm{new}}=w_j+\eta e x_j,\qquad b_{\mathrm{new}}=b+\eta e $$ ### Step 1: obtain the old output $$ z=0\times1+0\times1+0=0,\qquad o=0 $$ ### Step 2: compare it with the target $$ e=1-0=1 $$ ### Step 3: update each weight separately $$ w_{1,\mathrm{new}}=0+0.5\times1\times1=0.5 $$ $$ w_{2,\mathrm{new}}=0+0.5\times1\times1=0.5 $$ $$ b_{\mathrm{new}}=0+0.5\times1=0.5 $$ ### Step 4: interpret the updated prediction $$ z_{\mathrm{new}}=0.5\times1+0.5\times1+0.5=1.5 $$ Since 1.5 > 0, the new output is 1. **Final parameters: w = [0.5, 0.5], b = 0.5.** --- ## TEE · Drill 12 XOR using two hidden units Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s12 ## Question **Question.** Use h1 = step(x1+x2−0.5), h2 = step(x1+x2−1.5), output = step(h1−2h2−0.5), step(z) = 1 if z≥0. Find all four outputs. ## Answer — step by step ### Given h₁ = step(x₁+x₂−0.5); h₂ = step(x₁+x₂−1.5). Output = step(h₁−2h₂−0.5). step(z) = 1 when z ≥ 0, otherwise 0. ### To find Calculate hidden values and output for 00, 01, 10 and 11. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step ### Meaning of the given network XOR should output 1 when exactly one input is 1, and 0 when the inputs match. The first hidden unit detects “at least one input is 1”; the second detects “both inputs are 1”. The output combines these detections. $$ h_1=\operatorname{step}(x_1+x_2-0.5) $$ $$ h_2=\operatorname{step}(x_1+x_2-1.5) $$ $$ o=\operatorname{step}(h_1-2h_2-0.5) $$ For this question, step returns 1 at nonnegative input and 0 at negative input. **Step 1: Input pair 0, 0.** $$ z_{h_1}=0+0-0.5=-0.5\Rightarrow h_1=0 $$ $$ z_{h_2}=0+0-1.5=-1.5\Rightarrow h_2=0 $$ $$ z_o=0-2(0)-0.5=-0.5\Rightarrow o=0 $$ **Step 2: Input pair 0, 1.** $$ z_{h_1}=0+1-0.5=0.5\Rightarrow h_1=1 $$ $$ z_{h_2}=0+1-1.5=-0.5\Rightarrow h_2=0 $$ $$ z_o=1-2(0)-0.5=0.5\Rightarrow o=1 $$ **Step 3: Input pair 1, 0.** $$ z_{h_1}=1+0-0.5=0.5\Rightarrow h_1=1 $$ $$ z_{h_2}=1+0-1.5=-0.5\Rightarrow h_2=0 $$ $$ z_o=1-2(0)-0.5=0.5\Rightarrow o=1 $$ **Step 4: Input pair 1, 1.** $$ z_{h_1}=1+1-0.5=1.5\Rightarrow h_1=1 $$ $$ z_{h_2}=1+1-1.5=0.5\Rightarrow h_2=1 $$ $$ z_o=1-2(1)-0.5=-1.5\Rightarrow o=0 $$ **Final truth table:** 00 → 0; 01 → 1; 10 → 1; 11 → 0. --- ## TEE · Drill 13 Full ReLU backpropagation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s13 ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## TEE · Drill 14 Two output sigmoid backpropagation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s14 ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. --- ## TEE · Drill 15 A negative ReLU blocks this path Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s15 ## Question **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2. ## Answer — step by step ### Given x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2. Hidden activation: ReLU; output activation: linear. Target y = 1; loss: half squared error. ### To find Find the gradients of w₁ and w₂ and explain any zero gradients. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output. $$ z_h=w_1x+b_1,\quad h=\max(0,z_h),\quad o=w_2h+b_2 $$ $$ L=\frac12(o-y)^2,\quad \delta_o=o-y $$ $$ \frac{\partial L}{\partial w_2}=\delta_o h,\quad \delta_h=\delta_o w_2\operatorname{ReLU}'(z_h),\quad \frac{\partial L}{\partial w_1}=\delta_h x $$ ### Step 1: hidden weighted sum and activation $$ z_h=(-0.4)\times2+0.1=-0.8+0.1=-0.7 $$ $$ h=\max(0,-0.7)=0 $$ Because the raw hidden score is negative, ReLU's derivative at this point is 0. ### Step 2: output prediction and output delta $$ o=0.5\times0+0.2=0.2 $$ $$ \delta_o=0.2-1=-0.8 $$ ### Step 3: output weight gradient $$ \frac{\partial L}{\partial w_2}=(-0.8)\times0=0 $$ This is zero because the input to that connection, h, is zero. ### Step 4: hidden delta and input weight gradient $$ \delta_h=(-0.8)\times0.5\times0=0 $$ $$ \frac{\partial L}{\partial w_1}=0\times2=0 $$ **Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero. --- ## TEE · Drill 16 Batch and sequential SGD Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s16 ## Question **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch. ## Answer — step by step ### Given Model ŷ = wx; initial w = 0; η = 0.1. Samples in order: (x,y) = (1,1), then (2,2). Batch objective: mean half squared loss. ### To find Compare the weight after one batch update and one sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### Meaning and formulas For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample. $$ \hat y=wx,\quad L=\frac12(\hat y-y)^2,\quad g=(\hat y-y)x $$ $$ g_{\mathrm{batch}}=\frac{g_1+g_2}{2},\qquad w_{\mathrm{new}}=w-\eta g $$ **Step 1: Part A: one batch update from w = 0.** For sample (1,1), prediction = 0 × 1 = 0, so: $$ g_1=(0-1)\times1=-1 $$ For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so: $$ g_2=(0-2)\times2=-4 $$ Average these two gradients, then update once. $$ g_{\mathrm{batch}}=\frac{-1+(-4)}2=-2.5 $$ $$ w_{\mathrm{batch}}=0-0.1(-2.5)=0.25 $$ **Step 2: Part B: one SGD epoch, restarting at w = 0.** The first sample has gradient −1, just as above. $$ w_1=0-0.1(-1)=0.1 $$ For the second sample use this new weight, so prediction is: $$ \hat y_2=0.1\times2=0.2 $$ $$ g_2=(0.2-2)\times2=(-1.8)\times2=-3.6 $$ $$ w_2=0.1-0.1(-3.6)=0.1+0.36=0.46 $$ **Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates. --- ## TEE · Drill 17 Incomplete mini batch Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s17/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s17 ## Question **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it. ## Answer — step by step ### Given N = 103 samples; B = 20 samples per mini-batch. E = 3 epochs. Compare both remainder policies: keep or drop the final 3 examples. ### To find Find total updates for full-batch GD, SGD, and both mini-batch policies. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step ### Meaning and formulas An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs. $$ U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NE $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained. ### Step 1: batch GD All 103 samples form one batch per epoch. $$ U_{\mathrm{batch}}=3\times1=3 $$ ### Step 2: SGD Each sample gets its own update. $$ U_{\mathrm{SGD}}=103\times3=309 $$ ### Step 3: mini-batches with the remainder $$ 103=5\times20+3 $$ The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give: $$ U_{\mathrm{keep}}=6\times3=18 $$ ### Step 4: mini-batches without the remainder Only the five complete batches are used each epoch. $$ U_{\mathrm{drop}}=5\times3=15 $$ **Final answer:** 3, 309, 18 and 15 updates respectively. --- ## TEE · Drill 18 GD variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s18/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s18 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ \theta_1=1-0.1(1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ \theta_2=0.9-0.1(0.9)=0.81 $$ **Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## TEE · Drill 19 GD variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s19/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s19 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ \theta_1=-0.5-0.1(-1.5)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ \theta_2=-0.35-0.1(-1.35)=-0.215 $$ **Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## TEE · Drill 20 Momentum variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s20/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s20 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ **Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## TEE · Drill 21 Momentum variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s21/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s21 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ v_2=0.9(0.15)-0.1(-1.35)=0.27 $$ $$ \theta_2=-0.35+(0.27)=-0.08 $$ **Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update. --- ## TEE · Drill 22 Nesterov variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s22/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s22 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=1+0.9(0)=1 $$ $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81 $$ $$ g_2=0.81-(0)=0.81 $$ $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ $$ \theta_2=0.9+(-0.171)=0.729 $$ **Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. --- ## TEE · Drill 23 Nesterov variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s23/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s23 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=-0.5+0.9(0)=-0.5 $$ $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=-0.35+0.9(0.15)=-0.215 $$ $$ g_2=-0.215-(1)=-1.215 $$ $$ v_2=0.9(0.15)-0.1(-1.215)=0.2565 $$ $$ \theta_2=-0.35+(0.2565)=-0.0935 $$ **Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. --- ## TEE · Drill 24 AdaGrad variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s24/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s24 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_1=0+(2)^2=4 $$ $$ \sqrt{G_1}=\sqrt{4}=2 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** Add the new squared gradient to all previous squares. $$ G_2=4+(1)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=0.9-\frac{0.1\times(1)}{\sqrt{5}}=0.85527864 $$ --- ## TEE · Drill 25 AdaGrad variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s25/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s25 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Add the new squared gradient to all previous squares. $$ G_1=0+(-1)^2=1 $$ $$ \sqrt{G_1}=\sqrt{1}=1 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_2=1+(2)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=-0.4-\frac{0.1\times(2)}{\sqrt{5}}=-0.489442719 $$ --- ## TEE · Drill 26 RMSProp variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s26/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s26 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(2)^2=0.4 $$ $$ \sqrt{s_1}=\sqrt{0.4}=0.632455532 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{0.4}}=0.683772234 $$ **Step 2: Iteration 2 with supplied gradient 1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.4)+0.1(1)^2=0.46 $$ $$ \sqrt{s_2}=\sqrt{0.46}=0.678232998 $$ $$ \theta_2=0.683772234-\frac{0.1\times(1)}{\sqrt{0.46}}=0.536330278 $$ --- ## TEE · Drill 27 RMSProp variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s27/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s27 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(-1)^2=0.1 $$ $$ \sqrt{s_1}=\sqrt{0.1}=0.316227766 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{0.1}}=-0.183772234 $$ **Step 2: Iteration 2 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.1)+0.1(2)^2=0.49 $$ $$ \sqrt{s_2}=\sqrt{0.49}=0.7 $$ $$ \theta_2=-0.183772234-\frac{0.1\times(2)}{\sqrt{0.49}}=-0.46948652 $$ --- ## TEE · Drill 28 Adam variation 1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s28/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s28 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ --- ## TEE · Drill 29 Adam variation 2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s29/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s29 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ --- ## TEE · Drill 30 Regularized objective Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s30/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s30 ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. --- ## TEE · Drill 31 L1 update on positive and negative weights Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s31/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s31 ## Question **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero. ## Answer — step by step ### Given Old weights = [−0.5,0.3]. Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2. ### To find Find total gradients and the two updated weights. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formula The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j) $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed. **Step 1: Weight 1: old value −0.5.** $$ \lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1 $$ $$ g_{1,\mathrm{total}}=0.2+(-0.1)=0.1 $$ $$ w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52 $$ **Step 2: Weight 2: old value 0.3.** $$ \lambda\operatorname{sign}(0.3)=0.1(1)=0.1 $$ $$ g_{2,\mathrm{total}}=-0.1+0.1=0 $$ $$ w_{2,\mathrm{new}}=0.3-0.2(0)=0.3 $$ **Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero. --- ## TEE · Drill 32 L2 update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s32/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s32 ## Question **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw². ## Answer — step by step ### Given Old weights = [0.5,−0.2]; data gradients = [0.1,0.3]. λ = 0.2; η = 0.1; penalty = (λ/2)Σw². ### To find Find penalty gradients, total gradients and updated weights. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Formula and its reason The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda w_j $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ **Step 1: Weight 1.** $$ \lambda w_1=0.2\times0.5=0.1 $$ $$ g_{1,\mathrm{total}}=0.1+0.1=0.2 $$ $$ w_{1,\mathrm{new}}=0.5-0.1(0.2)=0.48 $$ **Step 2: Weight 2.** $$ \lambda w_2=0.2\times(-0.2)=-0.04 $$ $$ g_{2,\mathrm{total}}=0.3+(-0.04)=0.26 $$ $$ w_{2,\mathrm{new}}=-0.2-0.1(0.26)=-0.226 $$ **Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative. --- ## TEE · Drill 33 Inverted dropout Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s33/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s33 ## Question **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference. ## Answer — step by step ### Given Activations = [2,−1,0,3]. Keep probability q = 0.75; keep mask = [1,0,1,1]. ### To find Find training outputs and state the inference rule. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step ### Meaning and formula The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved. $$ a'_j=\frac{m_j a_j}{q} $$ The multiplication is entry by entry; there is no dot-product sum. ### Step 1: apply the mask $$ [1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3] $$ ### Step 2: divide each result by 0.75 $$ a'_1=\frac2{0.75}=2.666666667 $$ $$ a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0 $$ $$ a'_4=\frac3{0.75}=4 $$ **Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero. ### During inference Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training. --- ## TEE · Drill 34 Convolution with a negative coefficient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s34/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s34 ## Question **Question.** Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | --- | --- | | 1 | −1 | | 0 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel K are printed with the question. Stride S = 1; padding P = 0; bias = 0. Use the kernel without flipping it. ### To find Calculate all nine output cells. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−2) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(-1)+(1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 3 = 3** **2 + 3 = 5** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(-1)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(3)=5 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 2 = 2** **(−1) + 2 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(-1)+(2)=1 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(-1)+(1)=0 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −3 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−3) = −2** Row 2**0 + 0 = 0** **(−2) + 0 = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(-1)=-2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(-2)+(0)=-2 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + (−2) = 1** Row 2**0 + 1 = 1** **1 + 1 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(1)+(1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | −1 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + (−1) = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(1)+(0)=1 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 2 = 2** **1 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(2)=3 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 3 = 3** **(−1) + 3 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(-1)+(3)=2 $$ **Final output:** $\begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. --- ## TEE · Drill 35 Padding stride and channels Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s35/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s35 ## Question **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total. ## Answer — step by step ### Given Input shape = 31 × 31 × 3. Filter size = 5 × 5; number of filters = 8. Stride S = 2; padding P = 1 per side; one bias per filter. ### To find Find output shape, weights, biases and total parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning of each given value The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels. ### Formulas used $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ \text{weights}=K^2C_{\mathrm{in}}C_{\mathrm{out}},\quad \text{biases}=C_{\mathrm{out}} $$ $$ \text{parameters}=\text{weights}+\text{biases} $$ ### Step 1: find the output height and width Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33. $$ N_{\mathrm{out}}=\left\lfloor\frac{31+2(1)-5}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{28}{2}\right\rfloor+1=14+1=15 $$ The height uses the same calculation. **Output shape = 15 × 15 × 8.** ### Step 2: count weights in one filter Each filter covers 5 × 5 positions in each of 3 channels. $$ 5\times5\times3=75\text{ weights per filter} $$ ### Step 3: count all filters and their biases $$ \text{weights}=75\times8=600 $$ $$ \text{biases}=8,\qquad \text{total}=600+8=608 $$ **Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights. --- ## TEE · Drill 36 Max and average pooling Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s36/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s36 ## Question **Question.** Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_563e4ee8f7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | A | 1 | 4 | 2 | 3 | 0 | 2 | 5 | 1 | 6 | 3 | 2 | 0 | 1 | 7 | 4 | 8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input A is printed with the question. Window size = 2 × 2; stride = 2; no padding. ### To find Find the complete max-pooling and average-pooling outputs. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 4, 0, 2] **Step 2: add all selected entries.** 1 + 4 + 0 + 2 = 7 **Step 3: divide by the 4 entries in this window.** 7/4 = **1.75**. Maximum = **4** ### Pooling window (1,2) **Step 1: select the entries.** [2, 3, 5, 1] **Step 2: add all selected entries.** 2 + 3 + 5 + 1 = 11 **Step 3: divide by the 4 entries in this window.** 11/4 = **2.75**. Maximum = **5** ### Pooling window (2,1) **Step 1: select the entries.** [6, 3, 1, 7] **Step 2: add all selected entries.** 6 + 3 + 1 + 7 = 17 **Step 3: divide by the 4 entries in this window.** 17/4 = **4.25**. Maximum = **7** ### Pooling window (2,2) **Step 1: select the entries.** [2, 0, 4, 8] **Step 2: add all selected entries.** 2 + 0 + 4 + 8 = 14 **Step 3: divide by the 4 entries in this window.** 14/4 = **3.5**. Maximum = **8** **Average output:** $\begin{bmatrix}1.75 & 2.75\\4.25 & 3.5\end{bmatrix}$ . **Max output:** $\begin{bmatrix}4 & 5\\7 & 8\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. --- ## TEE · Drill 37 Complete small CNN Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s37/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s37 ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. --- ## TEE · Drill 38 Reverse a spatial formula Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s38/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s38 ## Question **Question.** A stride-1, valid 3 × 3 convolution outputs 18 × 18 × 12. What input spatial size is needed? Is input channel count determined? ## Answer — step by step ### Given Output shape = 18 × 18 × 12. Kernel = 3 × 3; stride = 1; valid padding P = 0. Input channel count is not supplied. ### To find Solve for input height/width; state whether input channels and parameter count are uniquely determined. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Formula used For stride 1 and valid padding, the output width is input width minus kernel width plus one. $$ N_{\mathrm{out}}=N-K+1 $$ ### Step 1: substitute the known output and kernel sizes $$ 18=N-3+1=N-2 $$ ### Step 2: isolate the unknown input size Add 2 to both sides. Doing the same operation to both sides preserves the equality. $$ 18+2=N-2+2 $$ $$ N=20 $$ ### Step 3: interpret the channel information The 12 output channels tell us there are 12 filters. They do not say how many channels the input has. An RGB input and a many-channel intermediate map could both feed 12 filters. With one bias per filter, the parameter formula would be: $$ P=(3\times3\times C_{\mathrm{in}}+1)\times12 $$ **Final answer:** input spatial shape 20 × 20; input channel count is undetermined from these givens. --- ## TEE · Drill 39 Bottleneck cost with new values Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s39/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s39 ## Question **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first. ## Answer — step by step ### Given Input = 16 × 16 × 64. Direct branch: 32 same-padded 3 × 3 filters. Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer. Spatial size remains 16 × 16; exclude bias additions. ### To find Find both MAC counts, absolute saving and percentage saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Meaning and formula A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ ### Step 1: direct 3 × 3 convolution Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 reduction from 64 to 8 channels $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ ### Step 3: 3 × 3 convolution now receives only 8 channels $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: add the two bottleneck stages $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: calculate the saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32. --- ## TEE · Drill 40 Residual addition Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s40/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s40 ## Question **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block. ## Answer — step by step ### Given x = [2,−3,1]; F(x) = [−1,1,0.5]. ### To find Find x+F(x), apply ReLU, and compare with ReLU(F(x)). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join. $$ H(x)=x+F(x),\qquad a_j=\max(0,H_j) $$ ### Step 1: add each pair $$ H_1=2+(-1)=1 $$ $$ H_2=-3+1=-2 $$ $$ H_3=1+0.5=1.5 $$ The residual output before ReLU is **[1, −2, 1.5]**. ### Step 2: apply ReLU $$ [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5] $$ ### Step 3: compare with the plain branch Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**. Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation. --- ## TEE · Drill 41 Two RNN steps Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s41/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s41 ## Question **Question.** h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂. ## Answer — step by step ### Given Initial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0. Inputs: x₁ = 1, x₂ = 2; activation: tanh. ### To find Calculate h₁, then use it to calculate h₂. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ### Meaning and formulas At time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\qquad h_t=\tanh(a_t) $$ The raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward. $$ \tanh(a)=\frac{e^{2a}-1}{e^{2a}+1} $$ ### Step 1: calculate the first raw sum $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ ### Step 2: activate it to obtain the first state $$ h_1=\tanh(0.5)=\frac{e^1-1}{e^1+1} $$ $$ h_1=\frac{2.718281828-1}{2.718281828+1}=0.462117157 $$ ### Step 3: use that state in the second raw sum $$ a_2=0.5(2)+0.2(0.462117157)+0 $$ $$ a_2=1+0.092423431=1.092423431 $$ ### Step 4: activate the second sum $$ h_2=\tanh(1.092423431)=0.797760954 $$ **Final answer:** h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5. --- ## TEE · Drill 42 LSTM with a negative candidate Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s42/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s42 ## Question **Question.** f = 0.8, i = 0.2, o = 0.6, candidate = −0.5, Cprevious = 0.3. Find C, h. ## Answer — step by step ### Given Activated gates: f = 0.8, i = 0.2, o = 0.6. Candidate = −0.5; old cell C = 0.3. ### To find Find the new cell and hidden state. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas An LSTM keeps a cell memory C and a visible hidden state h. The forget gate f controls old memory, the input gate i controls new candidate content, and the output gate o controls the visible state. These gate values are already supplied. $$ C=fC_{\mathrm{previous}}+i\widetilde C,\qquad h=o\tanh(C) $$ The candidate, written C with a tilde, may be negative. A negative candidate subtracts content when admitted. ### Step 1: retained old memory $$ fC_{\mathrm{previous}}=0.8\times0.3=0.24 $$ ### Step 2: admitted new content $$ i\widetilde C=0.2\times(-0.5)=-0.1 $$ ### Step 3: add them to obtain the new cell memory $$ C=0.24+(-0.1)=0.14 $$ ### Step 4: apply tanh to the cell memory $$ \tanh(0.14)=\frac{e^{0.28}-1}{e^{0.28}+1}=0.139092448 $$ ### Step 5: apply the output gate $$ h=0.6\times0.139092448=0.083455469 $$ **Final answer:** cell memory 0.14; hidden state approximately 0.083455469. C and h are different quantities. --- ## TEE · Drill 43 Calculate gates before the cell Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s43/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s43 ## Question **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h. ## Answer — step by step ### Given Raw gate scores: zf = zi = zo = 0. Raw candidate score = 0.5; old cell C = 0.2. ### To find Activate gates/candidate, then calculate the new cell and hidden state. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C) $$ $$ \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C) $$ ### Step 1: turn each zero score into a gate $$ \sigma(0)=\frac1{1+e^0}=\frac12=0.5 $$ Thus f = i = o = 0.5. A zero score is not a zero gate. ### Step 2: calculate candidate content $$ \widetilde C=\tanh(0.5)=0.462117157 $$ ### Step 3: combine retained memory and new content $$ fC_{\mathrm{previous}}=0.5\times0.2=0.1 $$ $$ i\widetilde C=0.5\times0.462117157=0.231058579 $$ $$ C=0.1+0.231058579=0.331058579 $$ ### Step 4: compute the visible hidden state $$ \tanh(C)=\tanh(0.331058579)\approx0.319471636 $$ $$ h=0.5\times0.319471636=0.159735818 $$ **Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818. --- ## TEE · Drill 44 GRU reset then mix Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s44/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s44 ## Question **Question.** hprevious = 0.4, r = 0.5, z = 0.25, x = 1, W = 0.2, U = 0.5, b = 0. Use candidate = tanh(Wx+U(r × hprevious)) and h = (1−z) hprevious+z candidate. ## Answer — step by step ### Given Old hidden state = 0.4; reset r = 0.5; update z = 0.25. x = 1; W = 0.2; U = 0.5; b = 0. Use h = (1−z)h\_old + z×candidate. ### To find Find reset contribution, candidate and new hidden state. ### Formula — which one and why **GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate. r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state. $$ \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b) $$ $$ h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h $$ **Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control. ### Solution — step by step ### Meaning and given convention The reset gate r scales the old state while making the candidate. The update gate z controls the amount of candidate used in the final mixture **under this question's convention**. Some books reverse the roles of z and 1 − z, so copy the supplied formula. $$ \widetilde h=\tanh\left(Wx+U(rh_{\mathrm{previous}})+b\right) $$ $$ h=(1-z)h_{\mathrm{previous}}+z\widetilde h $$ ### Step 1: reset the old state for candidate construction $$ rh_{\mathrm{previous}}=0.5\times0.4=0.2 $$ ### Step 2: form the candidate's raw score $$ Wx=0.2\times1=0.2 $$ $$ U(rh_{\mathrm{previous}})=0.5\times0.2=0.1 $$ $$ a=0.2+0.1+0=0.3 $$ ### Step 3: apply tanh $$ \widetilde h=\tanh(0.3)=0.291312612 $$ ### Step 4: mix the original old state with the candidate $$ 1-z=1-0.25=0.75 $$ $$ h=0.75(0.4)+0.25(0.291312612) $$ $$ h=0.3+0.072828153=0.372828153 $$ **Final answer: h ≈ 0.372828153.** The final old-state term uses 0.4, not the reset value 0.2. --- ## TEE · Drill 45 BPTT shared weights Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s45/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s45 ## Question **Question.** Linear recurrence hₜ = uxₜ+vhₜ₋₁+b, output h₂, target 1, half squared loss. u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1, x₂ = 2, η = 0.1. Find both states and one update. ## Answer — step by step ### Given Linear recurrence hₜ = uxₜ+vhₜ₋₁+b. u = 0.2; v = 0.5; b = 0; h₀ = 0. x₁ = 1; x₂ = 2; target for h₂ = 1. Half squared loss; learning rate η = 0.1. ### To find Find both states, all three shared-parameter gradients and one simultaneous update. ### Formula — which one and why **Backpropagation through time.** Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once. For h\_t=ux\_t+vh\_(t−1)+b with loss only at the last state: δ\_t = loss derivative at that state. The linear recurrence has activation slope 1. $$ \delta_T=h_T-y $$ $$ \delta_t=v\delta_{t+1} $$ $$ g_u=\sum_t\delta_tx_t $$ $$ g_v=\sum_t\delta_th_{t-1} $$ $$ g_b=\sum_t\delta_t $$ **Yaad rakhne ka cue:** Time ke har use ka gradient jodo; shared weight ko ek baar update karo. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas The same u, v and b appear at both times. u multiplies the input, v multiplies the previous state, and b is a shared bias. The loss is only at the final state h₂. Because the recurrence is linear, its local activation derivative is 1. $$ h_t=ux_t+vh_{t-1}+b,\qquad L=\frac12(h_2-y)^2 $$ $$ \delta_2=h_2-y,\qquad \delta_1=v\delta_2 $$ Each shared parameter receives a contribution from both of its uses. $$ \frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2 $$ $$ \frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1,\qquad \frac{\partial L}{\partial b}=\delta_1+\delta_2 $$ ### Step 1: forward through time 1 $$ h_1=0.2(1)+0.5(0)+0=0.2 $$ ### Step 2: forward through time 2 $$ h_2=0.2(2)+0.5(0.2)+0=0.4+0.1=0.5 $$ $$ L=\frac12(0.5-1)^2=0.125 $$ ### Step 3: carry the gradient backwards $$ \delta_2=0.5-1=-0.5 $$ $$ \delta_1=0.5(-0.5)=-0.25 $$ ### Step 4: add contributions for each shared parameter $$ \frac{\partial L}{\partial u}=(-0.25)(1)+(-0.5)(2)=-0.25-1=-1.25 $$ $$ \frac{\partial L}{\partial v}=(-0.25)(0)+(-0.5)(0.2)=0-0.1=-0.1 $$ $$ \frac{\partial L}{\partial b}=-0.25-0.5=-0.75 $$ ### Step 5: update once after summing $$ u_{\mathrm{new}}=0.2-0.1(-1.25)=0.325 $$ $$ v_{\mathrm{new}}=0.5-0.1(-0.1)=0.51 $$ $$ b_{\mathrm{new}}=0-0.1(-0.75)=0.075 $$ **Final answer:** states 0.2 and 0.5; new parameters u = 0.325, v = 0.51, b = 0.075. --- ## TEE · Drill 46 Mixed convolution pool dense and loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s46/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s46 ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. --- ## TEE · Drill 47 Mixed backpropagation and L2 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s47/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s47 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. --- ## TEE · Drill 48 Mixed dropout and a momentum update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s48/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s48 ## Question **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1. ## Answer — step by step ### Given Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5. Linear-output weights = [0.2,0.3]; bias = 0; target = 1. Half squared loss; update only the output weights. Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1. ### To find Find dropped/scaled activations, prediction, gradients, new velocities and new output weights. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Formulas used Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients. $$ a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j $$ The signed-velocity momentum convention is: $$ v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}} $$ ### Step 1: apply inverted dropout $$ a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0 $$ ### Step 2: prediction and error $$ o=0.2(2)+0.3(0)+0=0.4 $$ $$ o-y=0.4-1=-0.6 $$ ### Step 3: current weight gradients $$ g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0 $$ ### Step 4: remembered movement plus new correction $$ v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21 $$ $$ v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09 $$ ### Step 5: add velocities to old weights $$ w_{1,\mathrm{new}}=0.2+0.21=0.41 $$ $$ w_{2,\mathrm{new}}=0.3-0.09=0.21 $$ **Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero. --- ## TEE · Drill 49 Mixed CNN dimensions softmax and loss Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s49/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s49 ## Question **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE. ## Answer — step by step ### Given Input = 8 × 8 × 3. Conv: kernel 3, stride 1, padding 1, 2 filters. Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs. Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]). ### To find Find shapes, parameter counts, probabilities and CCE. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Formulas used Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}} $$ Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.367879441, 1, 0.135335283]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Correct class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**. --- ## TEE · Drill 50 Three step LSTM retention and replacement Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s50/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s50 ## Question **Question.** C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step. ## Answer — step by step ### Given Initial C₀ = 0; output gate o = 0.7 at all three steps. Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6). ### To find Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas At each time step use the previous **cell state C**, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step. $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ **Step 1: Time 1: admit new content.** $$ f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4 $$ $$ C_1=0+0.4=0.4 $$ $$ h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274 $$ **Step 2: Time 2: retain memory and admit no new content.** $$ f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0 $$ $$ C_2=0.36+0=0.36 $$ $$ h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824 $$ **Step 3: Time 3: retain a small part and add negative content.** $$ f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3 $$ $$ C_3=0.072-0.3=-0.228 $$ $$ h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772 $$ **Final cell states:** 0.4, 0.36, −0.228. **Final hidden states:** approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given. --- ## TEE · Drill 51 Inception concatenation and parameter count Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s51/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s51 ## Question **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters. ## Answer — step by step ### Given Input = 10 × 10 × 8. Branch A: 1 × 1 convolution, 4 filters. Branch B: 3 × 3 same convolution, 6 filters. Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters. ### To find Find each branch shape and parameters, then the concatenated shape and total parameters. ### Formula — which one and why **Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches. C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions. $$ C_{\mathrm{out}}=\sum_k C_k $$ $$ P_{\mathrm{total}}=\sum_k P_k $$ **Yaad rakhne ka cue:** Channels jodo; height-width same rakho. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values. $$ C_{\mathrm{combined}}=C_A+C_B+C_C $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Assume one bias per filter. Pooling has no learned parameters. **Step 1: Branch A.** Its 4 filters produce shape 10 × 10 × 4. $$ P_A=(1\times1\times8+1)\times4=9\times4=36 $$ **Step 2: Branch B.** Its 6 filters produce shape 10 × 10 × 6. $$ P_B=(3\times3\times8+1)\times6=73\times6=438 $$ **Step 3: Branch C.** Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2. $$ P_C=0+(1\times1\times8+1)\times2=18 $$ **Step 4: Join the shapes and add the parameters.** $$ C_{\mathrm{combined}}=4+6+2=12 $$ $$ P_{\mathrm{total}}=36+438+18=492 $$ **Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches. --- ## TEE · Drill 52 Projection shortcut Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s52/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s52 ## Question **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters. ## Answer — step by step ### Given Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16. Shortcut kernel = 1 × 1; stride 2; padding 0. Use one bias per projection filter. ### To find Choose the number of filters and find projection output shape and parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ ### Step 1: choose 16 filters One filter creates one channel, so 16 filters give the required 16 output channels. ### Step 2: use the specified 1 × 1 kernel and stride 2 $$ N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8 $$ The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1. ### Step 3: count weights and biases $$ \text{weights}=1\times1\times8\times16=128 $$ $$ \text{biases}=16,\qquad \text{total}=128+16=144 $$ **Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid. --- ## TEE · Revising solved examples Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c17-s53/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/17/tee-c17-s53 Revisit a direct example and then its changed-number version. Write the given values, the formula and the substitutions on separate lines. If an answer differs, compare from the first line until the difference appears. Use the mixed questions after the individual methods make sense. --- ## TEE · Start here Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-intro/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-intro Three hours; 100 marks. Answer Q1 and any four of Q2–Q7. Every question below is immediately followed by its full solution. This is the supplied December 2025 historical paper used for a timed rehearsal. F25 = S12, 3 December 2025, with S13 as a comparison answer key. S18 is another copy of this same paper. All questions are included; its historical choice rule does not remove questions from this practice bank. Questions below preserve the assessed tasks and numerical data, with wording and diagram layout normalized for readability. Full question statements and their necessary diagrams appear with the solutions. Read the full question, then follow the worked answer line by line. Solutions are independently worked; a historical answer key is not treated as infallible. --- ## TEE · Rehearsal Q1a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s0 **Question (5 marks).** Give four ML versus DL differences with examples. Choose and justify ML or DL for (1) 2,000 short product reviews, (2) labeled financial fraud records, (3) captions for 100,000 labeled natural-scene images. **Complete worked answer.** Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. **Choices:** (1) Start with traditional ML using text features because the labeled dataset is small; pretrained text models are a viable extension. (2) Start with a tabular ML classifier such as a tree ensemble; labeled structured features do not automatically require DL. (3) DL is appropriate for learning visual features and generating a word sequence; a CNN encoder with a sequence decoder is a syllabus-aligned design. Compare held-out metrics rather than claiming one method must always win. --- ## TEE · Rehearsal Q1b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s1 **Question (5 marks).** For automated leaf-disease image classification, discuss factors affecting loss selection. For classes Normal, Mild, Severe with target [0,1,0] and predicted probabilities [0.1,0.8,0.1], compute and interpret categorical cross-entropy. **Complete worked answer.** ### Given Classes = Normal,Mild,Severe; target = [0,1,0]. Predicted probabilities = [0.1,0.8,0.1]; use natural logarithms. ### To find Calculate categorical cross-entropy and interpret the result. ### Formula — which one and why **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step Select a loss according to the task and output meaning: real-valued regression often uses MSE/MAE, binary classification uses binary cross-entropy, and one-of-K classification uses categorical cross-entropy. Consider class imbalance, noisy labels/outliers, whether outputs are calibrated probabilities, and differentiability for optimization. **Step 1: expand the categorical cross-entropy sum.** $$ L=-[0\ln(0.1)+1\ln(0.8)+0\ln(0.1)] $$ **Step 2: keep the term for the true class.** $$ L=-\ln(0.8) $$ **Step 3: calculate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 4: negate the logarithm.** $$ L\approx0.223143551 $$ **Final answer:** loss ≈ 0.223144 nats. The true class received probability 0.8, so the loss is relatively small; probability 1 would give loss zero. --- ## TEE · Rehearsal Q1c Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s2 **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization. **Complete worked answer.** Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. --- ## TEE · Rehearsal Q1d Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s3 **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate. **Complete worked answer.** A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. --- ## TEE · Rehearsal Q2a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s4 **Question (10 marks).** A 2 → 2 → 1 sigmoid network has all four input-to-hidden weights 0.5, both hidden-to-output weights 0.25, hidden and output biases 0, and target 1. Perform forward and backward passes to update the x₁ → h₁ weight, h₁ → output weight and output bias at η = 0.5. The paper does not state the numerical inputs or loss. ![F25 Q2a connections with explicitly assumed inputs](https://deep-learning-study-room.vercel.app/assets/figures/net_f25.svg) Diagram: F25 Q2a connections with explicitly assumed inputs. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=1 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.5 + x2 × 0.5; bias 0 | h2 incoming: x1 × 0.5 + x2 × 0.5; bias 0 | o1 incoming: h1 × 0.25 + h2 × 0.25; bias 0 | Use these OLD connections for all gradients in the same pass. F25 Q2a connections with explicitly assumed inputs Tap the diagram to enlarge. **Complete worked answer.** ### Given Architecture 2 → 2 → 1; every neuron uses sigmoid. All four input-to-hidden weights = 0.5; both hidden-to-output weights = 0.25. All biases = 0; target = 1; η = 0.5. The paper omits inputs and loss: this worked solution explicitly assumes x = [1,1] and half squared loss. ### To find Calculate the forward pass, then gradients and updates for x₁→h₁, h₁→output and the output bias. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Assumptions:** use x = [1,1] and half squared loss. The paper permits suitable assumptions and its key also introduces assumed inputs. There is no unique numerical result without them. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.5 \times 1) + (0.5 \times 1) + (0) = 1 $$ $$ h_{1}=\frac{1}{1+e^{-(1)}}=0.731058579 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.5 \times 1) + (0.5 \times 1) + (0) = 1 $$ $$ h_{2}=\frac{1}{1+e^{-(1)}}=0.731058579 $$ **Output neuron 1:** $$ z_{o_1} = (0.25 \times 0.731058579) + (0.25 \times 0.731058579) + (0) = 0.365529289 $$ $$ o_{1}=\frac{1}{1+e^{-(0.365529289)}}=0.590378257 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.409621743]. Squared errors = [0.167789972]. Half-sum squared loss = 0.083894986; mean squared error = 0.167789972. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.590378257-1) \times 0.241831771 = -0.099059551 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.731058579**h1 × Receiving delta**−0.099059551**δ at o1 = Connection gradient**−0.072418335**h1 → o1 Now update this weight Old weight**0.25** → New weight**0.286209167** Subtract η × gradient = **−0.036209168**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.099059551)\times(0.731058579)\approx -0.072418335 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.25-0.5(-0.072418335)\approx 0.286209167 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.731058579**h2 × Receiving delta**−0.099059551**δ at o1 = Connection gradient**−0.072418335**h2 → o1 Now update this weight Old weight**0.25** → New weight**0.286209167** Subtract η × gradient = **−0.036209168**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.099059551)\times(0.731058579)\approx -0.072418335 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.25-0.5(-0.072418335)\approx 0.286209167 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.099059551** × Old outgoing weight**0.25** = Path contribution**−0.024764888** ↓ Add every path contribution**−0.024764888**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.25)\times(-0.099059551)\approx -0.024764888 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.024764888 $$ $$ \delta_{h_1} = -0.024764888 \times 0.196611933 = -0.004869072 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.099059551** × Old outgoing weight**0.25** = Path contribution**−0.024764888** ↓ Add every path contribution**−0.024764888**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.25)\times(-0.099059551)\approx -0.024764888 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.024764888 $$ $$ \delta_{h_2} = -0.024764888 \times 0.196611933 = -0.004869072 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.004869072**δ at h1 = Connection gradient**−0.004869072**x1 → h1 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**1**x2 × Receiving delta**−0.004869072**δ at h1 = Connection gradient**−0.004869072**x2 → h1 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.004869072**δ at h2 = Connection gradient**−0.004869072**x1 → h2 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**1**x2 × Receiving delta**−0.004869072**δ at h2 = Connection gradient**−0.004869072**x2 → h2 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0-0.5(-0.004869072)=0.002434536 $$ $$ b_{h_2,\mathrm{new}}=0-0.5(-0.004869072)=0.002434536 $$ $$ b_{o_1,\mathrm{new}}=0-0.5(-0.099059551)=0.049529775 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Requested values:** input weight **0.502434536**, output weight **0.286209167**, output bias **0.049529775**. Other supplied inputs produce different correct conditional results. --- ## TEE · Rehearsal Q2b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s5 **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction. **Complete worked answer.** A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs. --- ## TEE · Rehearsal Q3a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s6 **Question (10 marks).** Train one epoch of a single perceptron with η = 1, initial weights and bias 0, threshold 0, on [−1,1,−1] → 0, [−1,1,1] → 1, [1,−1,1] → 1, [−1,1,1] → 1. Then map biological-neuron components to artificial-neuron components. **Complete worked answer.** ### Given Initial weights and bias = 0; η = 1; threshold = 0. Samples in order: [−1,1,−1]→0; [−1,1,1]→1; [1,−1,1]→1; [−1,1,1]→1. Use output 1 when net > 0 and 0 otherwise, matching the supplied key; one epoch. ### To find Process each sample in order, carrying updated weights forward, and report final weights/bias. ### Formula — which one and why **Perceptron learning rule.** Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters. t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question. $$ z=\sum_iw_ix_i+b $$ $$ w_i^{\mathrm{new}}=w_i+\eta(t-o)x_i $$ $$ b^{\mathrm{new}}=b+\eta(t-o) $$ **Yaad rakhne ka cue:** Prediction nikalo; target−prediction se correction nikalo. ### Solution — step by step **State the threshold rule.** Use binary output 1 if z>0 and 0 otherwise, matching the associated key. Update Δw = η(t−o)x and Δb = η(t−o), with η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly. ### Epoch 1, sample 1 **Step 1: copy this sample and target.** x = [-1, 1, -1], target = 0. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, -1] = [0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [0, 0, 0], new b = 0. ### Epoch 1, sample 2 **Step 1: copy this sample and target.** x = [-1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (0 × 1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [-1, 1, 1] = [-1, 1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [-1, 1, 1], new b = 1. ### Epoch 1, sample 3 **Step 1: copy this sample and target.** x = [1, -1, 1], target = 1. **Step 2: calculate the current net.** z = (-1 × 1) + (1 × -1) + (1 × 1) + (1) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [1, -1, 1] = [1, -1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [0, 0, 2], new b = 2. ### Epoch 1, sample 4 **Step 1: copy this sample and target.** x = [-1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (2 × 1) + (2) = 4 **Step 3: apply the threshold.** output = 1. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, 1] = [0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [0, 0, 2], new b = 2. **Final after exactly the requested epochs:** w = **[0, 0, 2]**, b = **2**. A finite number of requested epochs does not itself prove convergence or correct classification of every sample. Biological dendrites receive signals, synapses influence connection strength, the cell body integrates information, and an axon carries an output signal. In the artificial analogy, inputs represent received signals, weights represent connection strengths, the weighted sum plus bias performs aggregation, and an activation function produces an output. The bias shifts the firing threshold. This is a simplified computational analogy, not a complete biological model. ![The artificial neuron multiplies each input by a weight then adds bias and activates](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: The artificial neuron multiplies each input by a weight then adds bias and activates. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate The artificial neuron multiplies each input by a weight then adds bias and activates Tap the diagram to enlarge. --- ## TEE · Rehearsal Q3b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s7 **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning. **Complete worked answer.** Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. --- ## TEE · Rehearsal Q4a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s8 **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters. **Complete worked answer.** ### Given N = 1,000; mini-batch size = 100. J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9. Use signed velocity. ### To find Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Step 1: count full-batch updates.** All 1,000 samples contribute to one update: 1 per epoch. **Step 2: count stochastic updates.** One update per sample: 1,000 per epoch. **Step 3: count mini-batches.** $$ 1000/100=10 $$ Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ. ### Momentum iteration 1 **Step 1: evaluate the gradient at the current parameter.** $$ g_1=1 $$ **Step 2: retain the previous displacement.** $$ 0.9(0)=0 $$ **Step 3: scale the current gradient.** $$ 0.1(1)=0.1 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_1=0-0.1=-0.1 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_1=1+(-0.1)=0.9 $$ ### Momentum iteration 2 **Step 1: evaluate the gradient at the current parameter.** $$ g_2=0.9 $$ **Step 2: retain the previous displacement.** $$ 0.9(-0.1)=-0.09 $$ **Step 3: scale the current gradient.** $$ 0.1(0.9)=0.09 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_2=-0.09-0.09=-0.18 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_2=0.9+(-0.18)=0.72 $$ **Hyperparameters:** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update. --- ## TEE · Rehearsal Q4b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s9 **Question (10 marks).** Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1]. **Complete worked answer.** ### Given Forget-gate outputs f = [0.1,0.9]; old cell = [2,1]. No input gate or candidate is supplied. ### To find Explain forgetting/retention and calculate the retained old-cell part only. ### Formula — which one and why **LSTM retained old memory.** Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell. f = activated forget gate; C\_old = previous cell state; ⊙ means multiply matching entries. $$ C_{\mathrm{retained}}=f\odot C_{\mathrm{old}} $$ **Yaad rakhne ka cue:** Forget gate × old cell = bacha hua purana hissa. ### Solution — step by step Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph. 1. Initialize the hidden state and run the forward recurrence from the first input to the last. 2. Compute the specified output loss, at the final step or across several steps. 3. Start from the last loss and propagate derivatives backwards through output and hidden operations. 4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states. 5. Sum all uses of each shared weight before applying the optimizer update. $$ \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T} $$ The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step. ![BPTT follows this unrolled network backwards and sums gradients of shared weights](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: BPTT follows this unrolled network backwards and sums gradients of shared weights. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge. The plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path. **Step 1: retain the first old-cell component.** $$ 0.1(2)=0.2 $$ **Step 2: retain the second old-cell component.** $$ 0.9(1)=0.9 $$ **Step 3: assemble the retained old-cell contribution.** $$ f\odot C_{\mathrm{old}}=[0.2,0.9] $$ **Final answer:** [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated. --- ## TEE · Rehearsal Q5a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s10 **Question (10 marks).** State the purpose of each LSTM gate. For “Artificial Intelligence has revolutionized every domain. The work force will have to update themselves to remain relevant,” explain a context change. Calculate C and h given i = 0.7, f = 0.4, o = 0.8, candidate = 0.9, Cprevious = 0.5. Why is LSTM more effective than a simple RNN for language modeling? **Complete worked answer.** ### Given Activated gate values: i = 0.7,f = 0.4,o = 0.8. Candidate = 0.9; old cell = 0.5. ### To find Explain the context change in the question; calculate the new cell and hidden state. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. The second sentence shifts attention toward the workforce while retaining useful prior AI context. Input gates can admit the new subject; forget gates can reduce obsolete details. **Step 1: retain old memory.** $$ fC_{old}=0.4(0.5)=0.2 $$ **Step 2: admit candidate content.** $$ i\widetilde C=0.7(0.9)=0.63 $$ **Step 3: add the two contributions.** $$ C=0.2+0.63=0.83 $$ **Step 4: apply tanh to the updated cell.** $$ \tanh(0.83)\approx0.680476006 $$ **Step 5: multiply by the output gate.** $$ h=0.8(0.680476006)\approx0.544380805 $$ **Final answer:** C = 0.83 and h ≈ 0.544380805. The supplied gates are already activated; do not apply sigmoid again. --- ## TEE · Rehearsal Q5b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s11 **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD. **Complete worked answer.** AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise: $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change. Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864. Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average. --- ## TEE · Rehearsal Q6a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s12 **Question (10 marks).** Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. Given matrices Input A Input A | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Stride 2; P = 0 (no padding). **Complete worked answer.** ### Given Input 128 × 128 × 3. Conv K = 3, filters = 16, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 3, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Dense 240 units. Dense 120 units. Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. ### To find Find every layer shape, its parameter count and the network total. Find every convolution output cell. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**128 × 128 × 3**height × width × channels → Convolution**16 filters · 3 × 3 · S = 1 · P = 0** → Output shape**126 × 126 × 16**height × width × channels Weights**432** + Biases**16** = Trainable parameters**448** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **128 × 128 × 3**. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{128+2(0)-3}{1}\right\rfloor+1=126 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **126 × 126 × 16**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times3\times16=432 $$ $$ N_{\mathrm{biases}}=16 $$ $$ P_1=432+16=448 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**126 × 126 × 16**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**63 × 63 × 16**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **126 × 126 × 16**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{126-2}{2}\right\rfloor+1=63 $$ Pooling acts independently on each channel. The output is **63 × 63 × 16**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**63 × 63 × 16**height × width × channels → Convolution**32 filters · 3 × 3 · S = 1 · P = 0** → Output shape**61 × 61 × 32**height × width × channels Weights**4,608** + Biases**32** = Trainable parameters**4,640** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **63 × 63 × 16**. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{63+2(0)-3}{1}\right\rfloor+1=61 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **61 × 61 × 32**. **3. Count trainable values.** Each filter receives all 16 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times16\times32=4{,}608 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_3=4{,}608+32=4{,}640 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**61 × 61 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**30 × 30 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **61 × 61 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{61-2}{2}\right\rfloor+1=30 $$ Pooling acts independently on each channel. The output is **30 × 30 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 5 — flatten Layer 5: what changes and what is learned Input shape**30 × 30 × 32**height × width × channels → Flatten**Rearrange activations** → Output shape**28,800**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **30 × 30 × 32**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=30\times30\times32=28{,}800 $$ Output: a vector with **28,800 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 6 — fully connected layer Layer 6: what changes and what is learned Input shape**28,800**input features → Dense layer**240 neurons** → Output shape**240**output features Weights**6,912,000** + Biases**240** = Trainable parameters**6,912,240** Every output neuron receives every input feature, with one bias per output neuron. Input: **28,800 features**. Output: **240 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=28{,}800\times240=6{,}912{,}000 $$ $$ N_{\mathrm{biases}}=240 $$ $$ P_6=6{,}912{,}000+240=6{,}912{,}240 $$ ### Layer 7 — fully connected layer Layer 7: what changes and what is learned Input shape**240**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**28,800** + Biases**120** = Trainable parameters**28,920** Every output neuron receives every input feature, with one bias per output neuron. Input: **240 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=240\times120=28{,}800 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_7=28{,}800+120=28{,}920 $$ ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**120**input features → Dense layer**5 neurons** → Output shape**5**output features Weights**600** + Biases**5** = Trainable parameters**605** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **5 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times5=600 $$ $$ N_{\mathrm{biases}}=5 $$ $$ P_8=600+5=605 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{128+2 \times 0-3}{1}\right\rfloor+1$ | 126 x 126 x 16 | 432 | 16 | 448 | | 2 pool | $\left\lfloor\frac{126-2}{2}\right\rfloor+1$ | 63 x 63 x 16 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{63+2 \times 0-3}{1}\right\rfloor+1$ | 61 x 61 x 32 | 4,608 | 32 | 4,640 | | 4 pool | $\left\lfloor\frac{61-2}{2}\right\rfloor+1$ | 30 x 30 x 32 | 0 | 0 | 0 | | 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 | | 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 | | 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 | | 8 dense | 120 → 5 | 5 | 600 | 5 | 605 | **Parameter substitutions, in layer order:** Layer 1: (3 x 3 x 3 + 1) x 16 = 448. Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640. Layer 6: (28800 + 1) x 240 = 6,912,240. Layer 7: (240 + 1) x 120 = 28,920. Layer 8: (120 + 1) x 5 = 605. **Total:** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. **Separate convolution calculation:** **Step 1 shape:** $\left\lfloor\frac{5+2 \times 0-3}{2}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −2 | | 4 | 0 | −4 | | 3 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−2) = 1** Row 2**4 + 0 + (−4) = 0** Row 3**3 + 0 + (−1) = 2** **1 + 0 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(1)\times(0)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(2)\times(-2)=0 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(0)+(2)=3 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 2 | 0 | −1 | | 4 | 0 | 0 | | 1 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 + (−1) = 1** Row 2**4 + 0 + 0 = 4** Row 3**1 + 0 + (−1) = 0** **1 + 4 + 0 = 5** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(2)\times(0)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(0)\times(-2)=4 $$ **Window row 3.** $$ R_3=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(1)+(4)+(0)=5 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −1 | | 0 | 0 | 0 | | 1 | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−1) = 2** Row 2**0 + 0 + 0 = 0** Row 3**1 + 0 + 0 = 1** **2 + 0 + 1 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Window row 2.** $$ R_2=(0)\times(2)+(3)\times(0)+(0)\times(-2)=0 $$ **Window row 3.** $$ R_3=(1)\times(1)+(1)\times(0)+(0)\times(-1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(2)+(0)+(1)=3 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 1 | 0 | −1 | | 0 | 0 | −4 | | 0 | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 + (−1) = 0** Row 2**0 + 0 + (−4) = −4** Row 3**0 + 0 + (−3) = −3** **0 + (−4) + (−3) = −7** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0)\times(2)+(1)\times(0)+(2)\times(-2)=-4 $$ **Window row 3.** $$ R_3=(0)\times(1)+(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(0)+(-4)+(-3)=-7 $$ **Final output:** $\begin{bmatrix}3 & 5\\3 & -7\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. --- ## TEE · Rehearsal Q6b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s13 **Question (10 marks).** Explain activations and the role of nonlinearity. Compare sigmoid, tanh, ReLU and Leaky ReLU including advantages and disadvantages. **Complete worked answer.** An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. --- ## TEE · Rehearsal Q7a Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s14 **Question (10 marks).** Explain feedforward information flow, weights, biases and activations, then calculate the output. Input x = [1, 0]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_412d332265.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 | W₂ | 0.5 | 0.6 Given matrices for this question Tap the diagram to enlarge. **Complete worked answer.** ### Given Input x = [1, 0]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. ### To find Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step A feedforward network transforms inputs layer by layer with no recurrent feedback. Weights scale contributions; bias shifts the preactivation; activation supplies the nonlinear transformation. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.4 \times 0) + (0.1) = 0.3 $$ $$ h_{1} = \max(0,0.3) = 0.3 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (0.7 \times 0) + (0.2) = 0.5 $$ $$ h_{2} = \max(0,0.5) = 0.5 $$ **Output neuron 1:** $$ z_{o_1} = (0.5 \times 0.3) + (0.6 \times 0.5) + (0.1) = 0.55 $$ $$ o_{1} = \max(0,0.55) = 0.55 $$ **Matrix-orientation note:** this solution uses column input and rows as receiving neurons, giving **0.55**. The historical key uses the transpose interpretation for W₁ and obtains 0.61. If the problem does not define its orientation, state your convention; do not silently swap it during calculation. Under the alternative: hidden values = [0.3,0.6], output = 0.5 × 0.3+0.6 × 0.6+0.1 = 0.61. --- ## TEE · Rehearsal Q7b Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c18-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/18/tee-c18-s15 **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages. **Complete worked answer.** A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification. ![One filter is reused at successive spatial positions](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: One filter is reused at successive spatial positions. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. One filter is reused at successive spatial positions Tap the diagram to enlarge. Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities. Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation. For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions. --- ## TEE · The one page method map Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c19-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/19/tee-c19-s0 | Question clue | First action | Core formula or check | | --- | --- | --- | | Neuron output | Multiply each input by its matching weight | $z=\sum_iw_ix_i+b$ , then activate | | Network prediction | Finish one whole layer before the next | $h=f(Wx+b)$ | | Backpropagation | State loss then prediction-minus-target | delta times sending activation | | Optimizer update | Identify current gradient and stored state | old parameter minus update | | CNN dimensions | Write N K S P and channels | $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ | | CNN parameters | Count one filter then all filters | $(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}}$ | | Pooling | Draw every valid window | Maximum, or mean of the window | | Dense parameters | Count connections and receiving biases | $(N_{\mathrm{in}}+1)N_{\mathrm{out}}$ | | RNN hidden state | Add new-input and old-state transforms | $\tanh(W_{xh}x+W_{hh}h_{t-1}+b)$ | | LSTM | Retain old add new then expose | Retain old cell, add admitted candidate, then apply output gate | | GRU | Check the stated z convention | $h=(1-z)h_{t-1}+z\widetilde h$ | --- ## TEE · All formulas: meanings, use and substitution Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c19-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/19/tee-c19-s1 These are the formulas used by this course. Each entry names its symbols and the type of calculation it serves. Numerical practice in the teaching chapters shows the longer working. Follow any different convention explicitly stated in the examination. ### Formula 1: Mean, variance and standard deviation $$ \mu=\frac{1}{n}\sum_i x_i,\quad v=\frac{1}{n}\sum_i(x_i-\mu)^2,\quad s=\sqrt{v} $$ **Symbols:** n is the count; mu is the mean; v is the population variance; s is the standard deviation. **Use:** Use when calculating a population mean/variance or standardization. Sample-variance estimates may use n−1 when explicitly requested. **Substitution example:** For [2,4]: mean = 3; deviations = −1,1; squares = 1,1; variance = 1; standard deviation = 1. ### Formula 2: Standardization $$ x_{scaled}=\frac{x-\mu_{train}}{s_{train}} $$ **Symbols:** Use the mean and standard deviation estimated from training data. **Use:** Use to express a value relative to the training distribution. The standard deviation must be nonzero. **Substitution example:** For x = 7, mean = 5, standard deviation = 2: (7−5)/2 = 1. ### Formula 3: Power, constant and sum derivatives $$ \frac{d}{dx}(cx^n)=cnx^{n-1},\quad\frac{d}{dx}c=0,\quad (f+g)\prime=f\prime+g\prime $$ **Symbols:** c is a constant, x is the changing variable and n is the power. **Use:** Use to differentiate the loss supplied in a gradient-descent question. **Substitution example:** For 0.7x³: derivative = 0.7 × 3 × x² = 2.1x². ### Formula 4: Chain rule $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ **Symbols:** z is the intermediate value linking w to L. **Use:** Multiply derivatives along a path; add contributions from distinct paths that meet. **Substitution example:** For z = 3w+1 and L = z² at w = 1: z = 4; derivative = (2×4)×3 = 24. ### Formula 5: Dot product and a neuron $$ z=\sum_i w_i x_i+b,\qquad a=f(z) $$ **Symbols:** x is the input; w is its matching weight; b is one bias; z is the weighted sum; a is the activated output. **Use:** Multiply each matching pair, add the products, add bias once, then activate. **Substitution example:** x = [2,1], w = [0.5,−0.2], b = 0.1: z = 1−0.2+0.1 = 0.9. ### Formula 6: Matrix multiplication $$ (AB)_{ij}=\sum_k A_{ik}B_{kj} $$ **Symbols:** i selects a row of A; j selects a column of B; k runs along the matching entries. **Use:** Use row-times-column sums. A with shape m×n and B with n×p produces m×p. **Substitution example:** Row [1,2] times column [3,4] gives 1×3+2×4 = 11. ### Formula 7: Layer forward pass $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ **Symbols:** l labels the layer; the previous layer activations become inputs to the next layer. **Use:** For receiving-neuron rows, compute each row dot product and add that receiving neuron’s bias. Finish the whole layer before moving on. ### Formula 8: Sigmoid and its derivative $$ \sigma(z)=\frac{1}{1+e^{-z}},\quad \sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ **Symbols:** e is approximately 2.71828; sigma is the output. **Use:** Use for a sigmoid forward pass and for its local backpropagation derivative. **Substitution example:** At z = 0: output = 1/(1+1) = 0.5; derivative = 0.5×0.5 = 0.25. ### Formula 9: Tanh and its derivative $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1},\quad \tanh\prime(z)=1-\tanh^2(z) $$ **Symbols:** Tanh gives an output between −1 and 1. **Use:** Use for a stated tanh activation, commonly an RNN hidden state or recurrent candidate. **Substitution example:** At z = 0: numerator = 0, denominator = 2; output = 0; derivative = 1. ### Formula 10: ReLU and its derivative $$ f(z)=\max(0,z),\quad f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ **Symbols:** max means choose the larger value. **Use:** Use for ReLU. At zero the derivative is mathematically undefined; use the convention stated by the question or implementation. **Substitution example:** For −2, output = 0 and derivative = 0. For 3, output = 3 and derivative = 1. ### Formula 11: Leaky ReLU $$ f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases} $$ **Symbols:** a is the small negative-side slope, commonly 0.01 unless specified. **Use:** For negative z multiply by a; the output remains negative. **Substitution example:** For z = −2 and a = 0.01: output = −0.02; derivative on this side is 0.01. ### Formula 12: Stable softmax $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_j z_j $$ **Symbols:** z values are class scores; p values are probabilities. **Use:** Subtract the same maximum from every score, exponentiate, add, divide each by that total. **Substitution example:** For [0,0]: exponentials = [1,1]; total = 2; probabilities = [0.5,0.5]. ### Formula 13: Squared and absolute losses $$ \mathrm{SSE}=\sum_i(o_i-y_i)^2 $$ $$ L_{\mathrm{half}}=\frac{\mathrm{SSE}}2 $$ $$ L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n $$ $$ L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i| $$ **Symbols:** o is prediction; y is target; n is the number of averaged terms. **Use:** Read whether the question uses a half sum, a mean, or an unhalved sum. The scaling changes gradients. **Substitution example:** For errors [−1,2]: half sum = (1+4)/2 = 2.5; MSE = 2.5; MAE = (1+2)/2 = 1.5. The first two coincide only because n = 2. ### Formula 14: Binary cross entropy $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Symbols:** y is 0 or 1; p is the predicted probability for class 1; ln is the natural logarithm. **Use:** Use for a sigmoid binary prediction. If y = 1 use −ln(p); if y = 0 use −ln(1−p). **Substitution example:** For y = 1 and p = 0.8: loss = 0.223144. ### Formula 15: Categorical cross entropy $$ H(p,q)=-\sum_i p_i\ln q_i,\qquad\log_2q=\frac{\ln q}{\ln2} $$ For a soft target distribution, retain every nonzero target-weighted term. Natural logs give nats; base-2 logs give bits. $$ L=-\sum_i y_i\ln p_i=-\ln p_{true} $$ **Symbols:** The final equality assumes one-hot labels, with exactly one y equal to 1. **Use:** Use for mutually exclusive multiclass labels. Select the probability of the true class. **Substitution example:** True class 2 and probabilities [0.1,0.8,0.1]: loss = −ln(0.8) = 0.223144. ### Formula 16: Output delta for half squared loss $$ \delta^{out}=(o-y)f\prime(z) $$ **Symbols:** Delta is the loss derivative with respect to the preactivation z. **Use:** Use with half squared loss. For unhalved one-output MSE multiply by 2; for an MSE averaged over m outputs multiply the half-sum delta by 2/m. **Substitution example:** Sigmoid output 0.5 and target 1: delta = (−0.5)×0.25 = −0.125. ### Formula 17: Sigmoid plus BCE; softmax plus CCE $$ \delta^{out}=p-y $$ **Symbols:** p and y can be vectors for softmax. **Use:** This is the combined derivative with respect to the logits. Do not multiply by the activation derivative again. **Substitution example:** Binary p = 0.8, y = 1: delta = −0.2. ### Formula 18: Hidden delta, weight gradient and bias gradient $$ \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out},\quad\frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\quad\frac{\partial L}{\partial b_j}=\delta_j $$ **Symbols:** a\_i is the activation sending a signal along the weight; delta\_j belongs to the receiving neuron. **Use:** Use OLD downstream weights throughout one backward pass. Add all outgoing-path contributions. **Substitution example:** If receiving delta = −0.2 and sending activation = 3: weight gradient = −0.6; bias gradient = −0.2. ### Formula 19: Gradient descent $$ \theta_{new}=\theta-\eta g,\qquad g=\frac{\partial J}{\partial\theta} $$ **Symbols:** theta is the parameter; eta is learning rate; g is the current gradient. **Use:** Recalculate g at the current parameter unless the question supplies gradients directly. **Substitution example:** theta = 1, eta = 0.1, g = −2 gives 1−0.1(−2) = 1.2. ### Formula 20: Batch and mini-batch gradient $$ g_{batch}=\frac1B\sum_{i=1}^{B}g_i,\quad U_{epoch}=\left\lceil\frac NB\right\rceil $$ **Symbols:** B is batch size; N is sample count; U is updates per epoch when the final incomplete batch is retained. **Use:** SGD uses B = 1. Full batch uses B = N. Average only when the stated objective is a mean. **Substitution example:** N = 105, B = 20: six updates, with five samples in the last batch. ### Formula 21: Momentum: signed velocity $$ v_t=\beta v_{t-1}-\eta g_t,\qquad\theta_t=\theta_{t-1}+v_t $$ **Symbols:** v stores signed movement; beta retains part of old movement. **Use:** Use this convention consistently. Other textbooks may define an unsigned gradient accumulator and subtract it instead. **Substitution example:** beta = 0.8, old v = −0.2, eta = 0.1, g = 1.8 gives v = −0.34. ### Formula 22: Nesterov: look-ahead gradient $$ \theta_{look}=\theta+\beta v,\quad v_{new}=\beta v-\eta g(\theta_{look}),\quad\theta_{new}=\theta+v_{new} $$ **Symbols:** The look-ahead point is temporary; the update is added to the original theta. **Use:** Evaluate the gradient at the look-ahead point, not at the old theta. **Substitution example:** theta = 1.8, beta = 0.8 and v = −0.2 gives look-ahead = 1.64. ### Formula 23: AdaGrad $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Symbols:** G is accumulated squared gradient; epsilon prevents division by zero. **Use:** Accumulate every squared gradient, separately for each parameter. Use the epsilon placement stated by the problem. **Substitution example:** Old G = 4 and g = −3 gives G = 13. ### Formula 24: RMSProp $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Symbols:** rho is the decay; s is a decaying mean of squared gradients. **Use:** Calculate the new square memory first, then update the parameter. **Substitution example:** rho = 0.9, old s = 4, g = −3 gives s = 3.6+0.9 = 4.5. ### Formula 25: Adam: both memories $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ **Symbols:** m remembers signed gradients; v remembers squared gradients. **Use:** Start with the initial states specified. Common defaults are beta1 = 0.9 and beta2 = 0.999. ### Formula 26: Adam: correction and update $$ \widehat m_t=\frac{m_t}{1-\beta_1^t},\quad\widehat v_t=\frac{v_t}{1-\beta_2^t},\quad\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ **Symbols:** t counts updates starting at 1, not 0. **Use:** Correct both zero-initialized memories before the update; do not multiply by the gradient a second time. **Substitution example:** At t = 1, m = 0.2 and beta1 = 0.9: corrected m = 0.2/0.1 = 2. ### Formula 27: L1 penalty and update $$ J=L+\lambda\sum_j|w_j|,\quad w_{new}=w-\eta\left[g_{data}+\lambda\operatorname{sign}(w)\right] $$ **Symbols:** lambda is regularization strength; sign(w) is +1 for positive w and −1 for negative w. **Use:** At zero, L1 has a subgradient rather than a unique derivative; use the specified convention. Include only parameters named in the penalty. **Substitution example:** w = −2, lambda = 0.1, data gradient = 0 gives penalty gradient = −0.1. ### Formula 28: L2 penalty and update with a half $$ J=L+\frac\lambda2\sum_jw_j^2,\quad w_{new}=w-\eta(g_{data}+\lambda w) $$ **Symbols:** The half cancels the 2 produced by differentiating a square. **Use:** If the objective uses lambda times the sum of squares WITHOUT a half, use 2 lambda w instead. **Substitution example:** w = 2, lambda = 0.1, data gradient = 0.3 gives total gradient = 0.5. ### Formula 29: Inverted dropout $$ q=1-p,\quad a_{train}=\frac{m\odot a}{q},\quad a_{test}=a $$ **Symbols:** p is drop probability; q is keep probability; mask m contains zeros and ones; the circled dot means elementwise multiplication. **Use:** During training, zero dropped entries and divide kept entries by q. Disable dropout at inference. **Substitution example:** a = 3, q = 0.5, mask = 1 gives 3/0.5 = 6. ### Formula 30: Convolution output dimensions $$ N_{out}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Symbols:** N is input height or width; P is padding on each side; K is kernel size; S is stride; floor rounds down. **Use:** Apply separately to height and width. Output channels equal the number of filters. This formula assumes dilation 1. **Substitution example:** N = 7, P = 1, K = 3, S = 2 gives floor(6/2)+1 = 4. ### Formula 31: Convolution value $$ Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o $$ **Symbols:** c runs over input channels; u,v run over kernel positions; o selects one filter. **Use:** This is the usual deep-learning cross-correlation operation. Use zeros outside the padded image. Multiply matching cells, add all channels, then add the filter bias once. ### Formula 32: Max, min and average pooling Min pooling keeps the smallest entry in each window. For a four-entry window: $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{max}=\max_{(i,j)\in W}x_{ij},\quad y_{avg}=\frac1{|W|}\sum_{(i,j)\in W}x_{ij} $$ **Symbols:** W is one pooling window; its size is the number of included cells. **Use:** Move the window by the stated stride; process channels separately. For padding, follow the stated average-pooling denominator convention. **Substitution example:** For [1,4,2,3]: maximum = 4; average = 10/4 = 2.5. ### Formula 33: Convolution parameter count $$ P_{conv}=(K_hK_wC_{in}+1)C_{out} $$ **Symbols:** Kernel height and width times input channels gives weights per filter; +1 is one bias; Cout is filter count. **Use:** Use for a fully connected, ungrouped convolution with bias. Output height and width do not multiply the parameter count. **Substitution example:** 3×3 kernel, 3 input channels, 2 filters: (27+1)×2 = 56. ### Formula 34: Flatten and dense parameter count $$ N_{flat}=HWC,\qquad P_{dense}=(N_{in}+1)N_{out} $$ **Symbols:** H,W,C are activation dimensions; Nin and Nout are dense-layer input and output counts. **Use:** Flatten has no trainable parameters. Dense biases belong to receiving neurons. **Substitution example:** 2×2×2 flattens to 8; dense 3 then has (8+1)×3 = 27 parameters. ### Formula 35: Convolution multiply-accumulate count $$ MAC=H_{out}W_{out}K_hK_wC_{in}C_{out} $$ **Symbols:** Each output value uses kernel area times input-channel count multiplications. **Use:** Used for Inception bottleneck comparisons. This counts one multiply-accumulate as one MAC and ignores bias additions. Some FLOP conventions count multiply and add separately. ### Formula 36: Concatenation and residual addition $$ C_{concat}=\sum_b C_b,\qquad y_{res}=F(x)+x $$ **Symbols:** b labels branches with matching spatial dimensions. **Use:** Inception concatenates channels; residual addition requires matching full shapes. Use a projection when needed. **Substitution example:** Channels 16+32+8+8 give 64. Adding two 14×14×64 tensors keeps 14×14×64. ### Formula 37: Vanilla RNN $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad o_t=W_{hy}h_t+b_y $$ **Symbols:** x\_t is current input; h\_previous is previous state; the same matrices are reused at every time step. **Use:** Calculate the two matrix-vector products, add bias, then tanh. Apply the stated output activation to o\_t when required. ### Formula 38: Perceptron update $$ \widehat y=\operatorname{step}(w^Tx+b),\quad w_{new}=w+\eta(y-\widehat y)x,\quad b_{new}=b+\eta(y-\widehat y) $$ **Symbols:** The step threshold and label encoding must match the question. **Use:** Use for a stated perceptron learning rule; this is not the sigmoid derivative rule. ### Formula 39: BPTT hidden delta $$ \delta_t=\left[W_{hy}^T(p_t-y_t)+W_{hh}^T\delta_{t+1}\right]\odot(1-h_t^2) $$ **Symbols:** Here p\_t is the softmax probability vector. The output term assumes softmax plus cross entropy at each stated output step; omit it at steps with no direct loss. **Use:** Backpropagate from the last time step. Shared-weight gradients sum contributions across time. ### Formula 40: BPTT recurrent-weight gradient $$ g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right) $$ For gradient norm clipping, τ > 0 is the maximum norm. Leave a zero gradient unchanged. This is a remedy for exploding gradients; it is distinct from the shared-weight gradient formula below. $$ \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^T,\quad\frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^T $$ **Symbols:** These are outer products, not scalar dot products. **Use:** Sum each occurrence of a shared weight. Apply the learning-rate update after accumulating the required contributions. ### Formula 41: LSTM gates and candidate $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ **Symbols:** f is forget gate; i is input gate; [h,x] means concatenation. **Use:** Compute each gate from its own weights and bias. Gate values lie between 0 and 1. ### Formula 42: LSTM candidate and output gate $$ \widetilde C_t=\tanh(W_C[h_{t-1},x_t]+b_C),\quad o_t=\sigma(W_o[h_{t-1},x_t]+b_o) $$ **Symbols:** The candidate can be negative; the gates are sigmoid outputs. **Use:** Do not confuse the candidate value with its input gate. ### Formula 43: LSTM cell and hidden state $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t,\quad h_t=o_t\odot\tanh(C_t) $$ **Symbols:** C is the cell state; h is the exposed hidden state. **Use:** Calculate old retained part, new admitted part, sum them, apply tanh, then multiply by output gate. ### Formula 44: GRU gates, candidate and update $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z),\quad r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ **Symbols:** z is update gate; r is reset gate. **Use:** Use the exact gate and reset placement convention in the question. ### Formula 45: GRU state under the retain-old convention $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h),\quad h_t=z_t\odot h_{t-1}+(1-z_t)\odot\widetilde h_t $$ **Symbols:** In THIS convention, z weights the old state. Some sources reverse the two coefficients. **Use:** Read the stated update equation before substituting. For vector states, all gate products here are elementwise. ### Formula 46: Batch normalization and learned scale/shift Use during training to normalize intermediate activations before a learned scale and shift. m is the number of values; μ\_B and σ²\_B are the batch mean and population variance; γ and β are learned. Epsilon is inside the square root. Ordinary inference uses stored running statistics. The linked full lesson calculates every stage for [2,4]. $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ ### Formula 47: Global average pooling Take a separate average for each full spatial feature map. H and W count rows and columns; c selects one of C channels; a is an activation value. Output has C values. The linked lesson computes two channel averages and the following dense-layer parameters. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ ### Formula 48: Top-k error percentage N is the total number of evaluated examples. C\_k counts those whose true class is among the k highest-scoring predictions. Count failures, divide by total, then convert to percent. The worked lesson contrasts top-1 and top-5 using the same 100 images. $$ E_k=\frac{N-C_k}{N}\times100\% $$ ## Expanded formula guide and worked examples ## Mean, variance and standard deviation The mean is a measure of location; variance and standard deviation quantify dispersion. Distinguish population and sample conventions before calculating these statistics. $$ \mu=\frac{1}{n}\sum_i x_i $$ $$ v=\frac{1}{n}\sum_i(x_i-\mu)^2 $$ $$ s=\sqrt{v} $$ ## Mean, variance and standard deviation from ordinary arithmetic **Definition.** The arithmetic mean measures the centre of a numerical dataset. Variance measures the average squared deviation from that mean, and standard deviation expresses dispersion in the original units. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i,\qquad \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2,\qquad \sigma=\sqrt{\sigma^2} $$ These are population formulas, or empirical moments treating the supplied dataset as the complete reference set. An unbiased sample-variance estimate instead divides the squared-deviation sum by N−1. Use the convention specified by the question; the two denominators are not interchangeable. **Question. Find the population mean, variance and standard deviation of [3, 3, 7, 7].** ### Given Observations = [3,3,7,7]; N = 4. Use population variance (divide by N). ### To find Find the population mean μ, variance σ² and standard deviation σ. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Answer:** mean = 5, population variance = 4 and population standard deviation = 2. Variance has squared units; standard deviation has the same units as the observations. With the sample convention, the variance would instead be 16/3. ### Population mean and spread: [1, 2, 3] **Question. For the population [1, 2, 3], find mean, population variance and standard deviation.** ### Given Values = [1, 2, 3]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 1+2+3=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (1-2)=-1 $$ $$ (-1)^2=1 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (3-2)=1 $$ $$ (1)^2=1 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{1+0+1}{3}\approx0.666666667 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.666666667}\approx0.816496581 $$ **Final answer:** μ = 2; σ² ≈ 0.666666667; σ ≈ 0.816496581. ### Population mean and spread: [2, 2, 2] **Question. For the population [2, 2, 2], find mean, population variance and standard deviation.** ### Given Values = [2, 2, 2]; N = 3. Use population division by N. ### To find Mean μ, variance σ² and standard deviation σ. ### Formula — which one and why The question specifies a population, so use N rather than the N−1 sample-variance divisor. μ is the centre; σ² averages squared distances from it; σ converts spread back to the original units. $$ \mu=\frac1N\sum_i x_i $$ $$ \sigma^2=\frac1N\sum_i(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean, distance, square, mean again, then root. ### Solution — step by step **Step 1: add all values.** $$ 2+2+2=6 $$ **Step 2: divide by the number of values.** $$ \mu=\frac{6}{3}=2 $$ **Step 3: find and square deviation 1.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 4: find and square deviation 2.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 5: find and square deviation 3.** $$ (2-2)=0 $$ $$ (0)^2=0 $$ **Step 6: average the squared deviations.** $$ \sigma^2=\frac{0+0+0}{3}\approx0.000000000 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt{0.000000000}\approx0.000000000 $$ **Final answer:** μ = 2; σ² ≈ 0.000000000; σ ≈ 0.000000000. Standardization is undefined with this zero standard deviation. ## Standardization Standardization has a meaning before it has a formula: measure how far an input is from its training mean, then express that distance in standard-deviation units. The worked question below calculates the mean and standard deviation first, so neither number appears without explanation. $$ x_{scaled}=\frac{x-\mu_{train}}{s_{train}} $$ ## Standardization: what the answer 1 actually means **Definition.** Standardization is an affine transformation that expresses an observation as its signed distance from a reference mean, measured in reference standard-deviation units. $$ z=\frac{x-\mu_{\mathrm{train}}}{\sigma_{\mathrm{train}}},\qquad \sigma_{\mathrm{train}}>0 $$ Subtracting the mean centres the reference data. Dividing by its standard deviation rescales its spread. The result is dimensionless. A positive z lies above the training mean, a negative z below it, and z = 0 equals it. Standardization does not make an arbitrary distribution Gaussian and does not restrict values to [0,1]. **Question. A feature has training values [3, 3, 7, 7]. Using population standard deviation, standardize the training values and a new observation x = 9.** ### Given Training observations = [3,3,7,7]. New observation = 9. Estimate the population standard deviation from the training values. ### To find Find the training mean and standard deviation; standardize all four training values and the new observation. ### Formula — which one and why **Mean and spread.** The mean is the sum divided by the number of values. Population variance averages squared distances from that mean; standard deviation is its square root. N = number of values; xᵢ = value i; μ = mean; σ² = variance; σ = standard deviation. Σ means add the indicated terms. $$ \mu=\frac{1}{N}\sum_{i=1}^{N}x_i $$ $$ \sigma^2=\frac{1}{N}\sum_{i=1}^{N}(x_i-\mu)^2 $$ $$ \sigma=\sqrt{\sigma^2} $$ **Yaad rakhne ka cue:** Mean = sabka total ÷ count; variance = squared distances ka mean. **Standardization.** Subtract the training mean to measure distance from the centre; divide by the training standard deviation to express that distance in standard-deviation units. x = original value; μ = training mean; σ = training standard deviation, which must be nonzero; z = standardized value. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Pehle centre hatao, phir spread se divide karo. ### Solution — step by step **Step 1: add all four observations.** $$ 3+3+7+7=20 $$ **Step 2: divide by the count to obtain the mean.** $$ \mu=\frac{20}{4}=5 $$ **Step 3: subtract the mean from each observation.** $$ [3-5,3-5,7-5,7-5]=[-2,-2,2,2] $$ **Step 4: square each deviation.** $$ [(-2)^2,(-2)^2,2^2,2^2]=[4,4,4,4] $$ **Step 5: add the squared deviations.** $$ 4+4+4+4=16 $$ **Step 6: divide by four for population variance.** $$ \sigma^2=\frac{16}{4}=4 $$ **Step 7: take the nonnegative square root.** $$ \sigma=\sqrt4=2 $$ **Step 8: standardize each original training observation.** $$ \frac{3-5}{2}=-1 $$ $$ \frac{7-5}{2}=1 $$ The repeated observations give [-1, −1, 1, 1]. **Step 9: subtract the training mean from the new observation.** $$ 9-5=4 $$ **Step 10: divide by the training standard deviation.** $$ z_{\mathrm{new}}=\frac42=2 $$ **Answer:** the new observation is two training standard deviations above the training mean. Do not recompute the mean and standard deviation using the new observation; validation and test data use the training transformation to avoid leakage. A constant feature has zero standard deviation and needs a stated handling policy. ### Standardization with x = 3 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 3 and interpret the result.** ### Given x = 3; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=3-5=-2 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{-2}{2}=-1 $$ **Step 3: interpret sign and size.** The value is 1 standard-deviation unit(s) below the training mean. **Final answer:** z = -1. Use the same training statistics when transforming a new test input. ### Standardization with x = 5 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 5 and interpret the result.** ### Given x = 5; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=5-5=0 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{0}{2}=0 $$ **Step 3: interpret sign and size.** The value is 0 standard-deviation unit(s) away from the training mean. **Final answer:** z = 0. Use the same training statistics when transforming a new test input. ### Standardization with x = 9 **Question. Training mean is 5 and training standard deviation is 2. Standardize x = 9 and interpret the result.** ### Given x = 9; training mean μ = 5; training standard deviation σ = 2. ### To find Standardized value z and its meaning. ### Formula — which one and why Subtract the training centre, then measure that distance in units of the training standard deviation. A negative z means below the training mean; positive means above it; zero means equal to it. $$ z=\frac{x-\mu}{\sigma} $$ **Yaad rakhne ka cue:** Centre hatao, spread se divide; answer batata hai kitne σ door. ### Solution — step by step **Step 1: find distance from the mean.** $$ x-\mu=9-5=4 $$ **Step 2: divide by one standard-deviation unit.** $$ z=\frac{4}{2}=2 $$ **Step 3: interpret sign and size.** The value is 2 standard-deviation unit(s) above the training mean. **Final answer:** z = 2. Use the same training statistics when transforming a new test input. ## Power, constant and sum derivatives Derivatives describe local rates of change. Partial derivatives hold other independent variables fixed; their collection forms the gradient used for optimization. $$ \frac{d}{dx}(cx^n)=cnx^{n-1} $$ $$ \frac{d}{dx}c=0 $$ $$ (f+g)\prime=f\prime+g\prime $$ ## Derivatives without a calculus course **Definition.** A derivative measures the local rate of change of a function. A partial derivative varies one argument while holding the others fixed. The gradient of a scalar-valued function collects its partial derivatives into a vector. $$ \nabla_\theta J=\begin{bmatrix}\partial J/\partial\theta_1\\\vdots\\\partial J/\partial\theta_d\end{bmatrix} $$ The rules needed for the supplied neural-network numericals are: | Function or operation | Derivative | | --- | --- | | Constant c | 0 | | xⁿ | nxⁿ⁻¹ | | c f(x) | c f′(x) | | f(x) + g(x) | f′(x) + g′(x) | | f(x)g(x) | f′(x)g(x) + f(x)g′(x) | | eˣ | eˣ | | ln x, x > 0 | 1/x | | f(g(x)) | f′(g(x))g′(x) | ### A parameter derivative is not an input derivative For z = wx + b, the variable named in the denominator determines which derivative is required. $$ \frac{\partial z}{\partial w}=x,\qquad\frac{\partial z}{\partial x}=w,\qquad\frac{\partial z}{\partial b}=1 $$ At x = 2, w = 3 and b = 1, z = 7, but the three derivatives are 2, 3 and 1 respectively. A function value and its derivative are distinct quantities. ### Gradient direction For a small parameter displacement Δθ, first-order Taylor expansion gives: $$ J(\theta+\Delta\theta)\approx J(\theta)+\nabla J(\theta)^T\Delta\theta $$ The gradient points in the direction of greatest local increase under the Euclidean norm. Choosing Δθ opposite to the gradient gives a local descent direction. This is the mathematical basis of gradient descent; a finite step can still overshoot if the learning rate is too large. ### Differentiate before substituting: 3w^2+5 **Question. Let J(w) = 3w^2+5. At w = 2, find J and dJ/dw.** ### Given J(w) = 3w^2+5; w = 2. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(2)=3(2)^2+5=17 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=3(2)w^{2-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(2)=3(2)(2)^{1}=12 $$ **Final answer:** J = 17; derivative = 12. ### Differentiate before substituting: 0.7w^3+0 **Question. Let J(w) = 0.7w^3+0. At w = 1, find J and dJ/dw.** ### Given J(w) = 0.7w^3+0; w = 1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(1)=0.7(1)^3+0=0.7 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=0.7(3)w^{3-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(1)=0.7(3)(1)^{2}=2.1 $$ **Final answer:** J = 0.7; derivative = 2.1. ### Differentiate before substituting: 2w^1+4 **Question. Let J(w) = 2w^1+4. At w = -1, find J and dJ/dw.** ### Given J(w) = 2w^1+4; w = -1. ### To find The function value and its derivative value. ### Formula — which one and why The power rule brings the exponent down as a multiplier and reduces its power by one. An added constant has derivative zero. c is the coefficient; n is the power; k is an added constant. J and dJ/dw are different quantities. $$ \frac{d(cw^n+k)}{dw}=cnw^{n-1} $$ **Yaad rakhne ka cue:** Power neeche multiply; exponent ek kam; constant ka zero. ### Solution — step by step **Step 1: evaluate the function itself.** $$ J(-1)=2(-1)^1+4=2 $$ **Step 2: differentiate symbolically.** $$ J\prime(w)=2(1)w^{1-1}+0 $$ **Step 3: put the given w into the derivative.** $$ J\prime(-1)=2(1)(-1)^{0}=2 $$ **Final answer:** J = 2; derivative = 2. ## Chain rule The chain rule differentiates a composite function. Multiply local derivatives along a dependency path and add the contributions of different paths to the same variable. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ ## The chain rule is multiplying linked effects **Definition.** The chain rule finds a rate of change when one quantity depends on another through intermediate calculations. Multiply the local rates along one path. If several paths connect the same starting quantity to the final answer, add their complete contributions. ### First understand why the rates multiply Suppose w controls u and u controls v. If each one-unit change in w makes u change by 3 units, and each one-unit change in u makes v change by 2 units, the combined rate is 3 × 2 = 6. Adding 3+2 would describe neither dependency. The notation du/dw reads “the rate at which u changes when w changes”. A partial derivative ∂L/∂w means the same kind of rate while other independent parameters are held fixed. It is not ordinary division of the values L and w. **Yaad rakhne ka cue:** ek raaste par slopes multiply; alag raaston ke full answers add. ### Chain-rule example 1: two linear links **Question. Let u = 3w+1 and v = 2u. At w = 1, find u, v and dv/dw.** ### Given w = 1; u = 3w+1; v = 2u. ### To find The two function values and the rate dv/dw. ### Formula — which one and why v depends on w through u. Both links must be included, so multiply the two local derivatives. du/dw is the slope of 3w+1. dv/du is the slope of 2u. $$ \frac{dv}{dw}=\frac{dv}{du}\frac{du}{dw} $$ **Yaad rakhne ka cue:** Andar ka slope × bahar ka slope. ### Solution — step by step **Step 1: calculate the intermediate value.** $$ u=3(1)+1=4 $$ **Step 2: calculate the final value.** $$ v=2(4)=8 $$ **Step 3: differentiate the first link.** $$ \frac{du}{dw}=3 $$ **Step 4: differentiate the second link.** $$ \frac{dv}{du}=2 $$ **Step 5: multiply along the path.** $$ \frac{dv}{dw}=2(3)=6 $$ **Final answer:** u = 4, v = 8 and dv/dw = 6. The function value 8 and its rate 6 are different. ### 2. One output: calculate both parameter gradients **Question.** A linear neuron has z = wx + b and half squared loss L = ½(z − y)². Given x = 2, y = 4, w = 1 and b = 0, obtain both parameter gradients and perform one gradient-descent update with η = 0.1. ### Given Input x = 2; target y = 4. Old weight w = 1; old bias b = 0. Linear prediction z = wx+b; loss L = ½(z−y)². Learning rate η = 0.1; update both parameters once. ### To find Find the loss gradient for w and for b, then w\_new and b\_new. Finally evaluate the changed prediction and loss. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ![The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.](https://deep-learning-study-room.vercel.app/assets/figures/chain_linear_revision.svg) Diagram: The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient.. Labels: One output: values and derivatives | One output: values and derivatives | Given: x = 2, y = 4, w = 1, b = 0 | FORWARD VALUES | z = wx + b | z = 2 | e = z − y | e = −2 | L = ½e² | L = 2 | BACKWARD DERIVATIVES | Loss derivative reaches the neuron | ∂L/∂z = (∂L/∂e)(∂e/∂z) = (−2)(1) = −2 | Weight: multiply by x = 2 | ∂L/∂w = (−2)(2) = −4 | Bias: multiply by 1 | ∂L/∂b = (−2)(1) = −2 | Then update: w → 1.4 and b → 0.2 (η = 0.1). The forward values are z = 2, error = −2 and loss = 2. The backward loss derivative is −2; multiply it by x = 2 for the weight gradient and by 1 for the bias gradient. Tap the diagram to enlarge. **Step 1 — evaluate the original model.** Use the given parameters before applying any update. $$ z=wx+b=1(2)+0=2 $$ $$ e=z-y=2-4=-2 $$ $$ L=\frac12e^2=\frac12(-2)^2=2 $$ Forward pass: three distinct quantities Prediction z**2**1 × 2 + 0 → Error e**−2**Prediction 2 − target 4 → Half squared loss**2**½ × (−2)² The prediction is 2, the signed error is −2 and the loss is 2. These are three different quantities, even when some numerical values happen to coincide. **Step 2 — differentiate the loss, keeping the target constant.** First differentiate the square, then the error inside it. $$ \frac{\partial L}{\partial e}=\frac12(2e)=e $$ $$ \frac{\partial e}{\partial z}=\frac{\partial(z-y)}{\partial z}=1 $$ $$ \delta_z=\frac{\partial L}{\partial z}=e(1)=-2 $$ The symbol δ (delta) is a convenient name for the derivative with respect to a neuron's preactivation. Delta is not automatically the weight gradient; the weight gradient also depends on the input to that connection. **Step 3 — differentiate the neuron's affine expression.** When differentiating with respect to w, x is a constant coefficient. When differentiating with respect to b, the coefficient of b is 1. $$ \frac{\partial z}{\partial w}=x=2 $$ $$ \frac{\partial z}{\partial b}=1 $$ **Step 4 — multiply along each parameter's path.** $$ g_w=\frac{\partial L}{\partial w}=\delta_z\frac{\partial z}{\partial w}=(-2)(2)=-4 $$ $$ g_b=\frac{\partial L}{\partial b}=\delta_z\frac{\partial z}{\partial b}=(-2)(1)=-2 $$ Backward pass: a separate derivative for each parameter Shared loss derivative δz**−2**Both parameter paths start here. Weight path: multiply by x = 2**−4**∂L/∂w = (−2) × 2 Bias path: multiply by 1**−2**∂L/∂b = (−2) × 1 Both derivatives use the original forward-pass values. The weight and bias are different parameters, so keep their gradients separate. **Step 5 — update both parameters simultaneously.** Gradient descent subtracts the learning rate times the corresponding gradient. $$ w_{\mathrm{new}}=w-\eta g_w=1-0.1(-4)=1.4 $$ $$ b_{\mathrm{new}}=b-\eta g_b=0-0.1(-2)=0.2 $$ One simultaneous parameter update Weight**1 → 1.4**Subtract 0.1 × (−4) Bias**0 → 0.2**Subtract 0.1 × (−2) Both updates use gradients from the same original forward pass. Re-evaluate the model afterward. Subtracting a negative gradient increases the parameter. Do not recompute the bias gradient using the already changed weight; that would be a different update procedure. **Step 6 — perform a new forward pass to evaluate the update.** $$ z_{\mathrm{new}}=1.4(2)+0.2=3 $$ $$ L_{\mathrm{new}}=\frac12(3-4)^2=0.5 $$ **Final answer:** weight gradient −4, bias gradient −2; updated weight 1.4 and bias 0.2; new prediction 3 and loss 0.5. If a different question fixes the bias at zero, update only the weight; its new prediction would be 2.8, not 3. ### 3. Two outputs: why backward contributions are added A shared intermediate value can influence more than one output. The derivative with respect to that shared value must include every output path. Adding paths happens before passing the accumulated derivative further backward. **Question.** A shared linear hidden value is h = wx + b. Two linear outputs are o₁ = 2h and o₂ = −h; their targets are y₁ = 1 and y₂ = 0. The total loss is L = ½(o₁ − y₁)² + ½(o₂ − y₂)². At x = 2, w = 0.5 and b = 0.1, calculate the loss and the gradients with respect to h, w and b. Treat the output multipliers 2 and −1 as fixed. ### Given h = wx+b; outputs o₁ = 2h and o₂ = −h. x = 2; w = 0.5; b = 0.1; targets = [1,0]. Loss = half the sum of the two squared output errors. Output multipliers 2 and −1 stay fixed. ### To find Find total loss and its derivatives with respect to h, w and b. ### Formula — which one and why **Chain rule with more than one path.** The shared hidden value affects each output. Multiply derivatives within each path, then add the complete path contributions before continuing to w and b. oⱼ = output j; yⱼ = its target; h = shared hidden value; ∂oⱼ/∂h is the multiplier on that output path. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial h}=\sum_j(o_j-y_j)\frac{\partial o_j}{\partial h} $$ $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h} $$ **Yaad rakhne ka cue:** Har raaste ka effect pehle multiply, junction par sab add. ### Solution — step by step ![The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.](https://deep-learning-study-room.vercel.app/assets/figures/chain_branch_revision.svg) Diagram: The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias.. Labels: Two outputs: add complete path contributions | Two outputs: add complete path contributions | h = wx + b = 0.5(2) + 0.1 = 1.1 | Shared hidden value | h = 1.1 | o₁ = 2h; target y₁ = 1 | o₁ = 2.2 | o₂ = −h; target y₂ = 0 | o₂ = −1.1 | × 2 | × (−1) | Output derivative: 2.2 − 1 = 1.2 | Output derivative: −1.1 − 0 = −1.1 | BACK TO h | Path 1: (1.2)(2) = 2.4 | Path 2: (−1.1)(−1) = 1.1 | Add at the shared value, then continue backward | ∂L/∂h = 2.4 + 1.1 = 3.5 | ∂L/∂w = 3.5 × 2 = 7 | ∂L/∂b = 3.5 × 1 = 3.5 The shared hidden value affects both losses. The first path contributes 2.4 and the second contributes 1.1, giving a combined derivative of 3.5 before continuing to the weight and bias. Tap the diagram to enlarge. **Step 1 — forward values.** $$ h=0.5(2)+0.1=1.1 $$ $$ o_1=2(1.1)=2.2,\qquad o_2=-(1.1)=-1.1 $$ $$ L_1=\frac12(2.2-1)^2=0.72 $$ $$ L_2=\frac12(-1.1-0)^2=0.605 $$ $$ L=L_1+L_2=0.72+0.605=1.325 $$ **Step 2 — output derivatives.** Each half squared loss differentiates to prediction minus target. The other loss does not depend on this output. $$ \frac{\partial L}{\partial o_1}=o_1-y_1=2.2-1=1.2 $$ $$ \frac{\partial L}{\partial o_2}=o_2-y_2=-1.1-0=-1.1 $$ **Step 3 — calculate the contribution of each complete path.** The derivative of 2h with respect to h is 2; the derivative of −h is −1. $$ \frac{\partial L_1}{\partial h}=(1.2)(2)=2.4 $$ $$ \frac{\partial L_2}{\partial h}=(-1.1)(-1)=1.1 $$ **Step 4 — add the two contributions at the shared hidden value.** $$ \frac{\partial L}{\partial h}=\frac{\partial L_1}{\partial h}+\frac{\partial L_2}{\partial h} $$ $$ \frac{\partial L}{\partial h}=2.4+1.1=3.5 $$ Adding the raw output errors, 1.2 + (−1.1), would omit the output multipliers. Each error must first be multiplied by the local derivative on its own path. **Step 5 — continue to the hidden weight and bias.** Since h = wx + b, its derivatives with respect to w and b are x and 1. $$ \frac{\partial L}{\partial w}=\frac{\partial L}{\partial h}x=3.5(2)=7 $$ $$ \frac{\partial L}{\partial b}=\frac{\partial L}{\partial h}(1)=3.5 $$ **Final answer:** L = 1.325, ∂L/∂h = 3.5, ∂L/∂w = 7 and ∂L/∂b = 3.5. If the hidden value instead includes a nonlinear activation, multiply the accumulated downstream derivative by that activation's derivative before multiplying by x. This is the hidden-delta calculation in multilayer backpropagation. ### Chain-rule example 4: a square after a linear link **Question. Let z = 3w+1 and L = z². At w = 1, find z, L and dL/dw.** ### Given w = 1; z = 3w+1; L = z². ### To find z, L and the derivative of L with respect to w. ### Formula — which one and why The square acts on z, not directly on w. Differentiate the square first and include the change of z with w. dL/dz = slope of the outer square; dz/dw = slope of the inner linear expression. $$ \frac{dL}{dw}=\frac{dL}{dz}\frac{dz}{dw} $$ $$ \frac{d(z^2)}{dz}=2z $$ **Yaad rakhne ka cue:** Square ka 2z; andar ka 3; dono multiply. ### Solution — step by step **Step 1: evaluate the inner expression.** $$ z=3(1)+1=4 $$ **Step 2: evaluate the loss expression.** $$ L=4^2=16 $$ **Step 3: differentiate the square and use z = 4.** $$ \frac{dL}{dz}=2z=2(4)=8 $$ **Step 4: differentiate the inner expression.** $$ \frac{dz}{dw}=3 $$ **Step 5: multiply, because the changes occur in sequence.** $$ \frac{dL}{dw}=8(3)=24 $$ **Final answer:** z = 4, L = 16 and dL/dw = 24. The combined operation is multiplication, not 8+3. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. ## Dot product and a neuron A neuron starts with matched input–weight pairs. Multiply inside each pair, add the products, add the bias once, and then apply the activation. The summation sign is simply a shorter way to write those additions. $$ z=\sum_i w_i x_i+b $$ $$ a=f(z) $$ ## Model answer: explain an artificial neuron ## Question **Question.** Explain the structure and working of an artificial neuron. Describe the roles of inputs, weights, bias and activation with a numerical example. ## Answer — step by step An artificial neuron is a computational unit that applies an activation function to an affine combination of its inputs. Its trainable weights and bias determine the preactivation; its output is passed to subsequent layers. ![Follow each input through its weight, add one bias at the receiving neuron, then apply activation.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Follow each input through its weight, add one bias at the receiving neuron, then apply activation.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Follow each input through its weight, add one bias at the receiving neuron, then apply activation. Tap the diagram to enlarge. ### Formula and symbols $$ z=w_1x_1+w_2x_2+\cdots+w_nx_n+b,\qquad a=f(z) $$ | Part | Meaning | What it does | | --- | --- | --- | | x₁, x₂, … | Input features | Supply the information for this calculation | | w₁, w₂, … | Weights | Control the contribution of each matching input | | b | Bias | Shifts the weighted sum independently of the inputs | | z | Net input | The number before activation | | f | Activation rule | Transforms z, often nonlinearly | | a | Activated output | The number sent onward | ### Worked ReLU neuron example **Question. An artificial neuron has x₁ = 2, x₂ = 1, w₁ = 0.5, w₂ = −0.2 and b = 0.1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2,1]; weights = [0.5,−0.2]; bias = 0.1; activation = ReLU. ### To find Preactivation z and activated output a. ### Formula — which one and why Each input contributes through its own weight. Add one bias after summing the products; ReLU then keeps a positive total. z means before activation; a means after activation. $$ z=w_1x_1+w_2x_2+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Product, product, sum, bias, phir activation. ### Solution — step by step **Step 1: multiply the first pair.** $$ 0.5(2)=1 $$ **Step 2: multiply the second pair.** $$ (-0.2)(1)=-0.2 $$ **Step 3: add both contributions.** $$ 1+(-0.2)=0.8 $$ **Step 4: add the bias once.** $$ z=0.8+0.1=0.9 $$ **Step 5: apply ReLU.** $$ a=\max(0,0.9)=0.9 $$ **Final answer:** z = 0.9 and a = 0.9. ReLU leaves this positive preactivation unchanged. The weights and bias are learned parameters. During training, their values are adjusted to reduce an objective. The activation function itself is usually chosen as part of the architecture. Adding a bias does not mean adding one bias after every input product; add the receiving neuron's bias once. ## What one artificial neuron does **Definition.** An artificial neuron is a calculation unit that multiplies each input by a matching weight, adds the products and a bias, and applies an activation function to produce its output. ### What goes into a neuron? An input is a number supplied to the calculation. With three inputs, x = [2,1,3] simply lists those three numbers in order. The matching weight list tells the neuron which multiplier to use for each input. A negative weight contributes a negative amount when its input is positive. The **bias** is one extra adjustable number added to the total. The total before activation is called z or **preactivation**. The **activation function** is the final rule applied to z: for example, ReLU replaces a negative z with zero, while sigmoid turns z into a number between 0 and 1. The resulting output is called a or the **activation**. **Yaad rakhne ka cue:** inputs × apne weights → products ka sum → ek bias → activation → output. ### Mathematical model and notation $$ z=\sum_{i=1}^{d}w_ix_i+b=w^Tx+b $$ $$ a=f(z) $$ | Symbol | Meaning and role | | --- | --- | | xᵢ | Input component i; the neuron receives d such components | | wᵢ | Trainable weight multiplying that input component | | b | Trainable bias added once to this neuron's total | | z | Preactivation: the result before applying the activation function | | f | Chosen activation function, such as sigmoid or ReLU | | a | Activation: the neuron's output after applying f | The dot product wᵀx is a compact notation for the sum of matching weight–input products. A positive weight and a negative weight can make opposite contributions for the same positive input. A larger weight magnitude means a larger contribution for a fixed input; it is not, by itself, a general measure of feature importance across differently scaled inputs. ### Why both a bias and an activation are needed The bias changes the preactivation independently of x. In a one-input affine function z = wx + b, it is the intercept. In a classifier with a decision boundary z = 0, a bias allows that boundary to be shifted instead of forcing it through the origin. The activation determines how z is transformed. Sigmoid produces a value between 0 and 1; ReLU retains positive preactivations and sets negative ones to zero. Nonlinear activations let stacked layers represent nonlinear functions. The preactivation and activation are distinct quantities even when a particular ReLU calculation gives them equal values. ![Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron.. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate Read the neuron in order: input contributions → affine sum z → activation function f → output a. Add one bias at the receiving neuron. Tap the diagram to enlarge. ### Worked numerical **Question.** A neuron has x = [2, 1, 3]ᵀ, w = [0.4, −0.2, 0.5]ᵀ and b = 0.1. Calculate its preactivation and sigmoid output. ### Given Inputs = [2,1,3]; matching weights = [0.4,−0.2,0.5]. Bias = 0.1; activation = sigmoid. ### To find Find the preactivation z, then the sigmoid output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid output.** The question specifies sigmoid, so turn the complete weighted sum into an output between 0 and 1. Negate z, evaluate the exponential, add 1, then take the reciprocal. z = sum before activation; a = sigmoid output; e ≈ 2.71828 is the exponential base. $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Minus-z ki e power, plus one, phir one divided by whole denominator. ### Solution — step by step **Step 1: multiply input 1 by weight 1.** $$ x_1w_1=2(0.4)=0.8 $$ **Step 2: multiply input 2 by weight 2.** $$ x_2w_2=1(-0.2)=-0.2 $$ The minus sign belongs to the weight, so the contribution stays negative. **Step 3: multiply input 3 by weight 3.** $$ x_3w_3=3(0.5)=1.5 $$ **Step 4: add the three contributions.** $$ s=0.8+(-0.2)+1.5=0.6+1.5=2.1 $$ **Step 5: add the bias once.** $$ z=s+b=2.1+0.1=2.2 $$ This is the preactivation. It is not yet the requested sigmoid output. **Step 6: negate z for the exponential.** $$ -z=-2.2 $$ $$ e^{-z}=e^{-2.2}\approx0.110803158 $$ **Step 7: calculate the complete denominator.** $$ 1+e^{-z}=1+0.110803158=1.110803158 $$ **Step 8: divide one by that denominator.** $$ a=\frac1{1+e^{-z}}=\frac1{1.110803158}\approx0.900249511 $$ **Final answer:** sigmoid output ≈ 0.900249511. It lies between 0 and 1. The raw sum was 2.2, whereas the activated output is approximately 0.90025. ### From one neuron to a layer With m receiving neurons and d input components, W has shape m × d. Row j contains the incoming weights of neuron j. The bias vector has m entries, one for each receiving neuron: $$ z=Wx+b,\qquad a=f(z) $$ Sigmoid, tanh and ReLU apply independently to each component. Softmax is different: it transforms a whole vector of output logits together. ### Neuron example 2: ReLU output **Question. A neuron receives inputs [2], weights [3] and bias 1. Calculate its preactivation and ReLU output.** ### Given Inputs = [2]. Weights = [3]; bias = 1. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(3)=6 $$ **Step 2: add the input contributions.** $$ s=(6)=6 $$ **Step 3: add the bias once.** $$ z=6+(1)=7 $$ **Step 4: apply the stated activation.** $$ a=\max(0,7)=7 $$ **Final answer:** z = 7; a = 7. ### Neuron example 3: ReLU output **Question. A neuron receives inputs [2, 1], weights [-1, 0.5] and bias 0.2. Calculate its preactivation and ReLU output.** ### Given Inputs = [2, 1]. Weights = [-1, 0.5]; bias = 0.2. Activation = ReLU. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (2)(-1)=-2 $$ **Step 2: multiply input 2 by its own weight.** $$ (1)(0.5)=0.5 $$ **Step 3: add the input contributions.** $$ s=(-2)+(0.5)=-1.5 $$ **Step 4: add the bias once.** $$ z=-1.5+(0.2)=-1.3 $$ **Step 5: apply the stated activation.** $$ a=\max(0,-1.3)=0 $$ **Final answer:** z = -1.3; a = 0. ### Neuron example 4: sigmoid output **Question. A neuron receives inputs [1, 2], weights [0.2, -0.1] and bias 0. Calculate its preactivation and sigmoid output.** ### Given Inputs = [1, 2]. Weights = [0.2, -0.1]; bias = 0. Activation = sigmoid. ### To find z before activation and a after activation. ### Formula — which one and why The weighted sum combines the inputs. The output must then use the activation explicitly requested. z names the sum; a names the activated result. They are not always equal. $$ z=\sum_i w_ix_i+b $$ $$ a=\frac1{1+e^{-z}} $$ **Yaad rakhne ka cue:** Sum alag, activation alag; activation lagana mat bhool. ### Solution — step by step **Step 1: multiply input 1 by its own weight.** $$ (1)(0.2)=0.2 $$ **Step 2: multiply input 2 by its own weight.** $$ (2)(-0.1)=-0.2 $$ **Step 3: add the input contributions.** $$ s=(0.2)+(-0.2)=0 $$ **Step 4: add the bias once.** $$ z=0+(0)=0 $$ **Step 5: apply the stated activation.** $$ a=\frac1{1+e^{0}}=\frac1{1+1}=0.5 $$ **Final answer:** z = 0; a = 0.5. ## Drill 4 Dot product and bias ## Question **Question.** Find z for x = [2,−1,0.5], w = [0.3,0.4,−0.2], b = 0.1; then apply ReLU. ## Answer — step by step ### Given Inputs x = [2, −1, 0.5]. Weights w = [0.3, 0.4, −0.2]; bias b = 0.1. Activation: ReLU. ### To find Find the weighted sum z and ReLU output. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas A dot product means multiply corresponding entries, then add the products. A bias is one extra value added after the sum. ReLU keeps a positive result and replaces a negative result with zero. $$ z=w_1x_1+w_2x_2+w_3x_3+b,\qquad a=\max(0,z) $$ Here x contains inputs, w contains weights, b is bias, z is the weighted sum, and a is the activated output. ### Step 1: multiply the matching pairs $$ w_1x_1=0.3\times2=0.6 $$ $$ w_2x_2=0.4\times(-1)=-0.4 $$ $$ w_3x_3=(-0.2)\times0.5=-0.1 $$ ### Step 2: add all products and add bias once $$ z=0.6+(-0.4)+(-0.1)+0.1 $$ $$ z=0.2-0.1+0.1=0.2 $$ ### Step 3: apply the activation $$ a=\max(0,0.2)=0.2 $$ **Final answer:** weighted sum z = 0.2; ReLU output a = 0.2. The two numbers happen to match because z is positive. ## Matrix multiplication Matrix multiplication implements affine transformations in a dense layer. Dimensions determine which products are valid; each output component is the dot product of a weight row and the input vector, plus its bias. $$ (AB)_{ij}=\sum_k A_{ik}B_{kj} $$ ## Matrix multiplication: every row has a job ### Row, column, vector and matrix — read these first A **row** runs horizontally, from left to right. A **column** runs vertically, from top to bottom. A **matrix** is a rectangular arrangement of numbers in rows and columns. Its size is written rows × columns. A 2 × 3 matrix therefore has two rows and three columns, not six rows. A **vector** is an ordered list of numbers. Written horizontally it is a row vector, such as [2, 1, 3]. Written vertically it is a column vector. Neural-network equations in these notes usually use column vectors. ### Transpose: rows become columns The superscript T means **transpose**, not a power. To obtain Aᵀ, copy row 1 of A as column 1, row 2 as column 2, and so on. Do not reverse the order within a row, change a sign or multiply anything. An m × n matrix becomes n × m. $$ [A^T]_{ij}=A_{ji} $$ The two indices are positions: i identifies a row and j a column. The rule says “swap the row and column addresses”. For example, the value at row 1, column 3 moves to row 3, column 1. **Yaad rakhne ka cue:** transpose = row ko column bana; number aur sign ko haath mat laga. ### Transpose example 1 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}2&1&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 1 row(s) and 3 column(s), so its size is 1 × 3. **Step 2: move original row 1 into new column 1.** Copy [2, 1, 3] down column 1, in the same order. **Step 3: write the transposed matrix.** $$ A^T=\begin{bmatrix}2\\1\\3\end{bmatrix} $$ **Step 4: state its new size.** The result has 3 row(s) and 1 column(s): 3 × 1. **Final answer:** Aᵀ has size 3 × 1. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 1 rows × 3 columns A: 1 rows × 3 columns | | | | | --- | --- | --- | | 2 | 1 | 3 | Read each colored row left to right. Aᵀ: 3 rows × 1 columns Aᵀ: 3 rows × 1 columns | | | --- | | 2 | | 1 | | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 2 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&2&3\\4&5&6\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 3 column(s), so its size is 2 × 3. **Step 2: move original row 1 into new column 1.** Copy [1, 2, 3] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4, 5, 6] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&4\\2&5\\3&6\end{bmatrix} $$ **Step 5: state its new size.** The result has 3 row(s) and 2 column(s): 3 × 2. **Final answer:** Aᵀ has size 3 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 3 columns A: 2 rows × 3 columns | | | | | --- | --- | --- | | 1 | 2 | 3 | | 4 | 5 | 6 | Read each colored row left to right. Aᵀ: 3 rows × 2 columns Aᵀ: 3 rows × 2 columns | | | | --- | --- | | 1 | 4 | | 2 | 5 | | 3 | 6 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 3 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}1&-2\\0&3\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 2 row(s) and 2 column(s), so its size is 2 × 2. **Step 2: move original row 1 into new column 1.** Copy [1, -2] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [0, 3] down column 2, in the same order. **Step 4: write the transposed matrix.** $$ A^T=\begin{bmatrix}1&0\\-2&3\end{bmatrix} $$ **Step 5: state its new size.** The result has 2 row(s) and 2 column(s): 2 × 2. **Final answer:** Aᵀ has size 2 × 2. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 2 rows × 2 columns A: 2 rows × 2 columns | | | | --- | --- | | 1 | −2 | | 0 | 3 | Read each colored row left to right. Aᵀ: 2 rows × 2 columns Aᵀ: 2 rows × 2 columns | | | | --- | --- | | 1 | 0 | | −2 | 3 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Transpose example 4 **Question. Find the transpose of the following matrix A and state the size of both matrices.** ### Given $$ A=\begin{bmatrix}-1\\4\\2\end{bmatrix} $$ ### To find Aᵀ and both matrix sizes. ### Formula — which one and why A transpose exchanges the row and column positions of every entry. It involves rearrangement, not multiplication. A is the original matrix; Aᵀ is its transpose; i and j are row/column positions. $$ [A^T]_{ij}=A_{ji} $$ **Yaad rakhne ka cue:** Row seedha column mein; double transpose se original wapas. ### Solution — step by step **Step 1: write the original size.** A has 3 row(s) and 1 column(s), so its size is 3 × 1. **Step 2: move original row 1 into new column 1.** Copy [-1] down column 1, in the same order. **Step 3: move original row 2 into new column 2.** Copy [4] down column 2, in the same order. **Step 4: move original row 3 into new column 3.** Copy [2] down column 3, in the same order. **Step 5: write the transposed matrix.** $$ A^T=\begin{bmatrix}-1&4&2\end{bmatrix} $$ **Step 6: state its new size.** The result has 1 row(s) and 3 column(s): 1 × 3. **Final answer:** Aᵀ has size 1 × 3. Transposing it again gives the original A. Transpose: follow the same numbers from a row into a column A: 3 rows × 1 columns A: 3 rows × 1 columns | | | --- | | −1 | | 4 | | 2 | Read each colored row left to right. Aᵀ: 1 rows × 3 columns Aᵀ: 1 rows × 3 columns | | | | | --- | --- | --- | | −1 | 4 | 2 | Each original row is now the matching colored column. The entry at row i, column j moves to row j, column i. Its value and sign do not change. ### Dot product: multiply matching entries, then add A **dot product** turns two equally long lists into one number. Match first with first, second with second, and so on. It is the weighted sum inside a neuron. If both lists are columns, wᵀ turns the weight column into a row so row × column is allowed. $$ w^Tx=w_1x_1+w_2x_2+w_3x_3 $$ This formula is the three-input case. With two inputs there are only two products; with d inputs there are d products. A bias is an additional term, added after these products. ### Dot-product example 1 **Question. For w = [2, 3] and x = [4, 1], calculate wᵀx.** ### Given Weights w = [2, 3]. Matching inputs x = [4, 1]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (2)(4)=8 $$ **Step 2: multiply matching pair 2.** $$ (3)(1)=3 $$ **Step 3: add the products.** $$ w^Tx=(8)+(3)=11 $$ **Final answer:** wᵀx = 11. ### Dot-product example 2 **Question. For w = [1, -2] and x = [2, 3], calculate wᵀx.** ### Given Weights w = [1, -2]. Matching inputs x = [2, 3]. Both lists contain 2 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (1)(2)=2 $$ **Step 2: multiply matching pair 2.** $$ (-2)(3)=-6 $$ **Step 3: add the products.** $$ w^Tx=(2)+(-6)=-4 $$ **Final answer:** wᵀx = -4. ### Dot-product example 3 **Question. For w = [0.5, 1, -1] and x = [4, 2, 3], calculate wᵀx.** ### Given Weights w = [0.5, 1, -1]. Matching inputs x = [4, 2, 3]. Both lists contain 3 numbers. ### To find One scalar value wᵀx. ### Formula — which one and why A dot product is a sum of matching products. There is no bias in this question. Σ means add; the index i visits each matching pair. $$ w^Tx=\sum_i w_ix_i $$ **Yaad rakhne ka cue:** Same position multiply; sab products add. ### Solution — step by step **Step 1: multiply matching pair 1.** $$ (0.5)(4)=2 $$ **Step 2: multiply matching pair 2.** $$ (1)(2)=2 $$ **Step 3: multiply matching pair 3.** $$ (-1)(3)=-3 $$ **Step 4: add the products.** $$ w^Tx=(2)+(2)+(-3)=1 $$ **Final answer:** wᵀx = 1. ### Matrix multiplication: each row makes one output To multiply A of size m × n by B of size n × p, the inner sizes n must match. The result has size m × p. Output entry (i,j) is row i of A dotted with column j of B. $$ C_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj} $$ For a layer receiving two inputs and producing two neuron outputs, W is 2 × 2, x is 2 × 1 and Wx is 2 × 1. Each row of W holds all weights entering one receiving neuron. Its matching bias is added once. **Yaad rakhne ka cue:** multiplication mein andar ke sizes match; answer ka size bahar ke sizes. Har output = ek row × ek column. **Definition.** A vector is an ordered collection of scalars; a matrix represents a linear transformation between vector spaces. A neural-network layer combines a linear transformation with an additive bias, giving an affine transformation. $$ z=Wx+b $$ For d input features and m receiving neurons, the dimensions are: $$ W\in\mathbb R^{m\times d},\quad x\in\mathbb R^{d\times1},\quad b,z\in\mathbb R^{m\times1} $$ The inner dimensions must match. Output component j is the dot product of row j of W with x, plus bias bⱼ. Wᵀ reverses the matrix dimensions. The Hadamard product u⊙v multiplies corresponding components and is different from a matrix product. ![Each row of the weight matrix calculates one receiving neuron's preactivation.](https://deep-learning-study-room.vercel.app/assets/figures/v_matrix.svg) Diagram: Each row of the weight matrix calculates one receiving neuron's preactivation.. Labels: A vector is a list. A matrix has rows and columns. | A vector is a list. A matrix has rows and columns. | The shape tells you how the values are arranged. | 2 | 1 | 3 | Vector | 3 × 1 | 1 | 2 | 3 | 4 | 5 | 6 | Matrix | 2 rows × 3 columns | Row 1 | Row 2 | A 2 × 3 matrix has six entries. The shape is not the multiplication result. Each row of the weight matrix calculates one receiving neuron's preactivation. Tap the diagram to enlarge. **Question.** Given W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ , x = [2, 1]ᵀ and b = [0.1, −0.2]ᵀ, calculate z = Wx + b. ### Given Weights W = $\begin{bmatrix}1 & -2\\0.5 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.1,−0.2]ᵀ. ### To find Find both components of z = Wx+b. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. ### Solution — step by step **Step 1: verify compatible dimensions.** W has 2 rows and 2 columns; x has 2 rows and 1 column. The inner dimensions match, and the result has 2 rows and 1 column. **Step 2: calculate the first receiving row products.** $$ 1(2)=2 $$ $$ (-2)(1)=-2 $$ **Step 3: sum row 1 and add its bias.** $$ z_1=2+(-2)+0.1=0.1 $$ **Step 4: calculate the second receiving row products.** $$ 0.5(2)=1 $$ $$ 3(1)=3 $$ **Step 5: sum row 2 and add its bias.** $$ z_2=1+3-0.2=3.8 $$ $$ z_1=1(2)+(-2)(1)+0.1=0.1 $$ Row 1 produces output component z1 1. Select weight row 1 1. Select weight row 1 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 2 | −2 | First × first; second × second. Sum the products**0** + Bias for this row**0.1** = z1**0.1** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z_2=0.5(2)+3(1)-0.2=3.8 $$ Row 2 produces output component z2 1. Select weight row 2 1. Select weight row 2 | | | | --- | --- | | 1 | −2 | | 0.5 | 3 | 2. Use the entire input column 2. Use the entire input column | | | --- | | 2 | | 1 | 3. Match and multiply 3. Match and multiply | | | | --- | --- | | 1 | 3 | First × first; second × second. Sum the products**4** + Bias for this row**−0.2** = z2**3.8** One receiving neuron corresponds to one row of W and one bias. The other weight row is not used for this component. $$ z=\begin{bmatrix}0.1\\3.8\end{bmatrix} $$ An activation is applied afterward if the question asks for the layer output. In backpropagation, the input gradient uses Wᵀ because it maps derivatives from m outputs back to d inputs. For a single example, an outer product δxᵀ produces a weight-gradient matrix with exactly the same dimensions as W. ### Two output columns **Question. Multiply A = $\begin{bmatrix}1 & 2\\3 & 4\end{bmatrix}$ by B = $\begin{bmatrix}2 & 0\\1 & 3\end{bmatrix}$ . Show all four output entries.** ### Given $$ A=\begin{bmatrix}1&2\\3&4\end{bmatrix} $$ $$ B=\begin{bmatrix}2&0\\1&3\end{bmatrix} $$ ### To find The 2 × 2 matrix C = AB. ### Formula — which one and why Matrix multiplication pairs one row of A with one column of B for each output. It is not entry-by-entry multiplication. Cᵢⱼ is the entry in output row i, column j. $$ C_{ij}=\sum_k A_{ik}B_{kj} $$ **Yaad rakhne ka cue:** Output ki position dekho: wahi row, wahi column. ### Solution — step by step **Step 1: check the dimensions.** (2 × 2)(2 × 2) is valid; output size = 2 × 2. **Step 2: row 1 with column 1.** $$ C_{11}=1(2)+2(1)=2+2=4 $$ **Step 3: row 1 with column 2.** $$ C_{12}=1(0)+2(3)=0+6=6 $$ **Step 4: row 2 with column 1.** $$ C_{21}=3(2)+4(1)=6+4=10 $$ **Step 5: row 2 with column 2.** $$ C_{22}=3(0)+4(3)=0+12=12 $$ **Step 6: assemble the output by position.** $$ C=\begin{bmatrix}4&6\\10&12\end{bmatrix} $$ **Final answer:** C = $\begin{bmatrix}4 & 6\\10 & 12\end{bmatrix}$ , with size 2 × 2. ### Two products that must not be confused **Question. Let a = [2,3]ᵀ and b = [4,5]ᵀ. Find the entry-by-entry product a ⊙ b and the outer product abᵀ.** ### Given a and b are both 2 × 1 columns: a contains 2,3 and b contains 4,5. ### To find The 2 × 1 entry-by-entry product and the 2 × 2 outer product. ### Formula — which one and why ⊙ multiplies corresponding positions; the outer product combines every value of a with every value of b. Backpropagation uses both operations for different reasons. ⊙ is called the Hadamard or elementwise product. bᵀ is a 1 × 2 row. $$ (a\odot b)_i=a_i b_i $$ $$ (ab^T)_{ij}=a_i b_j $$ **Yaad rakhne ka cue:** ⊙ mein same jagah; outer product mein har pair. ### Solution — step by step **Step 1: multiply matching entries.** $$ a\odot b=\begin{bmatrix}8\\15\end{bmatrix} $$ **Step 2: transpose b.** $$ b^T=\begin{bmatrix}4&5\end{bmatrix} $$ **Step 3: form row 1 of the outer product.** $$ 2(4)=8,\qquad2(5)=10 $$ **Step 4: form row 2 of the outer product.** $$ 3(4)=12,\qquad3(5)=15 $$ **Step 5: write both rows.** $$ ab^T=\begin{bmatrix}8&10\\12&15\end{bmatrix} $$ **Final answer:** Entry-by-entry result = [8,15]ᵀ. Outer product = $\begin{bmatrix}8 & 10\\12 & 15\end{bmatrix}$ . ## Drill 5 Matrix rows receive inputs ## Question **Question.** For the weight matrix W shown below, x = [2,1], b = [0.5,−0.5], find Wx+b and ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_ed1fcbfdf5.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W | 1 | 2 | −1 | 3 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given W = $\begin{bmatrix}1 & 2\\-1 & 3\end{bmatrix}$ . Input column x = [2,1]ᵀ; bias column b = [0.5,−0.5]ᵀ. Activation: ReLU on each result. ### To find Find Wx+b and the activated output. ### Formula — which one and why **Row–column multiplication.** Each receiving neuron uses one weight row. Multiply that row by the input column, add the products, then add that neuron’s bias. Wⱼᵢ = weight from input i to receiver j; xᵢ = input i; bⱼ = bias of receiver j; zⱼ = its preactivation. A superscript T means transpose: exchange rows and columns. $$ z_j=\sum_i W_{ji}x_i+b_j $$ **Yaad rakhne ka cue:** Ek row × poora input column = ek receiving neuron ka answer. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formula Each row supplies the weights of one receiving neuron. Multiply that row by the input column, then add that neuron's own bias. The two biases are separate; do not add both to every row. $$ z=Wx+b,\qquad a_j=\max(0,z_j) $$ $$ W=\begin{bmatrix}1&2\\-1&3\end{bmatrix},\quad x=\begin{bmatrix}2\\1\end{bmatrix},\quad b=\begin{bmatrix}0.5\\-0.5\end{bmatrix} $$ ### Step 1: calculate receiving neuron 1 from row 1 $$ z_1=(1\times2)+(2\times1)+0.5 $$ $$ z_1=2+2+0.5=4.5 $$ ### Step 2: calculate receiving neuron 2 from row 2 $$ z_2=((-1)\times2)+(3\times1)+(-0.5) $$ $$ z_2=-2+3-0.5=0.5 $$ ### Step 3: apply ReLU to each entry $$ a_1=\max(0,4.5)=4.5,\qquad a_2=\max(0,0.5)=0.5 $$ $$ z=\begin{bmatrix}4.5\\0.5\end{bmatrix},\qquad a=\begin{bmatrix}4.5\\0.5\end{bmatrix} $$ The shape is 2 rows by 1 column: one output for each receiving row. ## Layer forward pass A layer does the neuron calculation for several receiving neurons. Finish all its outputs before using them as the next layer’s inputs. The bracketed superscript is a layer label, not a power. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f(z^{(l)}) $$ ## Forward propagation through multiple neurons **Definitions.** A feedforward neural network has a directed acyclic computation graph: information moves from inputs toward outputs without recurrent feedback connections. A multilayer perceptron (MLP) is a feedforward network built from fully connected layers, with one or more hidden layers and nonlinear hidden activations. ### Input, hidden and output layers The **input layer** supplies the feature vector. A **hidden layer** learns an intermediate representation; its outputs are computed from preceding activations, not supplied as target labels. The **output layer** produces the prediction in the form needed by the task. In a dense connection between two layers, every receiving neuron has a weight for every preceding activation. Each receiving neuron also normally has its own bias. An ANN is the broader family containing feedforward and recurrent models. A CNN can be feedforward without being an MLP, because convolution uses local shared connections. ![A feedforward network calculates each hidden representation before using it to calculate the next layer.](https://deep-learning-study-room.vercel.app/assets/figures/network.svg) Diagram: A feedforward network calculates each hidden representation before using it to calculate the next layer.. Labels: INPUT | HIDDEN | OUTPUT | x1 | x2 | h1 | h2 | o | Each hidden neuron receives both inputs and one bias. | Forward uses activations. Backward uses OLD weights. A feedforward network calculates each hidden representation before using it to calculate the next layer. Tap the diagram to enlarge. ### General layer equations and dimensions Let a⁽⁰⁾ = x. For layer l: $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)} $$ $$ a^{(l)}=f^{(l)}(z^{(l)}) $$ If the preceding layer has n inputs and the current layer has m neurons, W has m rows and n columns. The vectors z, a and b have m components. Row j of W contains the incoming weights of neuron j. $$ W^{(l)}\in\mathbb R^{m\times n},\qquad b^{(l)},z^{(l)},a^{(l)}\in\mathbb R^{m\times1} $$ For one hidden layer, these computations compose to give: $$ h=f^{(1)}(W^{(1)}x+b^{(1)}) $$ $$ \hat y=f^{(2)}(W^{(2)}h+b^{(2)}) $$ The hidden activation must be nonlinear to obtain nonlinear representational power. Stacking affine layers alone still gives one affine mapping. A single threshold perceptron cannot separate XOR; a hidden nonlinear representation can make the two classes separable for the output layer. ### Forward propagation versus training Forward propagation evaluates these layer equations using the current parameters. Training additionally evaluates a loss, uses backpropagation to calculate gradients, and applies an optimizer update. The architecture specifies connections; backpropagation is a differentiation algorithm used to train that architecture. ### Complete two-layer numerical **Question. A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. Calculate every preactivation and the final output.** ### Given A feedforward network receives x = [1, 0]ᵀ. Its hidden-layer weights are W¹ = $\begin{bmatrix}0.2 & 0.4\\0.3 & 0.7\end{bmatrix}$ and biases are b¹ = [0.1, 0.2]ᵀ. The output weights are W² = [0.5, 0.6] and output bias is b² = 0.1. Use ReLU in both layers and one row per receiving neuron. ### To find Find all preactivations, hidden activations and final outputs. Find the requested activation values and slopes. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ![The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.](https://deep-learning-study-room.vercel.app/assets/figures/given_5f379b367f.svg) Diagram: The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron.. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 The given weights use rows for receiving neurons. There are two hidden neurons and one output neuron. Tap the diagram to enlarge. **Step 1: identify the matrix dimensions.** W¹ is 2 × 2 and x is 2 × 1, so the hidden preactivation has two entries. W² is 1 × 2, so it takes those two activations and produces one output. **Step 2: calculate the first hidden preactivation.** Multiply the first row by x, then add the first hidden bias. $$ z^{(1)}_1=0.2(1)+0.4(0)+0.1=0.3 $$ **Step 3: calculate the second hidden preactivation.** $$ z^{(1)}_2=0.3(1)+0.7(0)+0.2=0.5 $$ **Step 4: activate the hidden layer.** Both values are positive, so ReLU retains them. $$ a^{(1)}_1=\max(0,0.3)=0.3 $$ $$ a^{(1)}_2=\max(0,0.5)=0.5 $$ $$ a^{(1)}=\begin{bmatrix}0.3\\0.5\end{bmatrix} $$ **Step 5: calculate the output preactivation.** Use the activated hidden values as this layer's inputs. $$ z^{(2)}=0.5(0.3)+0.6(0.5)+0.1 $$ $$ z^{(2)}=0.15+0.30+0.10=0.55 $$ **Step 6: apply the output activation.** $$ \hat y=\max(0,0.55)=0.55 $$ **Final answer:** hidden preactivations and activations are [0.3, 0.5]ᵀ; output preactivation and prediction are both 0.55. They coincide here because every ReLU input is positive. No parameter has been updated during this forward pass. ## Model answer: forward propagation, backpropagation and updating ## Question **Question.** Explain how a feedforward neural network learns using forward propagation, a loss function, backpropagation and gradient descent. State the order of operations. ## Answer — step by step Training repeatedly uses labeled examples to improve a network's weights and biases. The four operations have different jobs. ![Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_training_loop.svg) Diagram: Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters.. Labels: The four operations in one training step | The four operations in one training step | Repeat this sequence using the current parameters. | Predict | Forward pass | Measure error | Compute loss | Find | gradients | Backpropagation | Adjust | Optimizer | update | Calculate all gradients before replacing the old weights. Four different jobs: calculate the prediction, measure the mistake, calculate gradients, and update parameters. Tap the diagram to enlarge. | Operation | Direction or action | Result | | --- | --- | --- | | Forward propagation | Inputs move through successive layers | Predictions and intermediate activations | | Loss calculation | Compare predictions with targets | A scalar objective for the example or batch | | Backpropagation | Apply the chain rule from the loss toward earlier layers | Gradients of the objective with respect to parameters | | Optimizer update | Use those gradients to change parameters | A new set of weights and biases | ### 1. Forward propagation At each layer, calculate its weighted sums and activations. Retain these values because derivatives such as the sigmoid derivative depend on them. Complete a whole layer before calculating the next. $$ z^{(l)}=W^{(l)}a^{(l-1)}+b^{(l)},\qquad a^{(l)}=f(z^{(l)}) $$ ### 2. Loss Use the loss specified by the task. For one regression output, half squared loss is one possible choice: $$ L=\frac12(a-y)^2 $$ Here a is the prediction and y is the target. Other stated losses produce different derivatives. ### 3. Backpropagation Multiply local derivatives along each dependency path. If a hidden unit influences several outputs, add the contributions from those branches. For a connection, the weight gradient is the receiving neuron's net-input derivative multiplied by the sending activation. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ Here δⱼ means the derivative of loss with respect to the receiving neuron's weighted sum. It is not automatically equal to raw prediction error; the activation and loss determine it. ### 4. Update and repeat $$ w_{\mathrm{new}}=w_{\mathrm{old}}-\eta\frac{\partial L}{\partial w} $$ Compute all gradients using the same pre-update weights. Then update the requested parameters. Run another forward pass to calculate the new loss. Backpropagation calculates gradients; gradient descent uses them. Those are related, separate operations. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 13 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 14 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Sigmoid and its derivative Sigmoid takes a raw score and changes it to a number between 0 and 1. Its derivative is a different number: it tells backpropagation how sensitive that output is to its raw score. The complete example below calculates both and shows where each is used. $$ \sigma(z)=\frac{1}{1+e^{-z}} $$ $$ \sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ### sigmoid example 1 **Question. For preactivation z = -2, calculate the sigmoid output and state its derivative.** ### Given z = -2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(-2)=2 $$ **Step 2: calculate the exponential.** $$ e^{2}\approx7.389056099 $$ **Step 3: add one in the denominator.** $$ 1+7.389056099\approx8.389056099 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{8.389056099}\approx0.119202922 $$ **Step 5: use the activated output in the slope.** $$ f\prime(-2)=(0.119202922)(1-0.119202922)\approx0.104993585 $$ **Final answer:** a ≈ 0.119202922. Derivative ≈ 0.104993585. ### sigmoid example 2 **Question. For preactivation z = 0, calculate the sigmoid output and state its derivative.** ### Given z = 0; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: add one in the denominator.** $$ 1+1.000000000\approx2.000000000 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{2.000000000}\approx0.500000000 $$ **Step 5: use the activated output in the slope.** $$ f\prime(0)=(0.500000000)(1-0.500000000)\approx0.250000000 $$ **Final answer:** a ≈ 0.500000000. Derivative ≈ 0.250000000. ### sigmoid example 3 **Question. For preactivation z = 2, calculate the sigmoid output and state its derivative.** ### Given z = 2; activation = sigmoid. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac1{1+e^{-z}} $$ $$ f\prime(z)=a(1-a) $$ **Yaad rakhne ka cue:** One over one plus e ki minus-z power; slope = output × (1−output). ### Solution — step by step **Step 1: negate the whole input.** $$ -z=-(2)=-2 $$ **Step 2: calculate the exponential.** $$ e^{-2}\approx0.135335283 $$ **Step 3: add one in the denominator.** $$ 1+0.135335283\approx1.135335283 $$ **Step 4: divide one by the complete denominator.** $$ a=\frac1{1.135335283}\approx0.880797078 $$ **Step 5: use the activated output in the slope.** $$ f\prime(2)=(0.880797078)(1-0.880797078)\approx0.104993585 $$ **Final answer:** a ≈ 0.880797078. Derivative ≈ 0.104993585. ## Tanh and its derivative Tanh can produce a negative or positive output. In the formula, calculate the exponential first, then form the top and bottom of the fraction separately. The derivative uses the activated answer, not the raw input. $$ \tanh(z)=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ \tanh\prime(z)=1-\tanh^2(z) $$ **Question. Calculate tanh(0.5) and its derivative with respect to the input.** ### Given z = 0.5. ### To find Find tanh(z) and its derivative. ### Formula — which one and why **Tanh and its slope.** Tanh preserves a signed output between −1 and 1. Once its output a is known, the local slope is 1−a². z = value entering tanh; a = its output. A prime means derivative, or local rate of change. $$ a=\tanh z=\frac{e^z-e^{-z}}{e^z+e^{-z}} $$ $$ \frac{da}{dz}=1-a^2 $$ **Yaad rakhne ka cue:** Tanh signed output deta hai; slope = 1 minus output ka square. ### Solution — step by step **Step 1: identify the input.** The input z is 0.5. In this version of the tanh formula, the exponential uses 2z. $$ 2z=2(0.5)=1,\qquad e^{2z}=e^1\approx2.718281828 $$ **Step 2: calculate the numerator.** Subtract 1 from that exponential. $$ e^{2z}-1=2.718281828-1=1.718281828 $$ **Step 3: calculate the denominator.** Add 1 to the same exponential. $$ e^{2z}+1=2.718281828+1=3.718281828 $$ **Step 4: divide the top by the bottom.** $$ a=\tanh(0.5)=\frac{1.718281828}{3.718281828}\approx0.462117157 $$ **Step 5: calculate the local derivative.** Square the output a and subtract it from 1. Keep the calculator's unrounded result. $$ a^2\approx0.213552267,\qquad1-a^2\approx0.786447733 $$ **Final answer:** tanh output ≈ 0.462117157; local derivative ≈ 0.786447733. The output becomes the next layer's input; the derivative is used when carrying a gradient backward. ### tanh example 1 **Question. For preactivation z = -2, calculate the tanh output and state its derivative.** ### Given z = -2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(-2)=-4 $$ **Step 2: calculate the exponential.** $$ e^{-4}\approx0.018315639 $$ **Step 3: calculate numerator and denominator separately.** $$ 0.018315639-1\approx-0.981684361 $$ $$ 0.018315639+1\approx1.018315639 $$ **Step 4: divide.** $$ a=\frac{-0.981684361}{1.018315639}\approx-0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(-2)=1-(-0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ -0.964027580. Derivative ≈ 0.070650825. ### tanh example 2 **Question. For preactivation z = 0, calculate the tanh output and state its derivative.** ### Given z = 0; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(0)=0 $$ **Step 2: calculate the exponential.** $$ e^{0}\approx1.000000000 $$ **Step 3: calculate numerator and denominator separately.** $$ 1.000000000-1\approx0.000000000 $$ $$ 1.000000000+1\approx2.000000000 $$ **Step 4: divide.** $$ a=\frac{0.000000000}{2.000000000}\approx0.000000000 $$ **Step 5: square the activated output for its slope.** $$ f\prime(0)=1-(0.000000000)^2\approx1.000000000 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 1.000000000. ### tanh example 3 **Question. For preactivation z = 2, calculate the tanh output and state its derivative.** ### Given z = 2; activation = tanh. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\frac{e^{2z}-1}{e^{2z}+1} $$ $$ f\prime(z)=1-a^2 $$ **Yaad rakhne ka cue:** Tanh ka answer −1 se 1; slope one minus output squared. ### Solution — step by step **Step 1: double z before exponentiation.** $$ 2z=2(2)=4 $$ **Step 2: calculate the exponential.** $$ e^{4}\approx54.598150033 $$ **Step 3: calculate numerator and denominator separately.** $$ 54.598150033-1\approx53.598150033 $$ $$ 54.598150033+1\approx55.598150033 $$ **Step 4: divide.** $$ a=\frac{53.598150033}{55.598150033}\approx0.964027580 $$ **Step 5: square the activated output for its slope.** $$ f\prime(2)=1-(0.964027580)^2\approx0.070650825 $$ **Final answer:** a ≈ 0.964027580. Derivative ≈ 0.070650825. ## ReLU and its derivative ReLU compares its input with zero and keeps the larger value. Thus a negative input becomes zero and a positive input passes unchanged. Its slope is zero on the negative side and one on the positive side. $$ f(z)=\max(0,z) $$ $$ f\prime(z)=\begin{cases}1&z>0\\0&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### ReLU example 1 **Question. For preactivation z = -2, calculate the ReLU output and state its derivative.** ### Given z = -2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0 $$ **Final answer:** a ≈ 0.000000000. Derivative ≈ 0.000000000. ### ReLU example 2 **Question. For preactivation z = 0, calculate the ReLU output and state its derivative.** ### Given z = 0; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### ReLU example 3 **Question. For preactivation z = 2, calculate the ReLU output and state its derivative.** ### Given z = 2; activation = ReLU. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\max(0,z) $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ## Leaky ReLU Leaky ReLU changes only the negative side: multiply a negative input by the small slope α. That negative sign remains. The value α is supplied by the question or chosen as a model setting; it is not the sigmoid function. $$ f(z)=\begin{cases}z&z\geq0\\az&z<0\end{cases} $$ ## Drill 8 Activation contrast ## Question **Question.** For z = [−3,0,2], find ReLU and Leaky ReLU with α = 0.1, and the sigmoid derivative at z = 0. ## Answer — step by step ### Given Inputs z = [−3,0,2]. Leaky-ReLU negative slope α = 0.1. For the sigmoid derivative, use z = 0. ### To find Find ReLU and Leaky-ReLU outputs and the sigmoid slope at zero. ### Formula — which one and why **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). ### Solution — step by step ### Formulas and their meanings ReLU compares the input with zero and returns the larger value. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Leaky ReLU uses the input unchanged when nonnegative, and multiplies it by α when negative. $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ ### Step 1: apply ReLU separately to all three inputs $$ \max(0,-3)=0,\quad \max(0,0)=0,\quad \max(0,2)=2 $$ **ReLU output = [0, 0, 2].** ### Step 2: use α = 0.1 only on the negative input $$ 0.1\times(-3)=-0.3 $$ The zero and positive entries remain 0 and 2. **Leaky ReLU output = [−0.3, 0, 2].** ### Step 3: calculate sigmoid before its derivative $$ \sigma(z)=\frac1{1+e^{-z}},\qquad \sigma'(z)=\sigma(z)[1-\sigma(z)] $$ $$ \sigma(0)=\frac1{1+e^0}=\frac1{1+1}=0.5 $$ $$ \sigma'(0)=0.5(1-0.5)=0.5\times0.5=0.25 $$ The sigmoid **value** is 0.5 but its **slope** is 0.25. They are different quantities. ### Leaky ReLU example 1 **Question. For preactivation z = -2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = -2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** -2 is negative, so use the negative branch. **Step 2: calculate the output.** $$ a=-0.2 $$ **Step 3: state the slope on that branch.** $$ f\prime(-2)=0.1 $$ **Final answer:** a ≈ -0.200000000. Derivative ≈ 0.100000000. ### Leaky ReLU example 2 **Question. For preactivation z = 0, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 0; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** Both branches meet at zero. The value is zero, but the left and right slopes differ. **Step 2: calculate the output.** $$ a=0 $$ **Step 3: state the slope on that branch.** The ordinary derivative at zero does not exist. A software subgradient convention may choose a value such as 0; state it if required. **Final answer:** a ≈ 0.000000000. Ordinary derivative is undefined at zero. ### Leaky ReLU example 3 **Question. For preactivation z = 2, calculate the Leaky ReLU output and state its derivative. Use negative-side slope α = 0.1.** ### Given z = 2; activation = Leaky ReLU. α = 0.1. ### To find The output a and the derivative with respect to z. ### Formula — which one and why Use the activation named in the question. The derivative is a second calculation required for a backward pass. z is the raw sum; a is the activated output; f′(z) is the local slope. $$ a=\begin{cases}z&z\ge0\\0.1z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive side slope 1; negative side ReLU 0, Leaky α. Zero par convention likho. ### Solution — step by step **Step 1: select the branch from the sign of z.** 2 is positive, so retain it. **Step 2: calculate the output.** $$ a=2 $$ **Step 3: state the slope on that branch.** $$ f\prime(2)=1 $$ **Final answer:** a ≈ 2.000000000. Derivative ≈ 1.000000000. ## Stable softmax Softmax turns several competing class scores into probabilities that add to one. Exponentiate each score, add those exponential values, and divide each one by their common total. Subtracting the same maximum score first gives the same probabilities with safer arithmetic. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ $$ c=\max_j z_j $$ ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. ### Softmax for [0, 0, 0] **Question. Find stable softmax probabilities for logits [0, 0, 0].** ### Given Logits = [0, 0, 0]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}0&0&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{0}\approx1.000000000 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{0}\approx1.000000000 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=1.000000000+1.000000000+1.000000000\approx3.000000000 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{3.000000000}\approx0.333333333 $$ **Final answer:** Probabilities ≈ [0.333333333, 0.333333333, 0.333333333]; they sum to 1 within rounding. ### Softmax for [1, 2, 3] **Question. Find stable softmax probabilities for logits [1, 2, 3].** ### Given Logits = [1, 2, 3]; there are three mutually exclusive classes. ### To find All three class probabilities. ### Formula — which one and why Subtracting one common maximum preserves probabilities and keeps exponentials well scaled. Normalize using one shared sum. A logit is a raw class score. c is the largest score; pᵢ is probability for class i. $$ c=\max_j z_j $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}} $$ **Yaad rakhne ka cue:** Max minus, e-power, same denominator se sab divide. ### Solution — step by step **Step 1: subtract the largest score from every entry.** $$ z-c=\begin{bmatrix}-2&-1&0\end{bmatrix} $$ **Step 2: exponentiate shifted entry 1.** $$ e^{-2}\approx0.135335283 $$ **Step 3: exponentiate shifted entry 2.** $$ e^{-1}\approx0.367879441 $$ **Step 4: exponentiate shifted entry 3.** $$ e^{0}\approx1.000000000 $$ **Step 5: add all exponentials into one denominator.** $$ D=0.135335283+0.367879441+1.000000000\approx1.503214724 $$ **Step 6: divide entry 1 by the same denominator.** $$ p_1=\frac{0.135335283}{1.503214724}\approx0.090030573 $$ **Step 7: divide entry 2 by the same denominator.** $$ p_2=\frac{0.367879441}{1.503214724}\approx0.244728471 $$ **Step 8: divide entry 3 by the same denominator.** $$ p_3=\frac{1.000000000}{1.503214724}\approx0.665240956 $$ **Final answer:** Probabilities ≈ [0.090030573, 0.244728471, 0.665240956]; they sum to 1 within rounding. ## Squared and absolute losses SSE, half SSE and MSE start with the same errors but apply different final divisions. First understand what an error and its square mean. The full question below keeps all three answers separate. $$ \mathrm{SSE}=\sum_i(o_i-y_i)^2 $$ $$ L_{\mathrm{half}}=\frac{\mathrm{SSE}}2 $$ $$ L_{\mathrm{MSE}}=\frac{\mathrm{SSE}}n $$ $$ L_{\mathrm{MAE}}=\frac1n\sum_i|o_i-y_i| $$ ## Loss: how one wrong prediction becomes a number **Definition.** A residual is the signed prediction error. Squared-error objectives aggregate squared residuals; their scaling convention changes both the reported loss and its derivative. $$ e_i=\hat y_i-y_i,\qquad \mathrm{SSE}=\sum_{i=1}^{n}e_i^2,\qquad L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2,\qquad \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2 $$ For this definition of n, each i indexes one included scalar prediction. In a multi-output batch, state whether the mean is over examples, output coordinates, or both. Half SSE and MSE happen to agree when n = 2; they are not generally the same objective. ### Derivatives and their scaling When differentiating with respect to prediction j, all other prediction terms are constant. The derivative of the j-th square is twice its residual: $$ \frac{\partial\mathrm{SSE}}{\partial\hat y_j}=2e_j,\qquad \frac{\partial L_{\mathrm{half}}}{\partial\hat y_j}=e_j,\qquad \frac{\partial\mathrm{MSE}}{\partial\hat y_j}=\frac{2}{n}e_j $$ The factor 1/2 cancels the derivative's factor 2. The factor 1/n instead produces 2/n. A weight derivative must also multiply by the derivative of the prediction with respect to that weight. **Question. Predictions are [0.2, 0.7, 0.4] and targets are [0, 1, 1]. Find SSE, half SSE and MSE, then their gradients with respect to the prediction vector.** ### Given Predictions = [0.2,0.7,0.4]. Targets = [0,1,1]; n = 3 scalar predictions. ### To find Find SSE, half SSE and MSE, then all three prediction-gradient lists. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step **Step 1: calculate the three signed prediction errors.** $$ e=[0.2-0,\ 0.7-1,\ 0.4-1]=[0.2,-0.3,-0.6] $$ **Step 2: square every error.** $$ e\odot e=[0.04,0.09,0.36] $$ **Step 3: add the three squared errors.** $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ **Step 4: apply the requested loss divisors.** $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245,\qquad \mathrm{MSE}=\frac{0.49}{3}\approx0.163333333 $$ **Step 5: multiply every error by 2 for the SSE gradient.** $$ \nabla_{\hat y}\mathrm{SSE}=[0.4,-0.6,-1.2] $$ **Step 6: use each error itself for the half-SSE gradient.** $$ \nabla_{\hat y}L_{\mathrm{half}}=[0.2,-0.3,-0.6] $$ **Step 7: multiply every error by 2/3 for the MSE gradient.** $$ \nabla_{\hat y}\mathrm{MSE}=\frac23[0.2,-0.3,-0.6]=[0.133333333,-0.2,-0.4] $$ The residual signs survive in the gradients even though the losses are nonnegative. Multiplying a loss by a positive constant leaves its unregularized minimizers unchanged, but scales its gradients and therefore its updates at a fixed learning rate. ## Drill 2 Half sum versus mean ## Question **Question.** Predictions = [0.2,0.7,0.4], targets = [0,1,1]. Find SSE, half SSE and MSE. ## Answer — step by step ### Given Predictions = [0.2, 0.7, 0.4]. Targets = [0, 1, 1]; number of outputs n = 3. ### To find Find SSE, half SSE, MSE and the prediction gradients requested in the question. ### Formula — which one and why **Squared-error objectives.** Calculate signed errors first and square them. SSE adds the squares; half SSE divides that sum by 2; MSE divides by the number of scalar predictions. These denominators are different conventions. ŷᵢ = prediction; yᵢ = target; eᵢ = signed error; n = number of scalar predictions included in the stated loss. $$ e_i=\hat y_i-y_i $$ $$ \mathrm{SSE}=\sum_i e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_i e_i^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i e_i^2 $$ **Yaad rakhne ka cue:** Pehle error square; phir question ke hisaab se sum, half ya mean. ### Solution — step by step ### Given values and loss definitions There are n = 3 prediction–target pairs. Use the supplied predictions directly; no rounding to class labels is required. Let eᵢ = ŷᵢ − yᵢ be the signed residual for pair i. $$ \mathrm{SSE}=\sum_{i=1}^{n}e_i^2 $$ $$ L_{\mathrm{half}}=\frac12\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}2 $$ $$ \mathrm{MSE}=\frac1n\sum_{i=1}^{n}e_i^2=\frac{\mathrm{SSE}}n $$ SSE is the sum of squared errors. Half SSE and MSE rescale that same sum by different denominators: 2 and n, respectively. ### Step 1: calculate each residual and its square | Pair | Prediction ŷᵢ | Target yᵢ | Residual ŷᵢ − yᵢ | Squared residual | | --- | --- | --- | --- | --- | | 1 | 0.2 | 0 | 0.2 − 0 = 0.2 | (0.2)² = 0.04 | | 2 | 0.7 | 1 | 0.7 − 1 = −0.3 | (−0.3)² = 0.09 | | 3 | 0.4 | 1 | 0.4 − 1 = −0.6 | (−0.6)² = 0.36 | ### Step 2: sum the squared residuals $$ \mathrm{SSE}=0.04+0.09+0.36=0.49 $$ Square each residual before summing. Squaring the sum of signed residuals would be a different quantity, because positive and negative residuals could cancel. ### Step 3: apply each requested normalization For half SSE, the denominator is 2: $$ L_{\mathrm{half}}=\frac{0.49}{2}=0.245 $$ For MSE, the denominator is the number of pairs, n = 3: $$ \mathrm{MSE}=\frac{0.49}{3}=0.163333333\ldots $$ **Final answer:** SSE = 0.49; half SSE = 0.245; MSE ≈ 0.163333. They represent the same residuals at different scales. Half SSE and MSE coincide when n = 2; they do not coincide for these three nonzero residuals. No derivative is required by this question. ### Absolute error uses a different operation **Question. Targets are [2, 4] and predictions are [1, 6]. Find mean absolute error.** ### Given Targets = [2,4]; predictions = [1,6]; n = 2. ### To find Find mean absolute error. ### Formula — which one and why **Mean absolute error.** Absolute value removes the sign of each prediction error without squaring its magnitude. Average those magnitudes. n = number of predictions. |e| means the nonnegative magnitude of error e. $$ \mathrm{MAE}=\frac1n\sum_i|\hat y_i-y_i| $$ **Yaad rakhne ka cue:** MAE mein magnitude; MSE mein square. ### Solution — step by step The signed errors are 1 − 2 = −1 and 6 − 4 = 2. Absolute value keeps the size of a number without its sign, so their absolute errors are 1 and 2. Add them to obtain 3. There are two prediction–target pairs, so divide by 2. $$ \mathrm{MAE}=\frac{|1-2|+|6-4|}{2}=\frac{1+2}{2}=1.5 $$ **Final answer: MAE = 1.5.** Do not square the errors for MAE. ## Binary cross entropy Binary cross-entropy measures how much probability was assigned to the correct binary answer. A target of 1 keeps the −ln(p) term; a target of 0 keeps the −ln(1−p) term. The factors y and 1−y select the appropriate term. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### Binary cross-entropy: target 1, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.8 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.223143551)=0.223143551 $$ **Final answer:** Loss ≈ 0.223143551 nats. ### Binary cross-entropy: target 0, predicted positive-class probability 0.8 **Question. A binary classifier predicts P(y=1) = 0.8. The target is y = 0. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 0; p = P(y=1) = 0.8; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ 1-p=1-0.8=0.2 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.2)\approx-1.609437912 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-1.609437912)=1.609437912 $$ **Final answer:** Loss ≈ 1.609437912 nats. ### Binary cross-entropy: target 1, predicted positive-class probability 0.5 **Question. A binary classifier predicts P(y=1) = 0.5. The target is y = 1. Calculate binary cross-entropy using natural logarithms.** ### Given Target y = 1; p = P(y=1) = 0.5; use natural log ln. ### To find The binary cross-entropy loss for this one example. ### Formula — which one and why The target selects the probability assigned to the correct class. Take its negative natural logarithm. p is always the probability of class 1. When y = 0, the correct-class probability is 1−p. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ **Yaad rakhne ka cue:** True class ki probability lo, uska minus ln lo. ### Solution — step by step **Step 1: identify the correct-class probability.** $$ p=0.5 $$ **Step 2: evaluate its natural logarithm.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: apply the minus sign outside the logarithm.** $$ L=-(-0.693147181)=0.693147181 $$ **Final answer:** Loss ≈ 0.693147181 nats. ## Categorical cross entropy For one-hot multiclass targets, only one target entry is 1. The other entries are zero, so the loss becomes minus the logarithm of the true class probability. The worked question first calculates every softmax probability, then selects the correct one. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ \log_2q=\frac{\ln q}{\ln2} $$ $$ L=-\sum_i y_i\ln p_i=-\ln p_{true} $$ ## Cross-entropy for probability distributions **Definition.** For a discrete target distribution p and a predicted distribution q over the same classes, cross-entropy is the target-weighted negative logarithm of the predicted probabilities. $$ H(p,q)=-\sum_{i=1}^{K}p_i\log q_i $$ Here pᵢ and qᵢ refer to the same class i, and each distribution sums to 1. The target supplies the weights outside the logarithm; the prediction goes inside it. If the target is one-hot, only the correct-class term remains. For a soft target with several nonzero entries, include all its nonzero terms. ### Logarithm base and units Natural logarithms produce **nats**; base-2 logarithms produce **bits**. These notes use ln unless a question states another base. On a calculator with ln, use the change-of-base rule for base 2: $$ \log_2 q=\frac{\ln q}{\ln 2} $$ For a target entry pᵢ = 0, its contribution is defined as zero. If pᵢ > 0 but qᵢ = 0, the loss is infinite. For numerical stability, software normally calculates cross-entropy from logits or carefully handles boundary probabilities. ### A target distribution with two nonzero entries **Question. A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms.** ### Given A two-class target distribution is p = [0.75, 0.25] and the model predicts q = [0.5, 0.5]. Calculate cross-entropy using natural logarithms and using base-2 logarithms. ### To find Find cross-entropy in the requested units. ### Formula — which one and why **Cross-entropy between distributions.** Weight each negative log prediction by the target probability. Unlike one-hot targets, several terms can contribute. pᵢ = target-distribution probability; qᵢ = predicted probability. Natural logs give nats; base-2 logs give bits. $$ H(p,q)=-\sum_i p_i\ln q_i $$ $$ H_2(p,q)=\frac{H(p,q)}{\ln2} $$ **Yaad rakhne ka cue:** Target probability batati hai kis log-loss ka kitna hissa lena hai. ### Solution — step by step **Step 1: write the two terms.** Both target entries are nonzero, so neither term can be discarded. $$ H(p,q)=-[0.75\ln(0.5)+0.25\ln(0.5)] $$ **Step 2: calculate the logarithms.** $$ \ln(0.5)\approx-0.693147181 $$ **Step 3: weight each negative logarithm by its target probability.** $$ -0.75\ln(0.5)\approx0.519860385 $$ $$ -0.25\ln(0.5)\approx0.173286795 $$ **Step 4: add the contributions, retaining calculator precision.** $$ H(p,q)\approx0.519860385+0.173286795\approx0.693147181\ \text{nats} $$ **Step 5: repeat with base 2.** Since 2⁻¹ = 0.5, log₂(0.5) = −1. $$ H_2(p,q)=-[0.75(-1)+0.25(-1)]=1\ \text{bit} $$ **Final answer:** 0.693147 nats or 1 bit. The distributions are unchanged; the logarithm base changes the unit and the numerical scale. ### Cross-entropy, entropy and relative entropy These names are related but do not denote the same quantity. Entropy uses the target distribution in both positions. Relative entropy, also called KL divergence, measures their difference through a probability ratio. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=\sum_i p_i\ln\frac{p_i}{q_i}=H(p,q)-H(p) $$ For fixed p, H(p) is constant with respect to the model parameters, so minimizing cross-entropy also minimizes this KL divergence. KL is clarified here to distinguish the terminology; it is not a separately named unit in the official syllabus. **Question. For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats.** ### Given For the same p = [0.75, 0.25] and q = [0.5, 0.5], find H(p) and DKL(p ∥ q) in nats. ### To find Find entropy and KL divergence in nats. ### Formula — which one and why **Entropy and relative entropy.** Entropy measures the target distribution’s uncertainty. Relative entropy measures the extra cross-entropy when q is used instead of p. p = target distribution; q = model distribution; H(p) = entropy; DKL = Kullback–Leibler divergence. $$ H(p)=-\sum_i p_i\ln p_i $$ $$ D_{\mathrm{KL}}(p\Vert q)=H(p,q)-H(p) $$ **Yaad rakhne ka cue:** KL = cross-entropy minus target ki own entropy. ### Solution — step by step **Step 1: calculate the target entropy.** $$ H(p)=-[0.75\ln(0.75)+0.25\ln(0.25)] $$ $$ \ln(0.75)\approx-0.287682072,\qquad\ln(0.25)\approx-1.386294361 $$ $$ H(p)\approx0.215761554+0.346573590=0.562335145 $$ **Step 2: subtract target entropy from cross-entropy.** $$ D_{\mathrm{KL}}(p\Vert q)\approx0.693147181-0.562335145=0.130812036 $$ **Final answer:** H(p) ≈ 0.562335 nats and DKL(p ∥ q) ≈ 0.130812 nats. For a one-hot target H(p) = 0, so cross-entropy and this KL divergence coincide; for this soft target they differ. ## Drill 9 Softmax and true class loss ## Question **Question.** Scores = [1,2,0], true one-hot class = [0,1,0]. Find probabilities and categorical cross-entropy. ## Answer — step by step ### Given Logits = [1,2,0]. One-hot target = [0,1,0]: class 2 is correct. ### To find Find all three softmax probabilities and CCE. ### Formula — which one and why **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Meaning and formulas The scores z are raw class scores. Softmax converts the whole group into class probabilities. The label [0,1,0] says class 2 is correct; it is not a set of predicted probabilities. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ Subtracting the same c from every score keeps the probabilities unchanged and avoids very large exponentials. $$ L_{\mathrm{CCE}}=-\sum_i y_i\ln p_i $$ ### Step 1: subtract the largest score The largest of [1,2,0] is 2. $$ [1-2,\;2-2,\;0-2]=[-1,0,-2] $$ ### Step 2: calculate the three exponentials $$ e^{-1}=0.367879441,\quad e^0=1,\quad e^{-2}=0.135335283 $$ ### Step 3: add them to get one common denominator $$ D=0.367879441+1+0.135335283=1.503214724 $$ ### Step 4: divide each exponential by this denominator $$ p_1=\frac{0.367879441}{1.503214724}=0.244728471 $$ $$ p_2=\frac{1}{1.503214724}=0.665240956 $$ $$ p_3=\frac{0.135335283}{1.503214724}=0.090030573 $$ The probabilities add to 1, apart from rounding. Class 2 has the highest score and the highest probability. ### Step 5: use the probability of the correct class in CCE $$ L=-[0\ln p_1+1\ln p_2+0\ln p_3]=-\ln p_2 $$ $$ L=-\ln(0.665240956)=0.407605964 $$ **Final answer:** probabilities ≈ [0.244728, 0.665241, 0.090031]; CCE ≈ 0.407606. ## Output delta for half squared loss Delta is a short name for the loss derivative with respect to a neuron’s raw input z. For half squared loss, multiply prediction minus target by the activation’s derivative. The result is then multiplied by an incoming activation to obtain a weight gradient. $$ \delta^{out}=(o-y)f\prime(z) $$ ## Loss scaling: derive the gradient instead of guessing a factor ### 1. One error, three possible objectives Let a be a prediction and y a fixed target. The error is a − y. The derivative of that error with respect to a is 1 because y stays fixed. Apply the chain rule to its square: ![Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.](https://deep-learning-study-room.vercel.app/assets/figures/v_loss_gradient.svg) Diagram: Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged.. Labels: Prediction error and its derivative | Prediction error and its derivative | Keep the target y fixed when differentiating. | Half squared loss | L = ½(ŷ − y)² | Derivative = ŷ − y | The 2 cancels the half. | Mean squared error | L = sum of squares ÷ n | Derivative per output | = (2/n)(ŷ − y) | The scale of the loss changes the scale of its gradient. Read the two conventions separately. A derivative for one convention must not be copied into the other unchanged. Tap the diagram to enlarge. $$ \frac{\partial (a-y)^2}{\partial a}=2(a-y)\times1 $$ Multiplying the objective by a constant multiplies its derivative by the same constant. This gives the common loss conventions below. | Stated loss over n outputs | Derivative with respect to one prediction aⱼ | | --- | --- | | SSE: sum of all squared errors | 2(aⱼ − yⱼ) | | Half SSE: half of that sum | aⱼ − yⱼ | | MSE: that sum divided by n | 2(aⱼ − yⱼ)/n | | Half MSE: that sum divided by 2n | (aⱼ − yⱼ)/n | ### 2. Why only the matching output appears Suppose three predictions are separate output variables a₁, a₂ and a₃. When differentiating the loss with respect to a₂, the a₁ and a₃ terms are constant with respect to that output variable, so their direct derivatives are zero. The remaining term is the error of output 2, with the correct scaling factor. Later, a shared hidden weight may influence all three predictions. Backpropagation then combines those separate output contributions through the chain rule. “Differentiate with respect to one output” and “differentiate with respect to a shared weight” are different stages. ### 3. The activation contributes another factor If a = sigmoid(z), the derivative of a with respect to its input is a(1 − a). For half SSE, the output delta is therefore: $$ \delta=\frac{\partial L}{\partial z}=(a-y)a(1-a) $$ For a linear output, the activation derivative is 1. For positive ReLU input it is also 1; for negative ReLU input it is 0. Always use the derivative of the activation actually given in the question. ### 4. A numerical comparison For a = 0.7 and y = 1, the signed error is −0.3. For half SSE with sigmoid, the activation derivative is 0.7 × 0.3 = 0.21. The output delta is −0.3 × 0.21 = −0.063. With plain SSE it doubles to −0.126. With MSE over three outputs it is −0.126/3 = −0.042. The forward prediction 0.7 did not change. The loss convention changed the gradient scale. This is why the first line of a backpropagation answer should state the loss being used. ## Complete sigmoid training example **Question.** A sigmoid neuron has input 2, weight 0, bias 0, target 1 and half squared loss. Will gradient descent increase or decrease the weight? Use learning rate 0.1 and calculate the exact update. ### Given x = 2; w = 0; b = 0; y = 1. Sigmoid output; half squared loss; η = 0.1. ### To find Find the weight-gradient sign, exact weight update and resulting weight. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Step 1 forward:** net = 0(2) + 0 = 0; sigmoid output = 0.5. ![The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.](https://deep-learning-study-room.vercel.app/assets/figures/v_sigmoid_training.svg) Diagram: The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order.. Labels: A sigmoid neuron: forward values, then one correction | A sigmoid neuron: forward values, then one correction | Same example as the text: x = 2, w = 0, b = 0, target = 1; half squared loss. | Raw sum | 0 × 2 + 0 = 0 | Sigmoid | sigmoid(0) = 0.5 | Error | 0.5 − 1 = −0.5 | Sigmoid slope = 0.5 × (1 − 0.5) = 0.25 | Delta = −0.5 × 0.25 = −0.125 | Weight gradient = −0.125 × 2 = −0.25 | New weight = 0 − 0.1 × (−0.25) = 0.025 | The weight increases because the current prediction is too small for this | positive input. The diagram uses the same input, weight, bias and target as the question. Follow prediction, activation slope, delta and update in order. Tap the diagram to enlarge. **Step 2 backward:** prediction minus target = 0.5 − 1 = −0.5. Sigmoid slope = 0.5(1 − 0.5) = 0.25. Delta = −0.5 × 0.25 = −0.125. **Step 3 gradients and update:** weight gradient = −0.125 × 2 = −0.25. New weight = 0 − 0.1(−0.25) = 0.025. Bias gradient = −0.125; new bias = 0.0125. Both increase. With positive input, these changes initially raise the prediction toward the target. Your intuition and arithmetic now agree. ## Drill 10 Sigmoid neuron gradient ## Question **Question.** x = 2, w = 0.5, b = 0, y = 1, half squared loss, η = 0.1. Find one weight and bias update. ## Answer — step by step ### Given x = 2; w = 0.5; b = 0; target y = 1. Activation: sigmoid; loss: half squared error. Learning rate η = 0.1. ### To find Find the output, delta, weight/bias gradients and one update. ### Formula — which one and why **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Meaning and formulas Here the loss is half squared error. It is different from binary cross-entropy, so we include the sigmoid derivative explicitly. $$ z=wx+b,\quad p=\frac1{1+e^{-z}},\quad L=\frac12(p-y)^2 $$ The symbol δ (delta) is a short name for the loss gradient with respect to the neuron's raw score z. $$ \delta=(p-y)p(1-p) $$ $$ \frac{\partial L}{\partial w}=\delta x,\quad \frac{\partial L}{\partial b}=\delta $$ $$ w_{\mathrm{new}}=w-\eta\delta x,\quad b_{\mathrm{new}}=b-\eta\delta $$ ### Step 1: weighted sum and sigmoid $$ z=0.5\times2+0=1 $$ $$ p=\frac1{1+e^{-1}}=0.731058579 $$ ### Step 2: error and activation slope $$ p-y=0.731058579-1=-0.268941421 $$ $$ p(1-p)=0.731058579\times0.268941421=0.196611933 $$ ### Step 3: multiply the linked effects to obtain delta $$ \delta=(-0.268941421)\times0.196611933=-0.052877093 $$ ### Step 4: multiply delta by the input to each parameter The weight receives x = 2. The bias effectively receives the constant input 1. $$ \frac{\partial L}{\partial w}=(-0.052877093)\times2=-0.105754186 $$ $$ \frac{\partial L}{\partial b}=-0.052877093 $$ ### Step 5: subtract learning rate times each gradient $$ w_{\mathrm{new}}=0.5-0.1(-0.105754186)=0.510575419 $$ $$ b_{\mathrm{new}}=0-0.1(-0.052877093)=0.005287709 $$ **Final answer:** new weight ≈ 0.510575419; new bias ≈ 0.005287709. Subtracting negative gradients increases both values here. ## Drill 13 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 14 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Sigmoid plus BCE; softmax plus CCE Some activation–loss pairs simplify together. Sigmoid with binary cross-entropy, and softmax with one-hot categorical cross-entropy, give prediction minus target at the raw-score level. This result already includes the activation’s effect. $$ \delta^{out}=p-y $$ ## Drill 7 Sigmoid with binary cross entropy ## Question **Question.** A binary classifier has logit z = 1, target 1. Find probability, BCE and the gradient with respect to z. ## Answer — step by step ### Given Logit z = 1; binary target y = 1. Output: sigmoid; loss: binary cross-entropy. ### To find Find probability, BCE and the loss derivative with respect to z. ### Formula — which one and why **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Meaning and formulas A logit is the raw score before sigmoid. Sigmoid converts it to a probability p. The target y = 1 says the positive class is correct. Binary cross-entropy measures how little probability the model assigned to the correct binary outcome. $$ p=\frac1{1+e^{-z}} $$ $$ L=-\left[y\ln p+(1-y)\ln(1-p)\right] $$ For sigmoid combined with this loss, the gradient with respect to the logit has the simplified form: $$ \frac{\partial L}{\partial z}=p-y $$ ### Step 1: convert logit 1 into probability $$ e^{-1}\approx0.367879441 $$ $$ 1+e^{-1}\approx1.367879441 $$ $$ p=\frac1{1.367879441}\approx0.731058579 $$ ### Step 2: substitute the target in the full loss formula $$ L=-\left[1\ln(0.731058579)+(1-1)\ln(1-0.731058579)\right] $$ The second term has multiplier zero, so it contributes zero. $$ L=-\ln(0.731058579)\approx0.313261688 $$ ### Step 3: calculate the combined gradient $$ \frac{\partial L}{\partial z}=0.731058579-1=-0.268941421 $$ **Final answer:** probability ≈ 0.731059; BCE ≈ 0.313262; logit gradient ≈ −0.268941. The simplified gradient already includes sigmoid's effect. Multiplying by sigmoid's slope again would apply it twice. ### The multiclass version, entry by entry **Question. Softmax probabilities are [0.1, 0.8, 0.1] and the one-hot target is [0, 1, 0]. For categorical cross-entropy on this one example, find the derivatives with respect to all three logits.** ### Given Softmax probabilities = [0.1,0.8,0.1]; target = [0,1,0]. One-example categorical cross-entropy. ### To find Find all three logit derivatives. ### Formula — which one and why **Softmax with categorical cross-entropy.** For one example with one-hot target, differentiating softmax together with CCE simplifies the logit derivative to predicted probability minus target. zᵢ = raw score before softmax; pᵢ = class probability; yᵢ = matching target entry. $$ \frac{\partial L}{\partial z_i}=p_i-y_i $$ **Yaad rakhne ka cue:** Softmax + CCE together: p minus y. Extra sigmoid slope nahi. ### Solution — step by step A logit is the raw score before softmax. Subtract each target entry from its matching probability entry. $$ \frac{\partial L}{\partial z_1}=0.1-0=0.1 $$ $$ \frac{\partial L}{\partial z_2}=0.8-1=-0.2 $$ $$ \frac{\partial L}{\partial z_3}=0.1-0=0.1 $$ **Final answer: [0.1, −0.2, 0.1].** Do not multiply these by a separate sigmoid derivative or by p(1−p). A mean over several examples introduces the corresponding averaging factor. ### Sigmoid+BCE logit gradient: p=0.8, y=1 **Question. A sigmoid output is p = 0.8 and target y = 1. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 1; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-1}{0.16}=-1.25 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(-1.25)(0.16)=-0.2 $$ **Final answer:** dL/dz = -0.2. No further sigmoid derivative is needed. ### Sigmoid+BCE logit gradient: p=0.8, y=0 **Question. A sigmoid output is p = 0.8 and target y = 0. For binary cross-entropy on one example, find the derivative with respect to the pre-sigmoid score z.** ### Given p = 0.8; y = 0; single-example BCE; p = sigmoid(z). ### To find dL/dz, not dL/dp. ### Formula — which one and why Apply the chain rule to BCE followed backward through sigmoid. The factors p(1−p) cancel, leaving p−y. p is an activated probability; z is its raw score; the requested derivative is with respect to z. $$ \frac{dL}{dp}=\frac{p-y}{p(1-p)} $$ $$ \frac{dp}{dz}=p(1-p) $$ $$ \frac{dL}{dz}=p-y $$ **Yaad rakhne ka cue:** BCE + sigmoid together ka shortcut p−y; chain already included. ### Solution — step by step **Step 1: calculate the sigmoid slope.** $$ p(1-p)=0.8(0.2)=0.16 $$ **Step 2: differentiate BCE with respect to p.** $$ \frac{dL}{dp}=\frac{0.8-0}{0.16}=5 $$ **Step 3: multiply the two local derivatives.** $$ \frac{dL}{dz}=(5)(0.16)=0.8 $$ **Final answer:** dL/dz = 0.8. No further sigmoid derivative is needed. ## Hidden delta, weight gradient and bias gradient A hidden neuron can influence several outputs. Add all those backward contributions before multiplying by its own activation derivative. For each connection, use the receiving neuron’s delta and the sending neuron’s activation. Keep the old weights unchanged until all gradients have been found. $$ \delta_j^h=f\prime(z_j^h)\sum_k w_{kj}^{out}\delta_k^{out} $$ $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ ## More outputs and vectorized backpropagation **Definition.** Vectorized backpropagation expresses the chain rule using matrix products and elementwise derivatives. When a hidden activation affects several outputs, its derivative contains the sum of the contributions from every downstream path. ### Why the hidden derivative is a sum If hidden activation hⱼ connects to output preactivations zₒ₁ and zₒ₂, changing hⱼ changes both outputs. The multivariable chain rule gives: $$ \frac{\partial L}{\partial h_j}=W_{2,1j}\delta_{o_1}+W_{2,2j}\delta_{o_2} $$ For any number of connected outputs, sum their contributions and then differentiate the hidden activation: $$ \delta_{h_j}=\left(\sum_k W_{2,kj}\delta_{o_k}\right)f'(z_{h_j}) $$ Use the connecting weights from the original forward pass. Adding only the contribution of the first output would omit a dependency of the loss. ![The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.](https://deep-learning-study-room.vercel.app/assets/figures/v_branch_gradient.svg) Diagram: The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative.. Labels: Two output paths both affect the hidden neuron | Two output paths both affect the hidden neuron | Example: hidden activation h = 0.6, using sigmoid in the hidden neuron. | h = 0.6 | Output 1 delta = 0.2 | Connecting weight = 0.4 | Output 2 delta = −0.1 | Connecting weight = 0.5 | Downstream sum = 0.2 × 0.4 + (−0.1) × 0.5 = 0.03 | Hidden delta = 0.03 × 0.6 × (1 − 0.6) = 0.0072 | Add the two downstream paths first. Then multiply by the hidden activation | derivative. The outgoing paths both contribute to the hidden derivative. Add the weighted output deltas, then apply the hidden activation derivative. Tap the diagram to enlarge. ### Matrix equations for one training sample Use column vectors throughout. Let W₁ map d inputs to m hidden units and W₂ map those units to k outputs. The following output-delta formula assumes **half sum of squared errors** and an elementwise output activation f₂: $$ L=\frac12\sum_{j=1}^{k}(\hat y_j-y_j)^2 $$ $$ \delta_2=(\hat y-y)\odot f_2'(z_2) $$ $$ \frac{\partial L}{\partial W_2}=\delta_2a_1^T,\qquad\frac{\partial L}{\partial b_2}=\delta_2 $$ $$ \delta_1=(W_2^T\delta_2)\odot f_1'(z_1) $$ $$ \frac{\partial L}{\partial W_1}=\delta_1x^T,\qquad\frac{\partial L}{\partial b_1}=\delta_1 $$ Here ⊙ means elementwise multiplication. The product δ₂a₁ᵀ is an **outer product**: every receiving delta is multiplied by every sending activation. | Quantity | Shape | Reason | | --- | --- | --- | | W₁ and its gradient | m × d | m receiving hidden neurons, d inputs | | W₂ and its gradient | k × m | k receiving output neurons, m hidden activations | | δ₂ | k × 1 | One derivative per output preactivation | | W₂ᵀδ₂ | m × 1 | The summed downstream derivative for each hidden unit | | δ₂a₁ᵀ | (k × 1)(1 × m) = k × m | The output-weight gradient has the same shape as W₂ | ### How the loss changes the output delta For MSE averaged over k output components, the squared-error derivative has the factor 2/k. For sigmoid with binary cross-entropy, or softmax with categorical cross-entropy and a normalized target, the paired derivative simplifies to prediction minus target: $$ \delta_{\mathrm{output}}=\hat y-y $$ That simplification already includes differentiation of the paired activation and loss. Multiplying by a sigmoid derivative again would be incorrect. For sigmoid with half squared error, the sigmoid derivative is still required. For a batch, sum per-example parameter gradients; divide by the batch size only if the objective is a batch mean. State the output reduction and batch reduction separately when both appear in a question. ## Backpropagation with one hidden neuron **Definition.** Backpropagation computes the derivatives of a scalar objective with respect to network parameters by applying the chain rule in reverse through the computational graph. An optimizer then uses these derivatives to update the parameters. ### Network equations and the meaning of delta Consider one input, one hidden neuron and one output neuron: $$ z_h=w_1x+b_1,\qquad h=f(z_h) $$ $$ z_o=w_2h+b_2,\qquad\hat y=g(z_o) $$ For the half squared loss used in this example: $$ L=\frac12(\hat y-y)^2 $$ Define a neuron's delta as the derivative of the loss with respect to its **preactivation**, not with respect to its output: $$ \delta_o=\frac{\partial L}{\partial z_o}=(\hat y-y)g'(z_o) $$ The output-weight derivative contains the hidden activation because zₒ = w₂h + b₂: $$ \frac{\partial L}{\partial w_2}=\delta_o h,\qquad\frac{\partial L}{\partial b_2}=\delta_o $$ To reach the hidden neuron, continue backwards through the old output weight and the hidden activation: $$ \delta_h=\delta_o w_2 f'(z_h) $$ $$ \frac{\partial L}{\partial w_1}=\delta_h x,\qquad\frac{\partial L}{\partial b_1}=\delta_h $$ The bias derivative equals the corresponding delta because the derivative of z with respect to its additive bias is 1. All these derivatives belong to the same forward pass; an updated weight must not be used partway through this backward pass. ### Complete worked numerical **Question.** x = 2, y = 1; w₁ = 0.5, b₁ = 0.1; w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### Given x = 2, y = 1. w₁ = 0.5, b₁ = 0.1. w₂ = 0.8, b₂ = 0.2. Both neurons use ReLU. Use half squared loss. Find the output and all four parameter gradients, then update all four parameters using η = 0.1. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ![The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.](https://deep-learning-study-room.vercel.app/assets/figures/v_single_backprop.svg) Diagram: The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights.. Labels: Follow this one-hidden-neuron example in both directions | Follow this one-hidden-neuron example in both directions | x = 2, target = 1. Both activations are ReLU; both raw sums are positive. | 2 | INPUT | 1.1 | HIDDEN | 1.08 | OUTPUT | weight 0.5 | weight 0.8 | 0.5 × 2 + 0.1 = 1.1 | 0.8 × 1.1 + 0.2 = 1.08 | Output delta = (1.08 − 1) × 1 = 0.08 | Back through old weight 0.8 | Hidden delta = 0.08 × 0.8 × 1 = 0.064 | For a weight gradient, multiply its receiving delta by the value entering that | connection. The forward pass evaluates hidden and output activations. The backward pass multiplies local derivatives through the same, unchanged weights. Tap the diagram to enlarge. **Step 1: forward propagation through the hidden neuron.** $$ z_h=0.5(2)+0.1=1.1 $$ $$ h=\operatorname{ReLU}(1.1)=1.1 $$ **Step 2: forward propagation through the output neuron.** $$ z_o=0.8(1.1)+0.2=1.08 $$ $$ \hat y=\operatorname{ReLU}(1.08)=1.08 $$ ReLU is not bounded above by 1, so this output is valid. It is not being interpreted as a probability here. **Step 3: calculate the loss and output delta.** $$ L=\frac12(1.08-1)^2=0.0032 $$ Both preactivations are positive, so both ReLU derivatives are 1. $$ \delta_o=(1.08-1)(1)=0.08 $$ **Step 4: calculate the output-layer parameter gradients.** $$ \frac{\partial L}{\partial w_2}=0.08(1.1)=0.088 $$ $$ \frac{\partial L}{\partial b_2}=0.08 $$ **Step 5: propagate the derivative to the hidden neuron.** Use w₂ = 0.8, its value during the forward pass: $$ \delta_h=0.08(0.8)(1)=0.064 $$ $$ \frac{\partial L}{\partial w_1}=0.064(2)=0.128 $$ $$ \frac{\partial L}{\partial b_1}=0.064 $$ **Step 6: update each parameter using its own gradient.** $$ w_{1,\mathrm{new}}=0.5-0.1(0.128)=0.4872 $$ $$ b_{1,\mathrm{new}}=0.1-0.1(0.064)=0.0936 $$ $$ w_{2,\mathrm{new}}=0.8-0.1(0.088)=0.7912 $$ $$ b_{2,\mathrm{new}}=0.2-0.1(0.08)=0.192 $$ | Parameter | Gradient from the original forward pass | Updated value | | --- | --- | --- | | w₁ | 0.128 | 0.4872 | | b₁ | 0.064 | 0.0936 | | w₂ | 0.088 | 0.7912 | | b₂ | 0.08 | 0.192 | **Final answer:** the original prediction is 1.08 and its loss is 0.0032. The table contains all requested gradients and updated parameters. A new forward pass would use all four new parameters together. ## Drill 13 Full ReLU backpropagation ## Question **Question.** Input x = [1, 2]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_0601bfe757.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.1 | 0.3 | −0.1 | W₂ | 0.4 | 0.5 Given matrices for this question Tap the diagram to enlarge. ![Worked example: all incoming weights and biases](https://deep-learning-study-room.vercel.app/assets/figures/net_drill13.svg) Diagram: Worked example: all incoming weights and biases. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=2 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.2 + x2 × 0.1; bias 0.1 | h2 incoming: x1 × 0.3 + x2 × -0.1; bias 0.2 | o1 incoming: h1 × 0.4 + h2 × 0.5; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: all incoming weights and biases Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 2]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. Target y = [1]. Learning rate η = 0.1. Use half squared loss and update all weights and biases. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.1 \times 2) + (0.1) = 0.5 $$ $$ h_{1} = \max(0,0.5) = 0.5 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (-0.1 \times 2) + (0.2) = 0.3 $$ $$ h_{2} = \max(0,0.3) = 0.3 $$ **Output neuron 1:** $$ z_{o_1} = (0.4 \times 0.5) + (0.5 \times 0.3) + (0.1) = 0.45 $$ $$ o_{1} = \max(0,0.45) = 0.45 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.55]. Squared errors = [0.3025]. Half-sum squared loss = 0.15125; mean squared error = 0.3025. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.45-1) \times 1 = -0.55 $$ The ReLU derivative is 1 here because this preactivation is positive. **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.5**h1 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.275**h1 → o1 Now update this weight Old weight**0.4** → New weight**0.4275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.55)\times(0.5)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.4-0.1(-0.275)\approx 0.4275 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.3**h2 × Receiving delta**−0.55**δ at o1 = Connection gradient**−0.165**h2 → o1 Now update this weight Old weight**0.5** → New weight**0.5165** Subtract η × gradient = **−0.0165**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.55)\times(0.3)\approx -0.165 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.5-0.1(-0.165)\approx 0.5165 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.4** = Path contribution**−0.22** ↓ Add every path contribution**−0.22**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.4)\times(-0.55)\approx -0.22 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.22 $$ $$ \delta_{h_1} = -0.22 \times 1 = -0.22 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.55** × Old outgoing weight**0.5** = Path contribution**−0.275** ↓ Add every path contribution**−0.275**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.5)\times(-0.55)\approx -0.275 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.275 $$ $$ \delta_{h_2} = -0.275 \times 1 = -0.275 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.22**x1 → h1 Now update this weight Old weight**0.2** → New weight**0.222** Subtract η × gradient = **−0.022**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.22)\times(1)\approx -0.22 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.2-0.1(-0.22)\approx 0.222 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**2**x2 × Receiving delta**−0.22**δ at h1 = Connection gradient**−0.44**x2 → h1 Now update this weight Old weight**0.1** → New weight**0.144** Subtract η × gradient = **−0.044**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.22)\times(2)\approx -0.44 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.1-0.1(-0.44)\approx 0.144 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.275**x1 → h2 Now update this weight Old weight**0.3** → New weight**0.3275** Subtract η × gradient = **−0.0275**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.275)\times(1)\approx -0.275 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.3-0.1(-0.275)\approx 0.3275 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**2**x2 × Receiving delta**−0.275**δ at h2 = Connection gradient**−0.55**x2 → h2 Now update this weight Old weight**−0.1** → New weight**−0.045** Subtract η × gradient = **−0.055**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.275)\times(2)\approx -0.55 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=-0.1-0.1(-0.55)\approx -0.045 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.1(-0.220000000)=0.122000000 $$ $$ b_{h_2,\mathrm{new}}=0.2-0.1(-0.275000000)=0.227500000 $$ $$ b_{o_1,\mathrm{new}}=0.1-0.1(-0.550000000)=0.155000000 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 14 Two output sigmoid backpropagation ## Question **Question.** Input x = [0.2, 0.4]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.1]; W₂ is shown below; b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_907c03b5dc.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.1 | 0.2 | −0.2 | 0.3 | W₂ | 0.2 | −0.1 | 0.3 | 0.4 Given matrices for this question Tap the diagram to enlarge. ![Worked example: both output paths must contribute to hidden gradients](https://deep-learning-study-room.vercel.app/assets/figures/net_drill14.svg) Diagram: Worked example: both output paths must contribute to hidden gradients. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=0.2 | x2=0.4 | h1 | h2 | o1 | target 1 | o2 | target 0 | h1 incoming: x1 × 0.1 + x2 × 0.2; bias 0.1 | h2 incoming: x1 × -0.2 + x2 × 0.3; bias 0.1 | o1 incoming: h1 × 0.2 + h2 × -0.1; bias 0 | o2 incoming: h1 × 0.3 + h2 × 0.4; bias 0.1 | Use these OLD connections for all gradients in the same pass. Worked example: both output paths must contribute to hidden gradients Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [0.2, 0.4]. W₁ is printed with the question. b₁ = [0.1, 0.1]. W₂ is printed with the question. b₂ = [0, 0.1]. Hidden and output activations are sigmoid. Target y = [1, 0]. Learning rate η = 0.2. Use half summed squared loss and update all parameters. ### To find Find all preactivations, hidden activations and final outputs. Calculate the half summed squared loss and its output derivatives. Find the sigmoid output and, if requested, its derivative. Find the requested deltas and parameter gradients, then update the specified parameters. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used before substituting numbers For every receiving neuron, z is its weighted sum and a is its activated output. The half squared loss sums the errors from all output neurons. $$ z_j=\sum_iw_{ji}a_i+b_j,\quad a_j=f(z_j),\quad L=\frac12\sum_k(o_k-y_k)^2 $$ Delta means the loss gradient with respect to a neuron's raw score. First calculate output deltas, then hidden deltas using every connected output path. $$ \delta_{o_k}=(o_k-y_k)f'(z_{o_k}) $$ $$ \delta_{h_j}=\left(\sum_kw_{kj}\delta_{o_k}\right)f'(z_{h_j}) $$ The gradient of a connection is its receiving delta multiplied by its sending activation. A bias receives the constant 1. $$ \frac{\partial L}{\partial w_{ji}}=\delta_j a_i,\qquad \frac{\partial L}{\partial b_j}=\delta_j $$ $$ w_{ji,\mathrm{new}}=w_{ji}-\eta\delta_j a_i,\quad b_{j,\mathrm{new}}=b_j-\eta\delta_j $$ For sigmoid, use f′ = a(1 − a). For ReLU, use f′ = 1 for a positive raw score and 0 for a negative raw score. Store the old weights until all gradients have been calculated. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.1 \times 0.2) + (0.2 \times 0.4) + (0.1) = 0.2 $$ $$ h_{1}=\frac{1}{1+e^{-(0.2)}}=0.549833997 $$ **Hidden neuron 2:** $$ z_{h_2} = (-0.2 \times 0.2) + (0.3 \times 0.4) + (0.1) = 0.18 $$ $$ h_{2}=\frac{1}{1+e^{-(0.18)}}=0.544878892 $$ **Output neuron 1:** $$ z_{o_1} = (0.2 \times 0.549833997) + (-0.1 \times 0.544878892) + (0) = 0.05547891 $$ $$ o_{1}=\frac{1}{1+e^{-(0.05547891)}}=0.513866171 $$ **Output neuron 2:** $$ z_{o_2} = (0.3 \times 0.549833997) + (0.4 \times 0.544878892) + (0.1) = 0.482901756 $$ $$ o_{2}=\frac{1}{1+e^{-(0.482901756)}}=0.618432848 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.486133829, 0.618432848]. Squared errors = [0.2363261, 0.382459187]. Half-sum squared loss = 0.309392643; mean squared error = 0.309392643. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.513866171-1) \times 0.249807729 = -0.121439988 $$ The derivative is o(1−o). **Output delta 2:** $$ \delta_{o_2} = 1 \times (0.618432848-0) \times 0.235973661 = 0.145933863 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.549833997**h1 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066771834**h1 → o1 Now update this weight Old weight**0.2** → New weight**0.213354367** Subtract η × gradient = **−0.013354367**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.121439988)\times(0.549833997)\approx -0.066771834 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.2-0.2(-0.066771834)\approx 0.213354367 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.544878892**h2 × Receiving delta**−0.121439988**δ at o1 = Connection gradient**−0.066170086**h2 → o1 Now update this weight Old weight**−0.1** → New weight**−0.086765983** Subtract η × gradient = **−0.013234017**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.121439988)\times(0.544878892)\approx -0.066170086 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=-0.1-0.2(-0.066170086)\approx -0.086765983 $$ **h1 → o2:** The calculation on h1 → o2 Sending activation**0.549833997**h1 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.080239399**h1 → o2 Now update this weight Old weight**0.3** → New weight**0.28395212** Subtract η × gradient = **0.01604788**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,1}}=\delta_{o_2}h_1 $$ $$ g_{2,1}^{(2)}=(0.145933863)\times(0.549833997)\approx 0.080239399 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,1,\mathrm{new}}=0.3-0.2(0.080239399)\approx 0.28395212 $$ **h2 → o2:** The calculation on h2 → o2 Sending activation**0.544878892**h2 × Receiving delta**0.145933863**δ at o2 = Connection gradient**0.079516282**h2 → o2 Now update this weight Old weight**0.4** → New weight**0.384096744** Subtract η × gradient = **0.015903256**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{2,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{2,2}}=\delta_{o_2}h_2 $$ $$ g_{2,2}^{(2)}=(0.145933863)\times(0.544878892)\approx 0.079516282 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{2,2,\mathrm{new}}=0.4-0.2(0.079516282)\approx 0.384096744 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**0.2** = Path contribution**−0.024287998** Output 2 Output delta**0.145933863** × Old outgoing weight**0.3** = Path contribution**0.043780159** ↓ Add every path contribution**0.019492161**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.2)\times(-0.121439988)\approx -0.024287998 $$ **Contribution from output 2.** $$ C_2=(0.3)\times(0.145933863)\approx 0.043780159 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.019492161 $$ $$ \delta_{h_1} = 0.019492161 \times 0.247516573 = 0.004824633 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.121439988** × Old outgoing weight**−0.1** = Path contribution**0.012143999** Output 2 Output delta**0.145933863** × Old outgoing weight**0.4** = Path contribution**0.058373545** ↓ Add every path contribution**0.070517544**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(-0.1)\times(-0.121439988)\approx 0.012143999 $$ **Contribution from output 2.** $$ C_2=(0.4)\times(0.145933863)\approx 0.058373545 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1+C_2\approx 0.070517544 $$ $$ \delta_{h_2} = 0.070517544 \times 0.247985885 = 0.017487356 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**0.2**x1 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.000964927**x1 → h1 Now update this weight Old weight**0.1** → New weight**0.099807015** Subtract η × gradient = **0.000192985**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(0.004824633)\times(0.2)\approx 0.000964927 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.1-0.2(0.000964927)\approx 0.099807015 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**0.4**x2 × Receiving delta**0.004824633**δ at h1 = Connection gradient**0.001929853**x2 → h1 Now update this weight Old weight**0.2** → New weight**0.199614029** Subtract η × gradient = **0.000385971**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(0.004824633)\times(0.4)\approx 0.001929853 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.2-0.2(0.001929853)\approx 0.199614029 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**0.2**x1 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.003497471**x1 → h2 Now update this weight Old weight**−0.2** → New weight**−0.200699494** Subtract η × gradient = **0.000699494**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(0.017487356)\times(0.2)\approx 0.003497471 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=-0.2-0.2(0.003497471)\approx -0.200699494 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**0.4**x2 × Receiving delta**0.017487356**δ at h2 = Connection gradient**0.006994942**x2 → h2 Now update this weight Old weight**0.3** → New weight**0.298601012** Subtract η × gradient = **0.001398988**. This weight **decreases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(0.017487356)\times(0.4)\approx 0.006994942 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.3-0.2(0.006994942)\approx 0.298601012 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0.1-0.2(0.004824633)=0.099035073 $$ $$ b_{h_2,\mathrm{new}}=0.1-0.2(0.017487356)=0.096502529 $$ $$ b_{o_1,\mathrm{new}}=0-0.2(-0.121439988)=0.024287998 $$ $$ b_{o_2,\mathrm{new}}=0.1-0.2(0.145933863)=0.070813227 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. ## Drill 15 A negative ReLU blocks this path ## Question **Question.** x = 2, w1 = −0.4, b1 = 0.1, w2 = 0.5, b2 = 0.2; hidden ReLU, linear output, y = 1, half squared loss. Find dw 1 and dw 2. ## Answer — step by step ### Given x = 2; w₁ = −0.4; b₁ = 0.1; w₂ = 0.5; b₂ = 0.2. Hidden activation: ReLU; output activation: linear. Target y = 1; loss: half squared error. ### To find Find the gradients of w₁ and w₂ and explain any zero gradients. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step ### Formulas used The hidden neuron uses ReLU, but the output is linear. “Linear output” means its weighted sum is already its output. $$ z_h=w_1x+b_1,\quad h=\max(0,z_h),\quad o=w_2h+b_2 $$ $$ L=\frac12(o-y)^2,\quad \delta_o=o-y $$ $$ \frac{\partial L}{\partial w_2}=\delta_o h,\quad \delta_h=\delta_o w_2\operatorname{ReLU}'(z_h),\quad \frac{\partial L}{\partial w_1}=\delta_h x $$ ### Step 1: hidden weighted sum and activation $$ z_h=(-0.4)\times2+0.1=-0.8+0.1=-0.7 $$ $$ h=\max(0,-0.7)=0 $$ Because the raw hidden score is negative, ReLU's derivative at this point is 0. ### Step 2: output prediction and output delta $$ o=0.5\times0+0.2=0.2 $$ $$ \delta_o=0.2-1=-0.8 $$ ### Step 3: output weight gradient $$ \frac{\partial L}{\partial w_2}=(-0.8)\times0=0 $$ This is zero because the input to that connection, h, is zero. ### Step 4: hidden delta and input weight gradient $$ \delta_h=(-0.8)\times0.5\times0=0 $$ $$ \frac{\partial L}{\partial w_1}=0\times2=0 $$ **Final answer:** both requested weight gradients are zero. The output bias gradient equals −0.8, so a zero weight gradient does not mean every parameter's gradient is zero. ## Gradient descent A gradient tells us the local direction in which loss rises. Gradient descent subtracts a small multiple of that gradient. Learning rate η controls the size of this correction. After the first update, calculate the next gradient at the new parameter. $$ \theta_{new}=\theta-\eta g $$ $$ g=\frac{\partial J}{\partial\theta} $$ ## Reading an optimizer formula without getting lost An optimizer updates the numbers the model is allowed to learn. Usually these are weights and biases. The letter θ is a convenient name for whichever parameter is currently being updated. ![Old value, current gradient and new value belong to different moments in the update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: Old value, current gradient and new value belong to different moments in the update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. Old value, current gradient and new value belong to different moments in the update. Tap the diagram to enlarge. ### Old, current and new At update t, θₜ₋₁ is the value before the update and θₜ is the value after it. The symbol gₜ is the loss gradient used in that update. If a question already supplies gₜ, use it. If it supplies a loss J(θ), differentiate J and evaluate the derivative at the stated position. $$ \theta_t=\theta_{t-1}-\eta g_t $$ η, read eta, is the learning rate. It multiplies the gradient to determine the basic correction. A positive gradient is subtracted; a negative gradient makes the new value larger because subtracting a negative is addition. ### What a stored state means An optimizer may retain a number from its previous update. This number is called its state or memory. It is an ordinary calculated number, not an additional neuron. Momentum stores movement; AdaGrad stores a sum of squared gradients; RMSProp stores a moving average of squares; Adam stores a moving average of gradients and another of squares. ### A weighted average from ordinary arithmetic If the old state is 4, the new squared gradient is 9 and the decay is 0.9, retain 90% of the old state and add 10% of the new square. $$ 0.9\times4=3.6,\qquad(1-0.9)\times9=0.9 $$ $$ s_{\mathrm{new}}=3.6+0.9=4.5 $$ The coefficients 0.9 and 0.1 add to 1. As updates continue, a past contribution gets multiplied by 0.9 again, so its influence fades. AdaGrad instead gives 4 + 9 = 13 because it retains the full old sum. ### Squared memory and signed direction For gradient −3, the square is 9. The stored scale is positive, but the update still needs the negative gradient to determine direction. Do not replace every occurrence of g with g². ### Epsilon and hats ε, read epsilon, is a small positive number that prevents an exactly zero denominator. A hat in Adam, such as m with a hat, labels a bias-corrected average. A hat does not mean squaring. The power βᵗ uses the update number t, so its value changes from the first update to the second. ## The ordinary gradient descent update **Definition.** Gradient descent is an iterative first-order optimization algorithm that updates parameters in the negative direction of the current objective gradient, scaled by a positive learning rate. ### Explanation and working $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ Here update numbers start at t = 1: calculate g₁ at θ₀, obtain θ₁, then calculate g₂ at θ₁. The subscript counts completed updates; it is not an exponent. Read it aloud: “new parameter equals old parameter minus learning rate times current gradient.” The old parameter is where you are. The gradient gives the local uphill direction. The learning rate controls how far you move in the opposite direction. ![Calculate the slope first, multiply it by the learning rate, then subtract that correction.](https://deep-learning-study-room.vercel.app/assets/figures/v_gd.svg) Diagram: Calculate the slope first, multiply it by the learning rate, then subtract that correction.. Labels: Move against the slope to reduce local loss | Move against the slope to reduce local loss | Illustration: L = w²; start at w = 2 and use learning rate 0.1. | w | old w = 2 | new w = 1.6 | Gradient = 2w = 4 | Correction = 0.1 × 4 | New w = 2 − 0.4 | A positive gradient is subtracted. A negative gradient is also subtracted, so w | increases. Calculate the slope first, multiply it by the learning rate, then subtract that correction. Tap the diagram to enlarge. **Question.** J(θ) = 0.7 θ³; θ₀ = 1; η = 0.1. Perform two iterations. ### Given J(θ) = 0.7θ³; initial θ₀ = 1; η = 0.1. ### To find Differentiate J and perform two gradient-descent iterations. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step **Step 1. Differentiate.** g(θ) = 0.7 × 3 × θ² = 2.1 θ². **Step 2. First update.** g₁ = 2.1(1)² = 2.1. θ₁ = 1 − 0.1(2.1) = 0.79. **Step 3. Second gradient.** Square the new value: 0.79² = 0.6241. Multiply: g₂ = 2.1 × 0.6241 = 1.31061. **Step 4. Second update.** θ₂ = 0.79 − 0.1(1.31061) = 0.79 − 0.131061 = **0.658939**. The paper calls this SGD. With only a scalar objective supplied, no sample selection is specified; the arithmetic is the stated gradient-descent update. Also, this cubic is not bounded below over all real values. Do not claim these two steps prove convergence to a global minimum. **Common error:** using the loss J itself instead of its derivative g. Another error is repeatedly using θ₀ to calculate later gradients. ## Drill 18 GD variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ \theta_1=1-0.1(1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ \theta_2=0.9-0.1(0.9)=0.81 $$ **Final parameter after two updates: 0.81.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 19 GD variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two GD updates. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas $$ g_t=J\prime(\theta_{t-1}),\quad \theta_t=\theta_{t-1}-\eta g_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ \theta_1=-0.5-0.1(-1.5)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ \theta_2=-0.35-0.1(-1.35)=-0.215 $$ **Final parameter after two updates: -0.215.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 6 Chain rule on a tiny loss ## Question **Question.** Let o = wx+b, L = ½(o−y)², x = 3, w = 0.2, b = 0.1, y = 1. Find dL/dw, dL/db and update at η = 0.1. ## Answer — step by step ### Given Model: o = wx+b; loss L = ½(o−y)². x = 3; y = 1; w = 0.2; b = 0.1. Learning rate η = 0.1. ### To find Find ∂L/∂w, ∂L/∂b and the updated weight and bias. ### Formula — which one and why **Chain rule for a linear neuron.** The weight changes the prediction, and the prediction changes the loss. Multiply those two local rates. Half squared loss differentiates to prediction minus target; the neuron contributes input x. z = prediction; y = target; e = z−y; g\_w and g\_b = loss derivatives for weight and bias. ∂L/∂w means “how fast loss changes when w changes”. η (eta) = learning rate. $$ z=wx+b $$ $$ L=\frac12(z-y)^2 $$ $$ g_w=\frac{\partial L}{\partial w}=(z-y)x $$ $$ g_b=\frac{\partial L}{\partial b}=z-y $$ $$ w_{\mathrm{new}}=w-\eta g_w $$ $$ b_{\mathrm{new}}=b-\eta g_b $$ **Yaad rakhne ka cue:** Ek path mein multiply; alag paths mil rahe hon toh add. ### Solution — step by step ### Meaning and formulas The output o is a linear prediction: no sigmoid or ReLU is specified. Changing w changes o, which changes L. The chain rule multiplies those linked effects. $$ o=wx+b,\qquad L=\frac12(o-y)^2 $$ $$ \frac{\partial L}{\partial o}=o-y,\quad \frac{\partial o}{\partial w}=x,\quad \frac{\partial o}{\partial b}=1 $$ $$ \frac{\partial L}{\partial w}=(o-y)x,\qquad \frac{\partial L}{\partial b}=o-y $$ The learning rate η says how much of the gradient to subtract. $$ w_{\mathrm{new}}=w-\eta\frac{\partial L}{\partial w},\qquad b_{\mathrm{new}}=b-\eta\frac{\partial L}{\partial b} $$ ### Step 1: obtain the current prediction and error $$ o=(0.2\times3)+0.1=0.6+0.1=0.7 $$ $$ o-y=0.7-1=-0.3 $$ $$ L=\frac12(-0.3)^2=\frac12(0.09)=0.045 $$ ### Step 2: calculate both gradients from this same prediction $$ \frac{\partial L}{\partial w}=(-0.3)\times3=-0.9 $$ $$ \frac{\partial L}{\partial b}=(-0.3)\times1=-0.3 $$ ### Step 3: update the weight $$ w_{\mathrm{new}}=0.2-0.1(-0.9)=0.2-(-0.09)=0.29 $$ ### Step 4: update the bias $$ b_{\mathrm{new}}=0.1-0.1(-0.3)=0.1-(-0.03)=0.13 $$ **Final answer:** weight gradient −0.9; bias gradient −0.3; new weight 0.29; new bias 0.13. Both updates use the old forward pass. ## Batch and mini-batch gradient A batch is a group of training examples used together for one update. An epoch is one pass through the dataset. Averaging gradients and counting updates are different questions, so both receive complete worked examples below. $$ g_{batch}=\frac1B\sum_{i=1}^{B}g_i $$ $$ U_{epoch}=\left\lceil\frac NB\right\rceil $$ ## Model answer: batch, stochastic and mini-batch training ## Question **Question.** Differentiate batch gradient descent, stochastic gradient descent and mini-batch gradient descent. Explain an epoch, a batch and an iteration. ## Answer — step by step All three methods update model parameters using gradients. Their difference is the number of training examples used to estimate the gradient for one update. ![All methods visit the same six examples. Their grouping determines how often they update.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: All methods visit the same six examples. Their grouping determines how often they update.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. All methods visit the same six examples. Their grouping determines how often they update. Tap the diagram to enlarge. A **batch** is the group of examples used together for an update. An **iteration**, in this context, is one optimizer update. An **epoch** is one complete pass through the chosen training dataset. One epoch usually contains several iterations. | Basis | Batch GD | Stochastic GD | Mini-batch GD | | --- | --- | --- | --- | | Examples in one gradient | Entire training dataset | One example | A smaller group of B examples | | Updates per epoch | One | N | Ceiling of N/B if the remainder is retained | | Gradient behavior | Full-data gradient | Noisy sample gradient | Intermediate averaging | | Practical implication | Updates can be expensive on large datasets | Many small, sequential updates | Common compromise for vectorized computation | ### Example to distinguish the three counts With 105 examples and batch size 20, five batches use 100 examples and a sixth uses the remaining 5. Therefore one epoch has 6 mini-batch updates. Three epochs give 18 updates. It would be 5 updates per epoch only if the problem explicitly discarded the incomplete batch. The basic parameter update is unchanged: $$ \theta_{\mathrm{new}}=\theta_{\mathrm{old}}-\eta g $$ For batch or mini-batch training, g normally represents the gradient of the stated aggregate loss over that group. For sequential SGD, update after each example and use the new parameter for the next example. Consequently, one SGD epoch need not equal one batch update even when both use the same examples. ## A tiny dataset shows why the methods differ **Question.** Train ŷ = wx on samples (1,2) and (2,4), starting at w = 1, with η = 0.1 and mean half squared loss. Compare one batch update with one SGD epoch in the listed order. ### Given Model ŷ = wx; initial w = 1; η = 0.1. Samples in order: (1,2), (2,4). Mean half squared loss for the batch. ### To find Compare one batch update with a sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### One batch update **Step 1: predict sample 1 with the original weight.** $$ \hat y_1=1(1)=1 $$ **Step 2: calculate its error.** $$ e_1=1-2=-1 $$ **Step 3: multiply its error by its input.** $$ g_1=(-1)(1)=-1 $$ **Step 4: predict sample 2 with the same original weight.** $$ \hat y_2=1(2)=2 $$ **Step 5: calculate its error.** $$ e_2=2-4=-2 $$ **Step 6: multiply its error by its input.** $$ g_2=(-2)(2)=-4 $$ **Step 7: average the two gradients.** $$ g=\frac{-1+(-4)}2=-2.5 $$ **Step 8: update once.** $$ w_{\mathrm{new}}=1-0.1(-2.5)=1.25 $$ ![This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.](https://deep-learning-study-room.vercel.app/assets/figures/v_batches.svg) Diagram: This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation.. Labels: Same 6 samples, different update groups | Same 6 samples, different update groups | Each outlined group produces one gradient update. | Batch GD | 1 | 2 | 3 | 4 | 5 | 6 | 1 update | Mini-batch, size 2 | 1 | 2 | 3 | 4 | 5 | 6 | 3 updates | SGD, size 1 | 1 | 2 | 3 | 4 | 5 | 6 | 6 updates | One epoch visits all six samples. The number of updates depends on the group | size. This grouping sketch explains the difference. The worked dataset below supplies its own values for the actual gradient calculation. Tap the diagram to enlarge. ### One sequential SGD epoch **Step 1: use the first sample gradient.** $$ g_1=-1 $$ **Step 2: update after the first sample.** $$ w_1=1-0.1(-1)=1.1 $$ **Step 3: predict sample 2 using the changed weight.** $$ \hat y_2=1.1(2)=2.2 $$ **Step 4: find sample 2 error.** $$ e_2=2.2-4=-1.8 $$ **Step 5: find the new sample 2 gradient.** $$ g_2=(-1.8)(2)=-3.6 $$ **Step 6: update after sample 2.** $$ w_2=1.1-0.1(-3.6)=1.46 $$ **Final answer:** batch weight = 1.25; sequential SGD weight = 1.46. SGD changes the weight between samples. Larger mini-batches use more memory and often make better use of parallel hardware. Smaller batches provide noisier gradient estimates and more updates per epoch. Wall-clock speed and generalization depend on the implementation and problem, so “SGD always converges fastest” is not a safe claim. ## Drill 16 Batch and sequential SGD ## Question **Question.** ŷ = wx, start w = 0, η = 0.1, data(1,1), (2,2). Use mean half squared loss for a batch. Compare one batch update with one sequential SGD epoch. ## Answer — step by step ### Given Model ŷ = wx; initial w = 0; η = 0.1. Samples in order: (x,y) = (1,1), then (2,2). Batch objective: mean half squared loss. ### To find Compare the weight after one batch update and one sequential SGD epoch. ### Formula — which one and why **Batch mean versus sequential SGD.** A batch update averages gradients evaluated at the same old parameter. Sequential SGD updates after each sample, so later samples use the changed parameter. B = number of examples in the batch; gᵢ = sample gradient; η = learning rate. For the linear half squared objective, sample gradient is (wx−y)x. $$ g_{\mathrm{batch}}=\frac1B\sum_i g_i $$ $$ g_i=(wx_i-y_i)x_i $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Batch: same old weight par average; SGD: har sample ke baad weight change. ### Solution — step by step ### Meaning and formulas For one sample, x is the input, y is the target, and w is the weight. Batch GD averages gradients evaluated at the **same old weight**. Sequential SGD updates after each sample and uses that new weight for the next sample. $$ \hat y=wx,\quad L=\frac12(\hat y-y)^2,\quad g=(\hat y-y)x $$ $$ g_{\mathrm{batch}}=\frac{g_1+g_2}{2},\qquad w_{\mathrm{new}}=w-\eta g $$ **Step 1: Part A: one batch update from w = 0.** For sample (1,1), prediction = 0 × 1 = 0, so: $$ g_1=(0-1)\times1=-1 $$ For sample (2,2), keep the same weight 0. Prediction = 0 × 2 = 0, so: $$ g_2=(0-2)\times2=-4 $$ Average these two gradients, then update once. $$ g_{\mathrm{batch}}=\frac{-1+(-4)}2=-2.5 $$ $$ w_{\mathrm{batch}}=0-0.1(-2.5)=0.25 $$ **Step 2: Part B: one SGD epoch, restarting at w = 0.** The first sample has gradient −1, just as above. $$ w_1=0-0.1(-1)=0.1 $$ For the second sample use this new weight, so prediction is: $$ \hat y_2=0.1\times2=0.2 $$ $$ g_2=(0.2-2)\times2=(-1.8)\times2=-3.6 $$ $$ w_2=0.1-0.1(-3.6)=0.1+0.36=0.46 $$ **Final answer:** batch result 0.25; sequential SGD result 0.46. The difference comes from the timing of the updates. ## Drill 17 Incomplete mini batch ## Question **Question.** N = 103, B = 20, epochs = 3. Count updates for batch GD, SGD, mini-batch keeping the remainder, and mini-batch dropping it. ## Answer — step by step ### Given N = 103 samples; B = 20 samples per mini-batch. E = 3 epochs. Compare both remainder policies: keep or drop the final 3 examples. ### To find Find total updates for full-batch GD, SGD, and both mini-batch policies. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. ### Solution — step by step ### Meaning and formulas An epoch means one pass through the dataset. An update happens once per batch used. N is the number of samples, B is the chosen batch size, and E is the number of epochs. $$ U_{\mathrm{batch}}=E,\qquad U_{\mathrm{SGD}}=NE $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil,\qquad U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ Ceiling rounds up to the next integer; floor rounds down. The final incomplete batch needs an update only if it is retained. ### Step 1: batch GD All 103 samples form one batch per epoch. $$ U_{\mathrm{batch}}=3\times1=3 $$ ### Step 2: SGD Each sample gets its own update. $$ U_{\mathrm{SGD}}=103\times3=309 $$ ### Step 3: mini-batches with the remainder $$ 103=5\times20+3 $$ The batch sizes are 20, 20, 20, 20, 20 and 3. Six batches per epoch give: $$ U_{\mathrm{keep}}=6\times3=18 $$ ### Step 4: mini-batches without the remainder Only the five complete batches are used each epoch. $$ U_{\mathrm{drop}}=5\times3=15 $$ **Final answer:** 3, 309, 18 and 15 updates respectively. ## Momentum: signed velocity Momentum carries some of the previous movement into the next update. Here v is signed movement, so the new parameter is the old parameter plus v. A negative v moves the parameter down. Use the same convention throughout the calculation. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ## Model answer: momentum and Nesterov ## Question **Question.** Explain momentum and Nesterov accelerated gradient. How does the point at which the gradient is measured differ? ## Answer — step by step Momentum retains a fraction of the previous movement and combines it with the current gradient correction. Movement in a persistent direction can build up, while alternating directions can partially cancel. This can reduce zig-zagging and improve progress through poorly scaled regions. ![Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Nesterov changes where the gradient is measured: at the look-ahead point implied by momentum. Tap the diagram to enlarge. ### Momentum formulas This answer uses signed displacement velocity v: a negative velocity moves the parameter left. η is learning rate and β is the fraction of previous velocity retained. $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ ### Nesterov's change Nesterov first forms a lookahead position using the old momentum. It measures the gradient there, then computes a new velocity and updates the actual old parameter. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look}}) $$ The velocity and parameter update formulas remain as displayed above. Add the new velocity to θ at the previous iteration, not to the lookahead point. | Basis | Momentum | Nesterov | | --- | --- | --- | | Gradient evaluated at | Current parameter | Lookahead parameter | | Information retained | Previous velocity | Previous velocity | | Intuition | Continue useful movement | Inspect the slope where old momentum points | | Initial zero velocity | No movement contribution yet | First lookahead equals current parameter | Both require suitable learning rate and momentum settings. Neither guarantees the global minimum of a general neural-network objective. Different books use different definitions of stored velocity, so the definition and update must be used together consistently. ## Momentum remembers movement **Definition.** Momentum is a gradient-based optimization method that combines the current gradient correction with a fraction of the previous parameter displacement. It can accumulate movement along persistent directions and damp some oscillations across alternating directions. ### Update equations and convention This course uses **signed displacement velocity** v. It already includes the learning rate, and the parameter update adds it to the old parameter: $$ g_t=J'(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ θ is the parameter, η is the learning rate and β controls how much previous displacement is retained, normally with 0 ≤ β < 1. A negative v moves θ in the negative direction. Every parameter has its own velocity; this optimizer state is distinct from the network's trainable parameters. ![Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.](https://deep-learning-study-room.vercel.app/assets/figures/v_momentum.svg) Diagram: Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter.. Labels: Momentum remembers movement | Momentum remembers movement | Use signed velocity: v = βv\_old − ηg; then w\_new = w\_old + v. | Keep movement | β × old velocity | Add correction | Subtract η × gradient | Move weight | Add the new velocity | The memory carries direction. It is not a sum of squared gradients. Combine the retained displacement with the current gradient correction, then apply the resulting velocity to the old parameter. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1, β = 0.9. Find two iterations using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Derivative used:** differentiating J gives J′(θ) = θ. The gradient must therefore be reevaluated at the current θ on each iteration. **Step 1: Iteration 1: gradient, velocity, parameter.** $$ g_1=J'(1)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Iteration 2: use θ₁ and v₁.** $$ g_2=J'(0.9)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ | Iteration | Parameter where gradient is evaluated | Gradient | New velocity | New parameter | | --- | --- | --- | --- | --- | | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.9 | −0.18 | 0.72 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.72. At iteration 2, the retained displacement contributes −0.09 and the new gradient correction contributes another −0.09. Their sum explains the larger displacement despite the smaller current gradient. ### Properties and alternative notation Momentum can improve progress through long, poorly scaled valleys, but excessive learning rate or momentum can overshoot. It does not guarantee a global minimum for a general neural-network objective. Some texts store a positive gradient accumulator u instead: $$ u_t=\beta u_{t-1}+g_t,\qquad\theta_t=\theta_{t-1}-\eta u_t $$ With constant η and consistent initialization, this corresponds to vₜ = −ηuₜ. A normalized moving average containing (1−β)gₜ is a different scaling convention. Use the exact recurrence stated in the question instead of combining pieces of different conventions. ## Drill 20 Momentum variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ g_2=0.9-(0)=0.9 $$ $$ v_2=0.9(-0.1)-0.1(0.9)=-0.18 $$ $$ \theta_2=0.9+(-0.18)=0.72 $$ **Final parameter after two updates: 0.72.** The second gradient is evaluated again using the parameter obtained after the first update. ## Drill 21 Momentum variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Momentum updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ g_t=J\prime(\theta_{t-1}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ g_2=-0.35-(1)=-1.35 $$ $$ v_2=0.9(0.15)-0.1(-1.35)=0.27 $$ $$ \theta_2=-0.35+(0.27)=-0.08 $$ **Final parameter after two updates: -0.08.** The second gradient is evaluated again using the parameter obtained after the first update. ## Nesterov: look-ahead gradient Nesterov first uses the old movement to choose a temporary look-ahead position. It measures the gradient there, calculates the new movement, then adds that movement to the original parameter. The temporary look-ahead position does not replace the original parameter in the final addition. $$ \theta_{look}=\theta+\beta v $$ $$ v_{new}=\beta v-\eta g(\theta_{look}) $$ $$ \theta_{new}=\theta+v_{new} $$ ## Nesterov looks before correcting **Definition.** Nesterov accelerated gradient is a momentum-based method that evaluates the gradient at a lookahead parameter, formed using the previous velocity, before calculating the new velocity and actual parameter update. ### What changes relative to momentum Ordinary momentum evaluates J′ at θₜ₋₁. Nesterov evaluates it after the anticipated displacement βvₜ₋₁. This changes the point where the objective's slope is measured: $$ \theta_{\mathrm{look},t}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J'(\theta_{\mathrm{look},t}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ The lookahead point is an auxiliary evaluation point, not the new actual parameter. In this signed-displacement convention, the new velocity is added to θₜ₋₁. ![Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.](https://deep-learning-study-room.vercel.app/assets/figures/v_nesterov.svg) Diagram: Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update.. Labels: Nesterov checks the slope at the look-ahead point | Nesterov checks the slope at the look-ahead point | Use the signed-velocity convention shown in the chapter. | Current w | Look-ahead | Updated w | Move by βv\_old | Correct using gradient here | Look-ahead = w + βv\_old | Nesterov measures the gradient after the look-ahead, before choosing its final | correction. Locate the lookahead point using the old velocity; measure the gradient there; use that gradient to calculate the actual update. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** J(θ) = ½θ², θ₀ = 1, v₀ = 0, η = 0.1 and β = 0.9. Perform two Nesterov updates using signed displacement velocity. ### Given J(θ) = ½θ²; θ₀ = 1; v₀ = 0. η = 0.1; β = 0.9; signed-displacement velocity. ### To find Find the gradient, velocity and actual parameter after each of two iterations. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step The derivative is J′(θ) = θ. Therefore the gradient equals the lookahead value in this particular objective. **Iteration 1 — step 1: locate the lookahead point.** $$ \theta_{\mathrm{look},1}=1+0.9(0)=1 $$ **Step 2: evaluate the gradient there.** $$ g_1=J'(1)=1 $$ **Step 3: calculate the new velocity and actual parameter.** $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Iteration 2 — step 1: use the previous actual parameter and velocity.** $$ \theta_{\mathrm{look},2}=0.9+0.9(-0.1)=0.81 $$ **Step 2: evaluate the gradient at 0.81, not at 0.9.** $$ g_2=J'(0.81)=0.81 $$ **Step 3: calculate the new velocity.** $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ **Step 4: update the old actual parameter.** $$ \theta_2=0.9+(-0.171)=0.729 $$ | Iteration | Old actual θ | Lookahead θ | Gradient | New v | New actual θ | | --- | --- | --- | --- | --- | --- | | 1 | 1 | 1 | 1 | −0.1 | 0.9 | | 2 | 0.9 | 0.81 | 0.81 | −0.171 | 0.729 | **Final answer:** θ₁ = 0.9 and θ₂ = 0.729. The first update matches ordinary momentum because v₀ = 0. At the second update, ordinary momentum would use gradient 0.9 and produce θ₂ = 0.72. Adding v₂ to 0.81 instead of 0.9 would count the old momentum contribution twice. ## Drill 22 Nesterov variation 1 ## Question **Question.** J(θ) = ½θ², θ₀ = 1, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½θ². Initial θ₀ = 1; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-0)^2,\qquad J\prime(\theta)=\theta-0 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 0. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=1+0.9(0)=1 $$ $$ g_1=1-(0)=1 $$ $$ v_1=0.9(0)-0.1(1)=-0.1 $$ $$ \theta_1=1+(-0.1)=0.9 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=0.9+0.9(-0.1)=0.81 $$ $$ g_2=0.81-(0)=0.81 $$ $$ v_2=0.9(-0.1)-0.1(0.81)=-0.171 $$ $$ \theta_2=0.9+(-0.171)=0.729 $$ **Final parameter after two updates: 0.729.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. ## Drill 23 Nesterov variation 2 ## Question **Question.** J(θ) = ½(θ−1)², θ₀ = -0.5, η = 0.1. Perform two Nesterov updates. Use β = 0.9, v₀ = 0 and signed velocity. ## Answer — step by step ### Given Objective: J(θ) = ½(θ−1)². Initial θ₀ = −0.5; η = 0.1; two iterations. β = 0.9; initial signed velocity v₀ = 0. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Nesterov lookahead.** First look ahead using the old velocity. Measure the gradient there, form the new velocity, then add that velocity to the original current parameter. θ\_look = temporary gradient-evaluation point; v = signed displacement; β = momentum coefficient; η = learning rate. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1} $$ $$ g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle look ahead; gradient wahan; update original position se. ### Solution — step by step ### Meaning of the symbols θ (theta) is the number we update. η (eta) is learning rate 0.1. g is the derivative of the objective, evaluated at the required position. A subscript counts updates; θ₀ means the starting value. ### Differentiate the objective first $$ J(\theta)=\frac12(\theta-1)^2,\qquad J\prime(\theta)=\theta-1 $$ The power rule brings down 2, which cancels the half. The inside derivative is 1, so the remaining gradient is θ minus 1. ### Update formulas v stores the signed movement from the previous update. β = 0.9 retains 90% of that movement. Start v₀ = 0. $$ \theta_{\mathrm{look}}=\theta_{t-1}+\beta v_{t-1},\quad g_t=J\prime(\theta_{\mathrm{look}}) $$ $$ v_t=\beta v_{t-1}-\eta g_t,\quad \theta_t=\theta_{t-1}+v_t $$ **Step 1: Update 1.** $$ \theta_{\mathrm{look}}=-0.5+0.9(0)=-0.5 $$ $$ g_1=-0.5-(1)=-1.5 $$ $$ v_1=0.9(0)-0.1(-1.5)=0.15 $$ $$ \theta_1=-0.5+(0.15)=-0.35 $$ **Step 2: Update 2.** $$ \theta_{\mathrm{look}}=-0.35+0.9(0.15)=-0.215 $$ $$ g_2=-0.215-(1)=-1.215 $$ $$ v_2=0.9(0.15)-0.1(-1.215)=0.2565 $$ $$ \theta_2=-0.35+(0.2565)=-0.0935 $$ **Final parameter after two updates: -0.0935.** The lookahead locates the gradient; the new velocity is added to the old actual parameter, not to the lookahead. ## AdaGrad AdaGrad stores a running sum of squared gradients for each parameter. Squaring removes the sign only in this stored total. The unsquared gradient in the numerator still decides the direction of the update. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ ## AdaGrad adds every squared gradient **Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator. ### State, effective learning rate and update For one parameter θ, let gₜ be its current gradient and s₀ = 0: $$ s_t=s_{t-1}+g_t^2 $$ $$ \eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon} $$ $$ \theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t $$ s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign. ![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg) Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update. The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge. ### Complete numerical: supplied gradients **Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter. **Iteration 1 — step 1: update the accumulator.** $$ s_1=0+2^2=4 $$ **Step 2: calculate the effective learning rate.** $$ \eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05 $$ **Step 3: apply the signed gradient.** $$ \theta_1=1-0.05(2)=0.9 $$ **Iteration 2 — step 1: retain s₁ and add the new squared gradient.** $$ s_2=4+1^2=5 $$ **Step 2: calculate the new coefficient.** $$ \eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360 $$ **Step 3: update from θ₁.** $$ \theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640 $$ **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5. ### Advantage, limitation and epsilon convention Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes. ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical. ## Drill 24 AdaGrad variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_1=0+(2)^2=4 $$ $$ \sqrt{G_1}=\sqrt{4}=2 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** Add the new squared gradient to all previous squares. $$ G_2=4+(1)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=0.9-\frac{0.1\times(1)}{\sqrt{5}}=0.85527864 $$ ## Drill 25 AdaGrad variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use AdaGrad for two updates, zero initial accumulators and ε = 0 for this hand calculation. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. G is a running sum of squared gradients. Squaring removes the sign only inside G; keep the gradient sign in the update numerator. Start G₀ = 0. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Add the new squared gradient to all previous squares. $$ G_1=0+(-1)^2=1 $$ $$ \sqrt{G_1}=\sqrt{1}=1 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** Add the new squared gradient to all previous squares. $$ G_2=1+(2)^2=5 $$ $$ \sqrt{G_2}=\sqrt{5}=2.236067977 $$ $$ \theta_2=-0.4-\frac{0.1\times(2)}{\sqrt{5}}=-0.489442719 $$ ## RMSProp RMSProp remembers a weighted average of squared gradients. With decay 0.9, keep 90% of the old stored value and add 10% of the new squared gradient. Calculate this new stored value before taking its square root and updating the parameter. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ ## A moving average remembers the recent past **Definition.** An exponentially weighted moving average (EWMA) is a recursively updated estimate in which older observations receive exponentially decreasing weights. Optimizers use it to summarize recent gradients or squared gradients. $$ m_t=\beta m_{t-1}+(1-\beta)g_t,\qquad0\le\beta<1 $$ For m₀ = 0, expanding the recurrence gives: $$ m_t=(1-\beta)\sum_{k=1}^{t}\beta^{t-k}g_k $$ The most recent gradient has coefficient 1 − β; a gradient k steps older is discounted by an additional factor βᵏ. Larger β smooths more strongly but responds more slowly to a change in direction. The heuristic averaging horizon is approximately 1/(1 − β); it is not a hard cutoff in the stored history. ### Signed gradients versus squared gradients An average of g contains directional information: positive and negative contributions can cancel. An average of g² estimates the scale through an uncentered second moment and cannot become negative when initialized nonnegatively. Adam tracks both quantities. RMSProp tracks a squared-gradient average in its basic form. **Question.** Let β = 0.9, m₀ = 0 and gradients g₁ = 2, g₂ = −1. Find m₁ and m₂. ### Given β = 0.9; initial m₀ = 0. Gradients in order: g₁ = 2, g₂ = −1. ### To find Find m₁ and m₂, preserving the sign of each gradient. ### Formula — which one and why **Exponential moving average.** Keep the specified fraction of the old state and add the complementary fraction of the new gradient. This tracks signed direction, so negative gradients can reduce the state. β (beta) = retained fraction; m = stored average; g = current gradient. m₀ is the initial state. $$ m_t=\beta m_{t-1}+(1-\beta)g_t $$ **Yaad rakhne ka cue:** Beta purani memory rakhta hai; 1−beta naya gradient leta hai. ### Solution — step by step **Step 1: find the recent-gradient coefficient.** $$ 1-0.9=0.1 $$ **Step 2: retain the old memory for step 1.** $$ 0.9(0)=0 $$ **Step 3: weight the first gradient.** $$ 0.1(2)=0.2 $$ **Step 4: add to obtain the first memory.** $$ m_1=0+0.2=0.2 $$ **Step 5: retain the first memory for step 2.** $$ 0.9(0.2)=0.18 $$ **Step 6: weight the second gradient.** $$ 0.1(-1)=-0.1 $$ **Step 7: add to obtain the second memory.** $$ m_2=0.18+(-0.1)=0.08 $$ **Final answer:** m₁ = 0.2; m₂ = 0.08. The second average remains positive although the latest gradient is negative because the previous positive contribution still has weight. If the recurrence is applied to the squared gradients instead, the second state is 0.9(0.4) + 0.1(1) = 0.46. These states have different meanings and are not interchangeable. ## RMSProp remembers recent squared gradients **Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator. ### Recurrence and interpretation Using decay β and zero initial state: $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update. If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever. ![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg) Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient. Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β = 0.9. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step **Iteration 1 — step 1: calculate the squared-gradient average.** $$ s_1=0.9(0)+0.1(2^2)=0.4 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_1}=\sqrt{0.4}\approx0.632455532 $$ $$ \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766 $$ **Step 3: update the parameter.** $$ \theta_1=1-0.316227766\approx0.683772234 $$ **Iteration 2 — step 1: use s₁, not the original zero state.** $$ s_2=0.9(0.4)+0.1(1^2)=0.46 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_2}=\sqrt{0.46}\approx0.678232998 $$ $$ \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956 $$ **Step 3: update from θ₁, retaining calculator precision.** $$ \theta_2\approx0.683772234-0.147441956\approx0.536330278 $$ **Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46. ### Comparison with AdaGrad and Adam AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum. ## Drill 26 RMSProp variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(2)^2=0.4 $$ $$ \sqrt{s_1}=\sqrt{0.4}=0.632455532 $$ $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{0.4}}=0.683772234 $$ **Step 2: Iteration 2 with supplied gradient 1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.4)+0.1(1)^2=0.46 $$ $$ \sqrt{s_2}=\sqrt{0.46}=0.678232998 $$ $$ \theta_2=0.683772234-\frac{0.1\times(1)}{\sqrt{0.46}}=0.536330278 $$ ## Drill 27 RMSProp variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use RMSProp for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use ρ = 0.9. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. ρ = 0.9. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. s is a moving average of squared gradients. ρ = 0.9 retains 90% of the old state; 1 − ρ = 0.1 supplies 10% of the new square. Start s₀ = 0. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_1=0.9(0)+0.1(-1)^2=0.1 $$ $$ \sqrt{s_1}=\sqrt{0.1}=0.316227766 $$ $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{0.1}}=-0.183772234 $$ **Step 2: Iteration 2 with supplied gradient 2.** Keep 90 percent of the old squared-gradient memory; add 10 percent of the new square. $$ s_2=0.9(0.1)+0.1(2)^2=0.49 $$ $$ \sqrt{s_2}=\sqrt{0.49}=0.7 $$ $$ \theta_2=-0.183772234-\frac{0.1\times(2)}{\sqrt{0.49}}=-0.46948652 $$ ## Adam: both memories Adam keeps two separate running averages. The first remembers signed gradients and can be negative. The second remembers squared gradients and is nonnegative. These stored averages are not weights, and they are not extra gradients to multiply together. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ ## Optimizer notation: old value, memory, new value ### 1. A parameter and a stored state are different θ is the parameter we want to learn. A memory such as v, s or m helps an optimizer decide how to change θ; it is not a second prediction. Each trainable parameter usually has its own corresponding memory entries. ![The time label tells you which value to use. Carry the previous memory into the current update.](https://deep-learning-study-room.vercel.app/assets/figures/v_optimizer_notation.svg) Diagram: The time label tells you which value to use. Carry the previous memory into the current update.. Labels: Read one update from old values to new values | Read one update from old values to new values | The subscript is a step label, not multiplication. | OLD | w at step t − 1 | CURRENT | gradient g at step t | NEW | w at step t | An optimizer may also carry a memory from the previous step. The time label tells you which value to use. Carry the previous memory into the current update. Tap the diagram to enlarge. At time t = 0, the problem supplies starting values. At update t = 1, read the old values, calculate the current gradient and new memory, then calculate the new parameter. At update t = 2, use the results of update 1 as the old values. ### 2. A supplied gradient versus a supplied objective If the question says gradients are 2 and −1, use those values in order. If it instead gives J(θ) = θ², first differentiate to get g(θ) = 2θ. Recalculate g using the current parameter at each iteration. Nesterov evaluates at its lookahead point instead. For θ₀ = 1 and η = 0.1 with ordinary GD, g₁ = 2 and θ₁ = 0.8. The next gradient is 2 × 0.8 = 1.6, not the old value 2. Therefore θ₂ = 0.8 − 0.1 × 1.6 = 0.64. ### 3. A squared gradient keeps size and loses sign Both 2² and (−2)² equal 4. Squared-gradient memory tells us about scale. The signed numerator still decides the update direction. Never replace the gradient in every part of an optimizer by its square. ### 4. Square root and epsilon belong to the denominator If s = 4 and ε = 0.01, the denominator √s + ε is 2 + 0.01 = 2.01. It is different from √(s + ε), which is √4.01. Use the placement specified by the formula. Small ε keeps a zero accumulator from causing division by zero. ### 5. A table prevents mixing iterations Write one row per iteration. Ordinary GD needs current θ, gradient and new θ. Momentum adds old and new velocity. Adam adds both memories and their corrected values. Complete a row before starting the next; otherwise a new value from iteration 2 may accidentally be used inside iteration 1. The worked examples below use this exact order, with every state shown. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 28 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 29 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ ## Adam: correction and update Adam starts its two averages at zero. Early averages therefore contain less total weight on observed gradients. Bias correction compensates for that initialization. The power in βᵗ uses the current update number, and both averages must be corrected before the parameter update. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \widehat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon} $$ ## Adam bias correction: why the denominator is one minus beta to a power **Definition.** Bias correction compensates for the reduced total observation weight caused by initializing an exponentially weighted accumulator to zero. Under a stationary gradient expectation, it removes the corresponding initialization bias in the moment estimate. ### Explanation and working ### 1. What the moving average actually contains Adam's signed-gradient memory is m. Starting with m₀ = 0 and β = 0.9, the first average is 90% of zero plus 10% of the first gradient. If g₁ = 2, then m₁ = 0.2. That small value partly reflects the artificial zero initialization. ![The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam_correction.svg) Diagram: The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor.. Labels: Correct the early zero-initialized memory | Correct the early zero-initialized memory | Example: β₁ = 0.9, first gradient = 2, initial m = 0. | Raw memory | m₁ = 0.9 × 0 + 0.1 × | 2 = 0.2 | Correction factor | 1 − 0.9¹ = 0.1 | Corrected memory | m̂₁ = 0.2 ÷ 0.1 = 2 | The small raw value is partly caused by the starting zero. Bias correction | removes that effect. The first raw average is pulled toward its starting zero. Divide by the missing-weight correction factor. Tap the diagram to enlarge. $$ m_1=(1-\beta)g_1 $$ At the second step, substitute the first expression into the recurrence: $$ m_2=\beta(1-\beta)g_1+(1-\beta)g_2 $$ ### 2. Add the coefficients After the first step, the total coefficient attached to observed gradients is 1 − β. After two steps it is β(1 − β) + (1 − β). Factor out the common term: $$ \beta(1-\beta)+(1-\beta)=(1-\beta)(1+\beta)=1-\beta^2 $$ After t steps the coefficients sum to 1 − βᵗ. Dividing the stored average by that quantity removes this zero-start coefficient deficit. $$ \widehat m_t=\frac{m_t}{1-\beta_1^t} $$ The squared-gradient memory uses its own decay β₂ and the same reasoning: $$ \widehat s_t=\frac{s_t}{1-\beta_2^t} $$ ### 3. A constant-gradient example makes the purpose visible Let both gradients be 2 and use β₁ = 0.9. The first raw mean is 0.2; the second is 0.9 × 0.2 + 0.1 × 2 = 0.38. $$ \widehat m_1=\frac{0.2}{1-0.9}=2 $$ $$ \widehat m_2=\frac{0.38}{1-0.9^2}=\frac{0.38}{0.19}=2 $$ Both corrected values now equal the constant observed gradient. If gradients differ over time, the corrected result is a weighted average, not necessarily the latest gradient. ### 4. Two betas mean two separate powers At update 2 with β₁ = 0.9 and β₂ = 0.999, the first correction denominator is 1 − 0.9² = 0.19. The second is 1 − 0.999² = 0.001999. Do not use 0.19 for both memories, and do not keep using the first-step denominators. ### 5. Correct first, then combine The parameter update uses the corrected signed mean divided by the square root of the corrected squared mean, plus ε in the stated position. The square root is taken after correcting s. The numerator is not the loss, and no extra multiplication by g follows. This derivation explains the formula; the following solved Adam examples show the complete arithmetic when gradients change sign or magnitude. ## Adam remembers both direction and squared size **Definition.** Adam, or adaptive moment estimation, uses bias-corrected exponentially weighted estimates of the first moment and the uncentered second moment of gradients to calculate parameter-specific updates. ### What the two states represent | State or coefficient | Meaning | | --- | --- | | mₜ | Moving average of signed gradients; retains directional information | | sₜ | Moving average of squared gradients; estimates their uncentered second moment | | β₁ | Decay coefficient for the first-moment estimate | | β₂ | Decay coefficient for the second-moment estimate | | t | Update number, starting at 1 for the first update | | η and ε | Base learning rate and small denominator stabilizer | The second moment is not the variance: it averages g² without subtracting a mean. Both states start at zero unless specified otherwise. These notes use s throughout this lesson; texts that use v for Adam's second moment mean the same state, not momentum's displacement velocity. ![The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.](https://deep-learning-study-room.vercel.app/assets/figures/v_adam.svg) Diagram: The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update.. Labels: Adam keeps two different memories | Adam keeps two different memories | One describes signed direction. The other describes squared size. | Current gradient g | Example: g = −2 | First moment m | Average of signed gradients | Second moment s | Average of squared gradients | Correct both → divide m̂ by (√ŝ + ε) → update | For g = −2: the direction contribution is negative; the squared-size contribution | uses 4. The current gradient feeds two distinct estimates. Correct both estimates before combining direction and scale in the parameter update. Tap the diagram to enlarge. ### Recurrences, bias correction and update $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ Zero initialization reduces the total weight initially attached to observed gradients. Divide by the appropriate accumulated observation weight: $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ The update uses the corrected signed average in the numerator and the root of the corrected squared-gradient average in the denominator: $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\varepsilon} $$ The derivation in the next lesson explains why the correction factors contain powers of the iteration number. No additional multiplication by gₜ belongs in this final update: its contribution is already inside mₜ. ### Complete numerical: changing gradient magnitude **Question.** θ₀ = 1, m₀ = s₀ = 0, η = 0.1, β₁ = 0.9, β₂ = 0.999. Given g₁ = 2 and g₂ = 1, find two Adam updates, neglecting epsilon only in the numerical arithmetic. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step **Iteration 1 — step 1: update the two raw states.** $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ s_1=0.999(0)+0.001(2^2)=0.004 $$ **Step 2: correct both states using t = 1.** $$ \hat m_1=\frac{0.2}{1-0.9^1}=\frac{0.2}{0.1}=2 $$ $$ \hat s_1=\frac{0.004}{1-0.999^1}=\frac{0.004}{0.001}=4 $$ **Step 3: update the parameter.** $$ \theta_1=1-\frac{0.1(2)}{\sqrt4}=0.9 $$ Adam iteration 1: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**2**g1 Previous raw m**0** Previous raw s**0** Signed average m1**0.2** ↓ Bias-corrected m̂1**2** Squared-gradient average s1**0.004** ↓ Bias-corrected ŝ1**4** ↓ Combine: ηm̂ / √ŝ**0.1**ε is neglected for this specific numerical. Old θ**1** → New θ**0.9** **Carry forward raw states:** m = 0.2, s = 0.004. The corrected values are used for this update. **Iteration 2 — step 1: carry forward the raw states m₁ and s₁.** The recurrence uses the raw states 0.2 and 0.004, not the corrected values 2 and 4. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ s_2=0.999(0.004)+0.001(1^2)=0.004996 $$ **Step 2: correct both states using t = 2.** $$ 1-0.9^2=0.19,\qquad1-0.999^2=0.001999 $$ $$ \hat m_2=\frac{0.28}{0.19}\approx1.473684211 $$ $$ \hat s_2=\frac{0.004996}{0.001999}\approx2.499249625 $$ **Step 3: divide the corrected first moment by the corrected root.** $$ \frac{\eta\hat m_2}{\sqrt{\hat s_2}}\approx\frac{0.1(1.473684211)}{\sqrt{2.499249625}}\approx0.093217964 $$ **Step 4: update from θ₁.** $$ \theta_2\approx0.9-0.093217964\approx0.806782036 $$ Adam iteration 2: follow the two state calculations Start with the current gradient and the two states saved from the previous iteration. Current gradient**1**g2 Previous raw m**0.2** Previous raw s**0.004** Signed average m2**0.28** ↓ Bias-corrected m̂2**1.473684211** Squared-gradient average s2**0.004996** ↓ Bias-corrected ŝ2**2.499249625** ↓ Combine: ηm̂ / √ŝ**0.093217964**ε is neglected for this specific numerical. Old θ**0.9** → New θ**0.806782036** **Carry forward raw states:** m = 0.28, s = 0.004996. The corrected values are used for this update. | State | After iteration 1 | After iteration 2 | | --- | --- | --- | | Raw first moment m | 0.2 | 0.28 | | Raw second moment s | 0.004 | 0.004996 | | Corrected first moment | 2 | 1.473684211 | | Corrected second moment | 4 | 2.499249625 | | Parameter θ | 0.9 | 0.806782036 | **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.806782. Store the raw moments for the next recurrence and use the corrected moments for the current parameter update. ### Properties and limitations Adam combines directional averaging with coordinate-specific scaling, which can be useful with noisy or differently scaled gradients. It requires two state values per parameter in addition to the parameter itself. η, β₁, β₂ and ε remain hyperparameters. Faster reduction of training loss does not establish better generalization, and Adam does not guarantee the global optimum of a general neural-network objective. ## Drill 28 Adam variation 1 ## Question **Question.** θ₀ = 1, η = 0.1, supplied successive gradients = [2, 1]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = 1; η = 0.1. Supplied gradients: [2,1]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(2)=0.2 $$ $$ v_1=0.999(0)+0.001(2)^2=0.004 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.004}{1-0.999^{1}}=4 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ **Step 2: Iteration 2 with supplied gradient 1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(0.2)+0.1(1)=0.28 $$ $$ v_2=0.999(0.004)+0.001(1)^2=0.004996 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.28}{1-0.9^{2}}=1.473684211 $$ $$ \widehat v_2=\frac{0.004996}{1-0.999^{2}}=2.499249625 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=0.9-\frac{0.1\times(1.473684211)}{\sqrt{2.499249625}}=0.806782036 $$ ## Drill 29 Adam variation 2 ## Question **Question.** θ₀ = -0.5, η = 0.1, supplied successive gradients = [-1, 2]. Use Adam for two updates, zero initial accumulators and ε = 0 for this hand calculation. Use β₁ = 0.9, β₂ = 0.999 and bias correction. ## Answer — step by step ### Given Initial θ₀ = −0.5; η = 0.1. Supplied gradients: [−1,2]. Initial accumulator(s) = 0; ε = 0 for this nonzero-denominator hand calculation. β₁ = 0.9; β₂ = 0.999; use bias correction. ### To find Find every stored state needed by the method and the parameter after each of two updates. ### Formula — which one and why **Adam with bias correction.** Keep two states: one averages signed gradients, the other averages squared gradients. Correct their initial bias toward zero before combining them in the parameter update. m = signed average; s (also written v) = squared average; β₁ and β₂ = retained fractions; t = iteration number. A hat marks a corrected value. Carry raw m and s into the next recurrence. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ s_t=\beta_2s_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t} $$ $$ \hat s_t=\frac{s_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Direction memory + square memory; hats lagao, phir update. ### Solution — step by step ### Symbols and formulas used θ is the parameter; η is the base learning rate; gₜ is the supplied gradient for update t. The gradients are already given, so no loss differentiation is needed here. Use ε = 0 only as specified in this exercise. m remembers signed gradients; v remembers squared gradients. The hats mean bias-corrected values, compensating for memories starting at zero. They are not extra powers. Here β₁ = 0.9 and β₂ = 0.999. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad \hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Use the supplied gradients in order. Initialize the stated memories to zero. Epsilon is set to zero only for this exercise because every denominator is nonzero. Keep calculator precision until the final result. **Step 1: Iteration 1 with supplied gradient -1.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_1=0.9(0)+0.1(-1)=-0.1 $$ $$ v_1=0.999(0)+0.001(-1)^2=0.001 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_1=\frac{-0.1}{1-0.9^{1}}=-1 $$ $$ \widehat v_1=\frac{0.001}{1-0.999^{1}}=1 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_1=-0.5-\frac{0.1\times(-1)}{\sqrt{1}}=-0.4 $$ **Step 2: Iteration 2 with supplied gradient 2.** First calculate the signed-gradient mean m and squared-gradient mean v. $$ m_2=0.9(-0.1)+0.1(2)=0.11 $$ $$ v_2=0.999(0.001)+0.001(2)^2=0.004999 $$ Correct each zero-initialized mean using this iteration number in the power. $$ \widehat m_2=\frac{0.11}{1-0.9^{2}}=0.578947368 $$ $$ \widehat v_2=\frac{0.004999}{1-0.999^{2}}=2.500750375 $$ Divide the corrected direction by the corrected root, then subtract the scaled step. $$ \theta_2=-0.4-\frac{0.1\times(0.578947368)}{\sqrt{2.500750375}}=-0.436610353 $$ ## L1 penalty and update L1 adds a cost based on the absolute sizes of weights. Its derivative is positive for a positive weight and negative for a negative weight. The optimizer adds this penalty gradient to the data gradient before applying the usual update. $$ J=L+\lambda\sum_j|w_j| $$ $$ w_{new}=w-\eta\left[g_{data}+\lambda\operatorname{sign}(w)\right] $$ ## Model answer: regularization and the L1–L2 difference ## Question **Question.** What is regularization? Compare L1 and L2 regularization, including the objective, gradient effect and influence of regularization strength. ## Answer — step by step Regularization restricts or discourages certain fitted solutions to improve generalization. Weight penalties add a cost for parameter size to the training objective. The optimizer must balance fitting the data against that extra cost. ![Follow the stated penalty convention, including lambda and any half factor, when you form the objective.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: Follow the stated penalty convention, including lambda and any half factor, when you form the objective.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. Follow the stated penalty convention, including lambda and any half factor, when you form the objective. Tap the diagram to enlarge. ### Objectives used in this course $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Jdata measures prediction error. λ controls the strength of the additional penalty. The sums are over the weights included by the stated question. | Basis | L1 | L2 | | --- | --- | --- | | Quantity penalized | Absolute weight magnitude | Squared weight magnitude | | Penalty derivative at nonzero w | λ sign(w) | 2λw under the displayed convention | | Typical effect | Encourages sparse solutions | Smoothly shrinks weight magnitudes | | Zero weights | Can promote exact zeros with suitable optimization | Usually leaves small nonzero weights | The total gradient is the data-loss gradient plus the penalty gradient. With w = 2, data gradient 0.3 and λ = 0.1, the displayed L2 penalty adds 2 × 0.1 × 2 = 0.4. The total gradient is 0.7; update using that total. Increasing λ strengthens the preference for smaller weights. Excessive regularization can prevent learning useful patterns and cause underfitting. L1 is not differentiable at zero, where a subgradient or suitable optimization method is used. If the examination writes λ/2 before the L2 sum, its penalty gradient becomes λw. Derive the factor from the printed objective. ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 31 L1 update on positive and negative weights ## Question **Question.** w = [−0.5,0.3], data gradients = [0.2,−0.1], λ = 0.1, η = 0.2. Use L1 and ordinary gradient descent away from zero. ## Answer — step by step ### Given Old weights = [−0.5,0.3]. Data gradients = [0.2,−0.1]; λ = 0.1; η = 0.2. ### To find Find total gradients and the two updated weights. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formula The supplied data gradient is only one part of the total gradient. L1 adds λ times the sign of the old weight. Sign is −1 for a negative weight and +1 for a positive weight. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda\operatorname{sign}(w_j) $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ Both weights here are nonzero, so the ambiguous derivative of absolute value at zero is not needed. **Step 1: Weight 1: old value −0.5.** $$ \lambda\operatorname{sign}(-0.5)=0.1(-1)=-0.1 $$ $$ g_{1,\mathrm{total}}=0.2+(-0.1)=0.1 $$ $$ w_{1,\mathrm{new}}=-0.5-0.2(0.1)=-0.5-0.02=-0.52 $$ **Step 2: Weight 2: old value 0.3.** $$ \lambda\operatorname{sign}(0.3)=0.1(1)=0.1 $$ $$ g_{2,\mathrm{total}}=-0.1+0.1=0 $$ $$ w_{2,\mathrm{new}}=0.3-0.2(0)=0.3 $$ **Final answer: [−0.52, 0.3].** L1's own contribution pulls toward zero, but the combined data-plus-penalty update can move a weight away from zero. ## Drill 30 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 47 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. ## L2 penalty and update with a half L2 adds a cost based on squared weights. Differentiating a square produces a factor of 2. If the objective includes a half, that half cancels the 2; if it does not, the 2 remains. Read the objective before choosing the gradient. $$ J=L+\frac\lambda2\sum_jw_j^2 $$ $$ w_{new}=w-\eta(g_{data}+\lambda w) $$ ## L1 and L2 add a cost for large weights **Definition.** Regularization modifies a training objective or learning procedure to constrain the fitted model and improve generalization. L1 and L2 regularization add parameter-norm penalties to the data loss. ### Explanation and working $$ J_{L1}=J_{\mathrm{data}}+\lambda\sum_i|w_i|,\qquad J_{L2}=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ Lambda controls how strongly you discourage weight size. L1 encourages sparsity, meaning many weights can become exactly zero under suitable optimization. L2 shrinks weights smoothly and usually leaves small nonzero values. Larger lambda can reduce overfitting, but excessive regularization can underfit. ![The same two weights produce different costs because one rule uses absolute values and the other uses squares.](https://deep-learning-study-room.vercel.app/assets/figures/v_penalties.svg) Diagram: The same two weights produce different costs because one rule uses absolute values and the other uses squares.. Labels: A penalty adds a preference for smaller weights | A penalty adds a preference for smaller weights | Example weights: −2 and 1. The loss convention sets any λ or ½ factors. | L1: absolute values | |−2| + |1| = 2 + 1 = 3 | Linear cost with magnitude | Can promote sparse weights. | L2: squared values | (−2)² + 1² = 4 + 1 = 5 | Bigger weights cost more | Smooth shrinkage toward zero. | The regularized objective adds the data loss and the stated penalty. The same two weights produce different costs because one rule uses absolute values and the other uses squares. Tap the diagram to enlarge. **Question.** Data loss = 0.5; weights [2, −1]; λ = 0.1. Compute L1-regularized and L2-regularized objectives using the displayed convention. ### Given Data loss = 0.5; weights = [2,−1]; λ = 0.1. The L2 convention here has no factor ½. ### To find Find L1/L2 penalties and the two regularized objective values. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty without a factor of one-half.** This stated convention multiplies the sum of squares by λ, so differentiating contributes 2λw. Do not use the half-scaled gradient for this objective. λ = penalty strength; J\_data = original data loss; wᵢ = weight i. $$ J=J_{\mathrm{data}}+\lambda\sum_iw_i^2 $$ $$ \frac{\partial J}{\partial w_i}=g_{\mathrm{data},i}+2\lambda w_i $$ **Yaad rakhne ka cue:** Half nahi diya toh L2 derivative mein 2 lambda × weight. ### Solution — step by step **Step 1: take absolute values for L1.** $$ |2|=2,\qquad|-1|=1 $$ **Step 2: add the absolute values.** $$ 2+1=3 $$ **Step 3: multiply by lambda.** $$ 0.1(3)=0.3 $$ **Step 4: add the data loss.** $$ J_{L1}=0.5+0.3=0.8 $$ **Step 5: square both weights for L2.** $$ 2^2=4,\qquad(-1)^2=1 $$ **Step 6: add the squares.** $$ 4+1=5 $$ **Step 7: multiply by lambda without adding a half.** $$ 0.1(5)=0.5 $$ **Step 8: add the data loss.** $$ J_{L2}=0.5+0.5=1.0 $$ For this L2 convention, the additional gradient is 2 λw. If the objective uses λ/2 times the squared sum, the additional gradient is λw instead. The factor is decided by the stated objective. ### L2 gradient update without a half factor **Question. A weight is 2, its data-loss gradient is 0.3, λ = 0.1 and η = 0.01. The objective adds λw². Perform one update.** ### Given w = 2; data gradient = 0.3; λ = 0.1; η = 0.01. The penalty has no factor ½. ### To find Penalty gradient, total gradient and updated weight. ### Formula — which one and why The derivative of λw² is 2λw. Add it to the data gradient before applying gradient descent. λ is penalty strength; η is learning rate; g is the full objective gradient. $$ g_{\mathrm{penalty}}=2\lambda w $$ $$ g=g_{\mathrm{data}}+g_{\mathrm{penalty}} $$ $$ w_{\mathrm{new}}=w-\eta g $$ **Yaad rakhne ka cue:** Penalty ka gradient data gradient mein jodo, phir update. ### Solution — step by step **Step 1: calculate the penalty gradient.** $$ 2(0.1)(2)=0.4 $$ **Step 2: add the data gradient.** $$ g=0.3+0.4=0.7 $$ **Step 3: scale by the learning rate.** $$ \eta g=0.01(0.7)=0.007 $$ **Step 4: subtract from the old weight.** $$ w_{\mathrm{new}}=2-0.007=1.993 $$ **Final answer:** Updated weight = 1.993. Bias penalties are omitted unless the objective includes them. ### Deriving the L2 update Under the objective written above, the data gradient and penalty gradient add: $$ \nabla_w J=\nabla_w J_{\mathrm{data}}+2\lambda w $$ For ordinary gradient descent, regrouping the update gives: $$ w_{t+1}=w_t-\eta(\nabla_w J_{\mathrm{data}}+2\lambda w_t)=(1-2\eta\lambda)w_t-\eta\nabla_w J_{\mathrm{data}} $$ This shows the shrinkage associated with this L2 convention. For an objective containing λ‖w‖²/2, replace 2λ by λ. This algebra explains the equivalence to appropriately scaled weight decay for ordinary gradient descent; adding an L2 term inside an adaptive optimizer is generally different from decoupled weight decay. ## Drill 32 L2 update ## Question **Question.** w = [0.5,−0.2], data gradients = [0.1,0.3], λ = 0.2, η = 0.1. Use(λ/2)Σw². ## Answer — step by step ### Given Old weights = [0.5,−0.2]; data gradients = [0.1,0.3]. λ = 0.2; η = 0.1; penalty = (λ/2)Σw². ### To find Find penalty gradients, total gradients and updated weights. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Formula and its reason The derivative of (λ/2)w² is λw: the 2 from the square cancels the half. Add that regularization gradient to the given data gradient before updating. $$ g_{j,\mathrm{total}}=g_{j,\mathrm{data}}+\lambda w_j $$ $$ w_{j,\mathrm{new}}=w_j-\eta g_{j,\mathrm{total}} $$ **Step 1: Weight 1.** $$ \lambda w_1=0.2\times0.5=0.1 $$ $$ g_{1,\mathrm{total}}=0.1+0.1=0.2 $$ $$ w_{1,\mathrm{new}}=0.5-0.1(0.2)=0.48 $$ **Step 2: Weight 2.** $$ \lambda w_2=0.2\times(-0.2)=-0.04 $$ $$ g_{2,\mathrm{total}}=0.3+(-0.04)=0.26 $$ $$ w_{2,\mathrm{new}}=-0.2-0.1(0.26)=-0.226 $$ **Final answer: [0.48, −0.226].** The second data gradient dominates the regularization term, explaining why that negative weight becomes more negative. ## Drill 30 Regularized objective ## Question **Question.** Data loss = 0.4, w = [−2,1,0.5], λ = 0.1. Calculate L1 objective and L2 objective using (λ/2)Σw². ## Answer — step by step ### Given Data loss = 0.4; weights = [−2,1,0.5]; λ = 0.1. For L2, the question includes the factor ½. ### To find Find each penalty and the total L1/L2 objective. ### Formula — which one and why **L1 penalty and gradient.** Add absolute weight magnitudes to penalize large weights. Away from zero, the penalty derivative depends on the weight’s sign. λ (lambda) = penalty strength; J\_data = data loss; sign(w) is +1 for positive and −1 for negative weights. At zero a subgradient convention is needed. $$ J=J_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ g_i=g_{\mathrm{data},i}+\lambda\operatorname{sign}(w_i) $$ **Yaad rakhne ka cue:** L1 absolute values leta hai; gradient mein sign aata hai. **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. ### Solution — step by step ### Meaning and formulas The data loss measures prediction error. A regularization penalty adds a cost for large weights. The objective J is the total being minimized. λ sets the strength of the penalty. $$ J_{L1}=L_{\mathrm{data}}+\lambda\sum_i|w_i| $$ $$ J_{L2}=L_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ Absolute value means ignore the sign. Squaring means multiply the number by itself. These are different operations. **Step 1: Part A: L1 objective.** $$ |-2|=2,\quad |1|=1,\quad |0.5|=0.5 $$ $$ \sum_i|w_i|=2+1+0.5=3.5 $$ $$ \text{L1 penalty}=0.1\times3.5=0.35 $$ $$ J_{L1}=0.4+0.35=0.75 $$ **Step 2: Part B: L2 objective with the given half.** $$ (-2)^2=4,\quad 1^2=1,\quad 0.5^2=0.25 $$ $$ \sum_iw_i^2=4+1+0.25=5.25 $$ $$ \frac\lambda2=\frac{0.1}2=0.05 $$ $$ \text{L2 penalty}=0.05\times5.25=0.2625 $$ $$ J_{L2}=0.4+0.2625=0.6625 $$ **Final answer:** L1 objective 0.75; L2 objective 0.6625. These are total objectives, not just penalty values. ## Drill 47 Mixed backpropagation and L2 ## Question **Question.** Single linear neuron ŷ = wx, x = 2, w = 0.4, y = 1. Objective J = ½(ŷ−y)²+(λ/2)w², λ = 0.1, η = 0.2. Find loss, gradient, new weight. ## Answer — step by step ### Given Prediction ŷ = wx; x = 2; w = 0.4; target y = 1. Data loss = ½(ŷ−y)²; penalty = (λ/2)w². λ = 0.1; η = 0.2. ### To find Find total objective, its weight gradient and the new weight. ### Formula — which one and why **L2 penalty with a factor of one-half.** Square each weight, sum and multiply by λ/2. The one-half cancels the factor 2 when differentiating, so the extra gradient is λw. λ = penalty strength; w = old weight; J\_data and g\_data = data loss and its gradient. $$ J=J_{\mathrm{data}}+\frac\lambda2\sum_iw_i^2 $$ $$ g_i=g_{\mathrm{data},i}+\lambda w_i $$ $$ w_i^{\mathrm{new}}=w_i-\eta g_i $$ **Yaad rakhne ka cue:** Half wala L2 derivative mein lambda × weight deta hai. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas This objective contains both prediction error and a weight penalty. Its gradient also has two parts. Compute both at the old weight, add them, then update once. $$ \hat y=wx,\qquad J=\frac12(\hat y-y)^2+\frac\lambda2w^2 $$ $$ \frac{\partial J}{\partial w}=(\hat y-y)x+\lambda w,\qquad w_{\mathrm{new}}=w-\eta\frac{\partial J}{\partial w} $$ ### Step 1: prediction and error $$ \hat y=0.4\times2=0.8,\qquad \hat y-y=0.8-1=-0.2 $$ ### Step 2: data loss $$ L_{\mathrm{data}}=\frac12(-0.2)^2=\frac12(0.04)=0.02 $$ ### Step 3: regularization penalty and total objective $$ R=\frac{0.1}{2}(0.4)^2=0.05(0.16)=0.008 $$ $$ J=0.02+0.008=0.028 $$ ### Step 4: calculate the two gradient contributions $$ g_{\mathrm{data}}=(-0.2)\times2=-0.4 $$ $$ g_{\mathrm{penalty}}=0.1\times0.4=0.04 $$ $$ g_{\mathrm{total}}=-0.4+0.04=-0.36 $$ ### Step 5: update $$ w_{\mathrm{new}}=0.4-0.2(-0.36)=0.4+0.072=0.472 $$ **Final answer:** total loss 0.028; gradient −0.36; new weight 0.472. ## Inverted dropout Inverted dropout uses a mask: zero means drop this activation, one means keep it. During training, divide each retained activation by the keep probability. During inference, dropout is disabled and this training-time division is not repeated. $$ q=1-p $$ $$ a_{train}=\frac{m\odot a}{q} $$ $$ a_{test}=a $$ ## Model answer: dropout with training and inference explained ## Question **Question.** Explain dropout as a regularization technique. Distinguish its behavior during training and inference, and explain inverted-dropout scaling. ## Answer — step by step Dropout randomly replaces selected activations by zero during training. It discourages excessive reliance on particular units and their joint activity. The mask changes between training applications; it does not permanently remove neurons or stored weights. ![The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout_expectation.svg) Diagram: The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask.. Labels: Why inverted dropout divides a kept value by q | Why inverted dropout divides a kept value by q | The text uses activation 3 and keep probability q = 0.5. | If kept: probability 0.5 | Training output = 3 ÷ 0.5 = 6 | Weighted contribution = 3 | If dropped: probability 0.5 | Training output = 0 | Weighted contribution = 0 | Average over masks: 0.5 × 6 + 0.5 × 0 = 3. At inference, use the ordinary value | 3. The mask temporarily keeps or drops this activation during training. Inference uses the ordinary value without the random mask. Tap the diagram to enlarge. Let p be the probability of dropping an activation and q = 1 − p the probability of keeping it. A mask value m = 1 means keep; m = 0 means drop. ### Inverted dropout during training $$ a'=\frac{ma}{q} $$ A kept activation is divided by q and a dropped activation becomes zero. This preserves the activation's expectation over masks. For a = 3 and q = 0.5, the kept result is 3/0.5 = 6 and the dropped result is 0. Their probability-weighted average is: $$ 0.5(6)+0.5(0)=3 $$ ### Inference Disable dropout and use the ordinary activation. Under inverted dropout there is no further division or multiplication by q at inference. Under the alternative classical convention, training does not rescale kept activations and inference includes the compensating scaling; do not combine the two conventions. Dropout has no trainable mask parameters. A dropped path contributes zero current data gradient through that activation, although a separately specified weight penalty can still contribute a parameter gradient. Excessive dropout can make optimization or fitting difficult. Choose its rate using validation data. ## Dropout removes activations during training **Definition.** Dropout is a stochastic regularization method that randomly sets selected unit activations to zero during training. It discourages reliance on specific co-adapted activations; the inference behavior depends on the scaling convention used during training. ### Explanation and working Dropout randomly sets selected activations to zero during training. This discourages dependence on one particular set of neurons. A dropped activation is not a permanently deleted neuron, and dropout does not reduce the model’s stored parameter count. ![Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.](https://deep-learning-study-room.vercel.app/assets/figures/v_dropout.svg) Diagram: Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability.. Labels: Dropout temporarily removes selected activations | Dropout temporarily removes selected activations | Example: keep probability q = 0.5. Inverted dropout during training. | Original | 0.8 | -0.4 | 0.6 | 0.2 | Mask | 1 | 0 | 1 | 0 | After mask ÷ q | 1.6 | 0 | 1.2 | 0 | The kept values double: 0.8 ÷ 0.5 = 1.6 and 0.6 ÷ 0.5 = 1.2. At inference, no | mask is used. Match each activation with its mask entry. Zero entries are removed; kept entries are divided by the keep probability. Tap the diagram to enlarge. Let drop probability be p and keep probability be q = 1 − p. A mask entry 1 means keep, 0 means drop. There are two internally consistent scaling conventions. $$ a_{\mathrm{train}}=m\odot a\quad\text{and}\quad a_{\mathrm{test}}=q a\quad\text{in classical dropout} $$ $$ a_{\mathrm{train}}=\frac{m\odot a}{q}\quad\text{and}\quad a_{\mathrm{test}}=a\quad\text{in inverted dropout} $$ **Question.** Activations [0.8, −0.4, 0.6, 0.2] have dropout rate 50%. Choose mask [1,0,1,0] and compute outputs. ### Given Activations = [0.8,−0.4,0.6,0.2]. Dropout rate = 50%, so keep probability q = 0.5. Chosen mask = [1,0,1,0]. ### To find Find training outputs under inverted dropout. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step **Step 1.** q = 1 − 0.5 = 0.5. Apply mask elementwise: [1 × 0.8, 0 × (−0.4), 1 × 0.6, 0 × 0.2] = [0.8,0,0.6,0]. This is the unscaled training output, used by the supplied re-exam key. **Step 2.** For inverted dropout, divide the kept outputs by q: [0.8/0.5,0,0.6/0.5,0] = **[1.6,0,1.2,0]**. Inference then uses the full network without that extra scaling. State the convention in an answer. Random dropout does not require exactly half the units to disappear in every small layer; each unit is kept or dropped according to the sampling rule. This mask is one possible outcome. If an exam explicitly specifies exactly two dropped units, follow that condition. ## Drill 33 Inverted dropout ## Question **Question.** Activations = [2,−1,0,3], keep probability 0.75, mask = [1,0,1,1]. Find training outputs and explain inference. ## Answer — step by step ### Given Activations = [2,−1,0,3]. Keep probability q = 0.75; keep mask = [1,0,1,1]. ### To find Find training outputs and state the inference rule. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. ### Solution — step by step ### Meaning and formula The mask entry m is 1 when an activation is kept and 0 when dropped. The keep probability q is 0.75. Inverted dropout divides the kept activations by q during training so that their expected value is preserved. $$ a'_j=\frac{m_j a_j}{q} $$ The multiplication is entry by entry; there is no dot-product sum. ### Step 1: apply the mask $$ [1(2),\;0(-1),\;1(0),\;1(3)]=[2,0,0,3] $$ ### Step 2: divide each result by 0.75 $$ a'_1=\frac2{0.75}=2.666666667 $$ $$ a'_2=\frac0{0.75}=0,\qquad a'_3=\frac0{0.75}=0 $$ $$ a'_4=\frac3{0.75}=4 $$ **Training output = [2.666666667, 0, 0, 4].** A kept activation that was already zero stays zero. ### During inference Disable the random mask and the division. If the incoming activations are the same [2,−1,0,3], they pass through unchanged. The scaling has already been accounted for during training. ## Drill 48 Mixed dropout and a momentum update ## Question **Question.** Hidden activations [1,2], inverted dropout q = 0.5, mask [1,0]. Linear output weights [0.2,0.3], bias 0, target 1, half squared loss. Update only output weights with momentum β = 0.9, old velocity [0.1,−0.1], η = 0.1. ## Answer — step by step ### Given Hidden activations = [1,2]; mask = [1,0]; keep probability q = 0.5. Linear-output weights = [0.2,0.3]; bias = 0; target = 1. Half squared loss; update only the output weights. Old signed velocity = [0.1,−0.1]; β = 0.9; η = 0.1. ### To find Find dropped/scaled activations, prediction, gradients, new velocities and new output weights. ### Formula — which one and why **Inverted dropout.** Multiply by the keep/drop mask, then divide retained activations by the keep probability. This preserves each activation’s expectation over random masks. aᵢ = original activation; mᵢ is 1 to keep or 0 to drop; q = keep probability = 1−dropout rate. Disable this masking/scaling at inference. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ $$ a_i^{\mathrm{test}}=a_i $$ **Yaad rakhne ka cue:** Mask se keep/drop; kept value ko keep probability se divide. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step ### Formulas used Dropout changes the inputs received by the output neuron. Use those changed inputs in both its forward calculation and its weight gradients. $$ a'_j=\frac{m_ja_j}{q},\quad o=\sum_jw_ja'_j+b,\quad g_j=(o-y)a'_j $$ The signed-velocity momentum convention is: $$ v_{j,\mathrm{new}}=\beta v_{j,\mathrm{old}}-\eta g_j,\qquad w_{j,\mathrm{new}}=w_j+v_{j,\mathrm{new}} $$ ### Step 1: apply inverted dropout $$ a'_1=\frac{1\times1}{0.5}=2,\qquad a'_2=\frac{0\times2}{0.5}=0 $$ ### Step 2: prediction and error $$ o=0.2(2)+0.3(0)+0=0.4 $$ $$ o-y=0.4-1=-0.6 $$ ### Step 3: current weight gradients $$ g_1=(-0.6)(2)=-1.2,\qquad g_2=(-0.6)(0)=0 $$ ### Step 4: remembered movement plus new correction $$ v_{1,\mathrm{new}}=0.9(0.1)-0.1(-1.2)=0.09+0.12=0.21 $$ $$ v_{2,\mathrm{new}}=0.9(-0.1)-0.1(0)=-0.09 $$ ### Step 5: add velocities to old weights $$ w_{1,\mathrm{new}}=0.2+0.21=0.41 $$ $$ w_{2,\mathrm{new}}=0.3-0.09=0.21 $$ **Final answer: [0.41, 0.21].** The second weight moves even with zero current gradient because its old momentum is nonzero. ## Convolution output dimensions This formula counts how many legal filter positions fit along one image dimension. Padding adds space at both ends, the kernel occupies K positions, and stride controls how far the next window moves. The floor keeps only complete valid positions. $$ N_{out}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ ## Reading image shapes before doing CNN arithmetic **Definition.** An image or intermediate CNN representation is a tensor indexed by spatial position and channel. This course writes its shape as height × width × channels, excluding the batch dimension. ### Explanation and working ### Height, width and channels The shape 4 × 5 × 3 means 4 rows, 5 columns and 3 channels. It contains 4 × 5 × 3 = 60 numbers. A grayscale image commonly has one intensity channel; an RGB image has red, green and blue channels. Intermediate CNN channels are learned feature maps, not necessarily colors. ![Count height, width and channels separately. Three channels are stacked at the same spatial positions.](https://deep-learning-study-room.vercel.app/assets/figures/v_image_shape.svg) Diagram: Count height, width and channels separately. Three channels are stacked at the same spatial positions.. Labels: Read height, width and channels separately | Read height, width and channels separately | The text uses 4 × 5 × 3: four rows, five columns and three channels. | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 1 | 2 | 3 | 4 | 5 | 3 stacked channels | H × W × C | H = 4 rows | W = 5 columns | C = 3 channels | Total entries = 60 | A filter spans the incoming channels. Its spatial width is a different quantity. Count height, width and channels separately. Three channels are stacked at the same spatial positions. Tap the diagram to enlarge. ### One position versus the whole map A filter placed at one location computes one output value. Sliding that same filter over all valid locations creates one feature map. A second filter creates a second feature map. Therefore the number of filters equals the number of output channels. ### Why filter depth matches input channels A 3 × 3 filter applied to a 3-channel input has 3 × 3 × 3 = 27 weights. At one location, it combines 27 input values into one sum. A single bias is then added for that filter. It does not produce three separate output channels just because the input has three channels. ### Reading a complete shape calculation The spatial-size formula computes height and width. It does not compute the channel count. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N = current input size; P = padding on one side; K = filter size; S = stride. The floor symbol means keep only complete placements. Add one because the initial placement is also an output position. ### Shape, parameters and computation are three different counts **Shape** counts the arrangement of output values. **Parameters** count learned weights and biases. **Computation** counts how many arithmetic operations are performed. Reusing the same filter at 100 positions creates 100 outputs but does not create 100 new sets of weights. In a multi-layer question, write the output shape of a layer first. That becomes the next layer's input shape. This is why the second convolution uses the first convolution's channel count, not the original image's channel count. ## Stride padding and floor **Definition.** Stride is the spacing between consecutive kernel positions. Padding extends the input boundary. Together with kernel size and dilation, they determine the spatial dimensions of a convolutional output. ### Explanation and working Stride is how far the filter moves each time. Stride 2 visits starting positions 0, 2, 4 and so on. Padding adds border values, usually zeros. Padding 1 means one new row or column on each side, adding 2 to a spatial dimension. ![The border adds to both sides of the input. The stride changes the next starting position of the filter window.](https://deep-learning-study-room.vercel.app/assets/figures/v_stride_padding.svg) Diagram: The border adds to both sides of the input. The stride changes the next starting position of the filter window.. Labels: Padding adds a border. Stride moves the window. | Padding adds a border. Stride moves the window. | Example: a 3 × 3 input becomes 5 × 5 after one zero border. | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 | 3 | 0 | 0 | 4 | 5 | 6 | 0 | 0 | 7 | 8 | 9 | 0 | 0 | 0 | 0 | 0 | 0 | Padding P = 1 | 0 | 1 | 2 | 3 | 4 | Stride S = 2 | Starting positions: 0, 2, … | For each spatial dimension: add 2P, subtract K, divide by S, take floor, then add | 1. The border adds to both sides of the input. The stride changes the next starting position of the filter window. Tap the diagram to enlarge. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input size; P is padding per side; K is kernel size; S is stride. Floor means take the greatest integer not above the result: $\left\lfloor 2.5\right\rfloor$ = 2. It counts only positions where the entire window fits. The formula assumes dilation 1, the standard setting in these papers. Apply it separately to height and width for rectangular images or kernels. **Why plus one?** Even when the kernel exactly fits the image, there is one placement. If N = K, P = 0, S = 1, the formula is 0 + 1 = 1. ### Output dimension with a fractional intermediate result **Question. A convolution has input side N = 6, kernel side K = 3, stride S = 2 and padding P = 0. Find its output side.** ### Given N = 6; K = 3; S = 2; P = 0; dilation = 1. ### To find The integer output side length. ### Formula — which one and why The formula counts complete kernel placements. Flooring removes an incomplete final placement. N is input side; P is padding on each side; K is kernel side; S is stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Padding dono side; stride se divide; floor; phir +1. ### Solution — step by step **Step 1: include padding and subtract kernel size.** $$ 6+2(0)-3=3 $$ **Step 2: divide by the stride.** $$ \frac32=1.5 $$ **Step 3: take the floor.** $$ \lfloor1.5\rfloor=1 $$ **Step 4: count the first placement.** $$ N_{\mathrm{out}}=1+1=2 $$ **Final answer:** Output side = 2; a square input gives a 2 × 2 map per filter. ### Padding on both sides **Question. A square convolution input has side 5, kernel side 3, stride 1 and padding 1 on each side. Find the padded-input size and output size.** ### Given N = 5; K = 3; S = 1; P = 1 on each side. ### To find Padded input and output dimensions. ### Formula — which one and why Padding adds P entries on each of two opposite sides; the output counts complete kernel placements. N = unpadded side; K = kernel side; S = stride; P = per-side padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\lfloor(N+2P-K)/S\rfloor+1 $$ **Yaad rakhne ka cue:** Padded input aur output ka size alag cheez hain. ### Solution — step by step **Step 1: count both padding edges.** $$ 2P=2(1)=2 $$ **Step 2: calculate the padded input side.** $$ N+2P=5+2=7 $$ **Step 3: subtract kernel width.** $$ 7-3=4 $$ **Step 4: divide by stride and take the floor.** $$ \lfloor4/1\rfloor=4 $$ **Step 5: include the first placement.** $$ N_{out}=4+1=5 $$ **Final answer:** Padded input = 7 × 7; output = 5 × 5 per filter. “Valid” usually means P = 0. For odd K and S = 1, padding (K − 1)/2 preserves the spatial size. For S greater than 1, software “same” padding can be asymmetric and commonly gives ceiling(N/S); do not force the stride-1 rule into that case. ## Even-sized kernels: SAME padding is not always symmetric **Definition.** A numerical padding value specifies how many entries to add at each boundary. SAME is an output-size convention. At stride 1, SAME preserves height and width; an even-sized kernel can require unequal padding on opposite sides. For a 2 by 2 kernel at stride 1, preserving the input size requires **one padding entry in total per axis**, not one on each side. For example, add no row above and one row below, and no column on the left and one on the right. Reversing which side receives the extra entry preserves the output size too, but changes which input cells contribute at each output location; follow the question's convention. **Hinglish cue:** P = 1 means dono taraf ek; SAME means output ka size bachao. Dono instructions ko same mat samjho. ### Example 1: one entry on every side **Question. A 5 × 5 single-channel input is processed by one 2 × 2 convolution filter. The stride is 1. Exactly one zero is padded on each of the four sides. Find the padded-input size and output size. (Practice: 3 marks)** ### Given Input height and width N = 5; kernel size K = 2; stride S = 1; padding P = 1 on each side; dilation = 1; one output filter. ### To find The padded input dimensions and the convolution output dimensions. ### Formula — which one and why The two opposite borders contribute P entries each. The output formula counts complete filter placements after padding. $$ N_{\mathrm{padded}}=N+2P $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is one input dimension; P is padding on each side; K is kernel size; S is stride. Apply the same calculation separately to height and width. One filter produces one output channel. ### Solution — step by step **Step 1: count the two padding borders.** $$ 2P=2(1)=2 $$ **Step 2: find the padded side length.** $$ N_{\mathrm{padded}}=5+2=7 $$ The padded input is 7 × 7 × 1. **Step 3: subtract the kernel size.** $$ 7-2=5 $$ **Step 4: divide by stride, take the floor and count the first placement.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5}{1}\right\rfloor+1=5+1=6 $$ **Step 5: write height, width and channels.** Both spatial dimensions equal 6. There is one filter, so there is one output channel. **Final answer:** padded input = **7 × 7 × 1**; output = **6 × 6 × 1**. This explicitly specified padding does not implement SAME for this even kernel. ### Example 2: preserve the size with SAME padding **Question. The same 5 × 5 single-channel input is processed by one 2 × 2 convolution filter with stride 1 and SAME padding. Find the total padding required along each axis and the output size. Use zero padding above and to the left, with any required extra padding below and to the right. (Practice: 3 marks)** ### Given Input side N = 5; kernel side K = 2; stride S = 1; desired output side Nout = 5; dilation = 1; one filter. Padding before the input is 0; padding after the input must be found. ### To find Padding above, below, left and right; padded-input shape; final output shape. ### Formula — which one and why For unequal padding, replace 2P by the sum of the before and after padding. Pbefore and Pafter are the entries added on the two opposite sides of one axis. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+P_{\mathrm{before}}+P_{\mathrm{after}}-K}{S}\right\rfloor+1 $$ At stride 1, the quantities are integers. To keep Nout = N, the total padding must equal K − 1. $$ P_{\mathrm{total}}=P_{\mathrm{before}}+P_{\mathrm{after}}=K-1 $$ ### Solution — step by step **Step 1: calculate total padding along the height.** $$ P_{\mathrm{total}}=2-1=1 $$ Only one extra row in total is needed. **Step 2: place the row using the stated convention.** Padding above = 0; padding below = 1. **Step 3: apply the same calculation to width.** Only one extra column in total is needed. Padding left = 0; padding right = 1. **Step 4: find the padded-input shape.** $$ H_{\mathrm{padded}}=5+0+1=6 $$ $$ W_{\mathrm{padded}}=5+0+1=6 $$ The padded input is 6 × 6 × 1. **Step 5: count the output positions along each axis.** $$ N_{\mathrm{out}}=\left\lfloor\frac{5+0+1-2}{1}\right\rfloor+1=4+1=5 $$ **Step 6: include the output channel.** One filter creates one output channel. **Final answer:** add **0 above, 1 below, 0 left and 1 right**; padded input = **6 × 6 × 1**; output = **5 × 5 × 1**. The two questions have the same input and kernel but different padding instructions. Read the stated boundary padding before applying the formula. ## Drill 35 Padding stride and channels ## Question **Question.** Input 31 × 31 × 3; 5 × 5 convolution,8 filters, stride 2, padding 1. Find shape, weights, biases, total. ## Answer — step by step ### Given Input shape = 31 × 31 × 3. Filter size = 5 × 5; number of filters = 8. Stride S = 2; padding P = 1 per side; one bias per filter. ### To find Find output shape, weights, biases and total parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning of each given value The input has height 31, width 31 and 3 channels. K = 5 is the filter width/height; S = 2 is the stride; P = 1 is padding on each side. Eight filters create eight output channels. ### Formulas used $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ \text{weights}=K^2C_{\mathrm{in}}C_{\mathrm{out}},\quad \text{biases}=C_{\mathrm{out}} $$ $$ \text{parameters}=\text{weights}+\text{biases} $$ ### Step 1: find the output height and width Padding adds one cell on each side, so the padded width is 31 + 2 × 1 = 33. $$ N_{\mathrm{out}}=\left\lfloor\frac{31+2(1)-5}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\left\lfloor\frac{28}{2}\right\rfloor+1=14+1=15 $$ The height uses the same calculation. **Output shape = 15 × 15 × 8.** ### Step 2: count weights in one filter Each filter covers 5 × 5 positions in each of 3 channels. $$ 5\times5\times3=75\text{ weights per filter} $$ ### Step 3: count all filters and their biases $$ \text{weights}=75\times8=600 $$ $$ \text{biases}=8,\qquad \text{total}=600+8=608 $$ **Final answer:** shape 15 × 15 × 8; 600 weights; 8 biases; 608 parameters. The 15 × 15 output locations reuse those weights. ## Convolution value One convolution output is a sum of matched input–kernel products. Move the window and repeat that exact rule for the next output. With several input channels, add contributions from every channel and add the filter bias once. $$ Y_{ij,o}=\sum_c\sum_u\sum_v K_{uvco}X_{iS+u-P,jS+v-P,c}+b_o $$ ## One filter over multiple channels: all contributions go into one cell **Definition.** Multi-channel convolution contracts the kernel's spatial and input-channel dimensions to produce one output value per filter at each spatial location. Different filters have independent learned kernels. ### Explanation and working ### 1. Separate spatial size from depth Imagine a 1 × 1 spatial location with three input channels [2, 5, 1]. A 1 × 1 filter still has one weight for each channel. Suppose its weights are [0.1, −0.2, 0.5] and its bias is 0.3. ![Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.](https://deep-learning-study-room.vercel.app/assets/figures/v_channel_products.svg) Diagram: Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example.. Labels: A 1 × 1 filter still mixes all three channels | A 1 × 1 filter still mixes all three channels | Use input [2, 5, 1], weights [0.1, −0.2, 0.5] and one bias 0.3. | Channel 1 | 2 × 0.1 = 0.2 | Channel 2 | 5 × (−0.2) = −1 | Channel 3 | 1 × 0.5 = 0.5 | One filter → one output value | 0.2 + (−1) + 0.5 + 0.3 = 0 | One spatial position can contain several channel values. The one-by-one filter | mixes them. Multiply each channel by its own weight, add the three contributions, then add the single bias. These are the numbers in the example. Tap the diagram to enlarge. $$ z=0.1(2)+(-0.2)(5)+0.5(1)+0.3 $$ $$ z=0.2-1+0.5+0.3=0 $$ The three channels contributed to one output value. A single filter creates one output channel, even when its input has several channels. ### 2. A larger filter has one spatial grid per input channel A 3 × 3 filter on a three-channel input has three 3 × 3 weight grids, totaling 27 weights. At one output location, calculate the nine products for channel 1, nine for channel 2 and nine for channel 3. Add all 27 products and then add the filter's one bias. The channel-wise sums are intermediate contributions; do not treat them as three final output channels unless three distinct filters were specified. ### 3. What changes when there are several filters? Each filter has its own full-depth weight set and bias. Four filters give four independent sums at each output location, so the result has four channels. The filters all receive the same input depth. For three input channels and four 3 × 3 filters, parameters are (27 + 1) × 4 = 112. Moving those filters over more image locations reuses these same 112 parameters. ### 4. One-by-one does not mean one weight A 1 × 1 convolution with 64 input channels and 16 filters has 64 weights and one bias per filter: (64 + 1) × 16 = 1,040 parameters. It mixes channel information at each spatial position. With 16 outputs it reduces depth from 64 to 16; height and width stay unchanged when stride is 1 and no padding is needed. This is the channel-reduction idea used by Inception bottlenecks and some residual projections. Spatial kernel size, input depth and output-channel count must all be read separately. ## Faculty example: a complete binary-image convolution **Question. Apply the shown 3 × 3 filter to the 5 × 5 image. Use stride 1, no padding and zero bias. Calculate every output value without flipping the filter. (Practice: 5 marks)** ### Given Image X = $\begin{bmatrix}1 & 1 & 1 & 0 & 0\\0 & 1 & 1 & 1 & 0\\0 & 0 & 1 & 1 & 1\\0 & 0 & 1 & 1 & 0\\0 & 1 & 1 & 0 & 0\end{bmatrix}$ . Filter K = $\begin{bmatrix}1 & 0 & 1\\0 & 1 & 0\\1 & 0 & 1\end{bmatrix}$ . These are the values in the Unit 4 lecture example. ### To find The output shape and all nine feature-map values. ### Formula — which one and why The output-size formula counts complete filter placements. At each placement, multiply matching image/filter cells and add the products. A filter entry of zero makes that particular product zero; it does not remove the image cell from the input. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Here N is input side, K is kernel side, P is padding per side and S is stride. The symbol K in the shape formula is the size 3, whereas the matrix K above contains the filter's coefficients. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ For this particular filter, only the four corners and the centre of each 3 × 3 window contribute. We can therefore write each sum using those five entries after explaining the four zero products. Below, their order is always **top-left, top-right, centre, bottom-left, bottom-right** within the current window. ![Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.](https://deep-learning-study-room.vercel.app/assets/figures/u4_binary_windows.svg) Diagram: Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions.. Labels: u4 binary windows | One filter, nine window positions | Input: 5 × 5 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | Shared filter: 3 × 3 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | → | Output: 3 × 3 | 4 | 3 | 4 | 2 | 4 | 3 | 2 | 3 | 4 | Orange window → orange output cell = 4. Move one cell at a time. Each output cell corresponds to a different 3 × 3 window. The filter coefficients are reused at all nine positions. Tap the diagram to enlarge. ### Solution — step by step **Step 1: find the shape.** (5 − 3)/1 + 1 = 3. The output is 3 × 3. **Step 2: top-left output, showing all nine products.** 1×1 + 1×0 + 1×1 + 0×0 + 1×1 + 1×0 + 0×1 + 0×0 + 1×1 = 1+0+1+0+1+0+0+0+1 = **4**. **Step 3: top-middle output.** Move the window one column right. Corners and centre are 1,0,1,0,1. Sum = 1+0+1+0+1 = **3**. **Step 4: top-right output.** Move one more column right. Corners and centre are 1,0,1,1,1. Sum = 1+0+1+1+1 = **4**. **Step 5: middle-left output.** Return to column 1 and move down one row. Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 6: middle-middle output.** Corners and centre are 1,1,1,0,1. Sum = 1+1+1+0+1 = **4**. **Step 7: middle-right output.** Corners and centre are 1,0,1,1,0. Sum = 1+0+1+1+0 = **3**. **Step 8: bottom-left output.** Corners and centre are 0,1,0,0,1. Sum = 0+1+0+0+1 = **2**. **Step 9: bottom-middle output.** Corners and centre are 0,1,1,1,0. Sum = 0+1+1+1+0 = **3**. **Step 10: bottom-right output.** Corners and centre are 1,1,1,1,0. Sum = 1+1+1+1+0 = **4**. **Final answer:** Y = $\begin{bmatrix}4 & 3 & 4\\2 & 4 & 3\\2 & 3 & 4\end{bmatrix}$ . All nine outputs use the same nine filter weights. They do not introduce nine different filters. **Yaad rakho:** Window badalta hai; filter ke numbers same rehte hain. ## Faculty example: combine three RGB channel sums **Question. At one position, a convolutional filter receives the following red, green and blue 3 × 3 patches and matching kernel slices. Calculate one output value with bias 1. (Practice: 3 marks)** ### Given Red patch = $\begin{bmatrix}0 & 0 & 0\\0 & 156 & 155\\0 & 153 & 154\end{bmatrix}$ ; red kernel = $\begin{bmatrix}-1 & -1 & 1\\0 & 1 & -1\\0 & 1 & 1\end{bmatrix}$ . Green patch = $\begin{bmatrix}0 & 0 & 0\\0 & 167 & 166\\0 & 164 & 165\end{bmatrix}$ ; green kernel = $\begin{bmatrix}1 & 0 & 0\\1 & -1 & -1\\1 & 0 & -1\end{bmatrix}$ . Blue patch = $\begin{bmatrix}0 & 0 & 0\\0 & 163 & 162\\0 & 160 & 161\end{bmatrix}$ ; blue kernel = $\begin{bmatrix}0 & 1 & 1\\0 & 1 & 0\\1 & -1 & 1\end{bmatrix}$ . Bias = 1. These are three slices of **one** full-depth filter, not three independent output filters. ### To find The contribution of each input channel and the final scalar output at this position. ### Formula — which one and why Multiply corresponding entries within each channel. Add the three channel sums because this filter spans all input channels. Add its single bias once, after accumulating the channel contributions. $$ z=s_R+s_G+s_B+b $$ The quantities s\_R, s\_G and s\_B are intermediate weighted sums, not separate final output channels. ### Solution — step by step **Step 1: red-channel products.** The top row and first column contribute zero. The remaining terms are 156×1 + 155×(−1) + 153×1 + 154×1. **Step 2: red-channel sum.** 156 − 155 + 153 + 154 = **308**. **Step 3: green-channel products.** The remaining patch terms are 167×(−1) + 166×(−1) + 164×0 + 165×(−1). **Step 4: green-channel sum.** −167 −166 +0 −165 = **−498**. **Step 5: blue-channel products.** The nonzero-patch terms are 163×1 + 162×0 + 160×(−1) + 161×1. **Step 6: blue-channel sum.** 163 +0 −160 +161 = **164**. **Step 7: add the channel contributions.** 308 −498 +164 = −26. **Step 8: add one filter bias.** −26 +1 = **−25**. **Final answer:** The convolution output is −25. If a subsequent ReLU is explicitly requested, that activation produces max(0,−25) = 0. The convolution value and activated value are different stages. **Yaad rakho:** Teen channels ka contribution jodo; ek filter ka bias ek baar jodo. ## The convolution calculation **Definition.** A convolutional layer calculates local weighted sums with a spatially shared kernel and optional bias. Standard deep-learning implementations ordinarily use cross-correlation, without flipping the kernel; examination calculations must follow the stated convention. ### Explanation and working In deep-learning software, the operation normally called convolution is cross-correlation: use the kernel as written, without flipping it. This is the convention in this course’s CNN numericals and the supplied latest answer key. Mathematical convolution would rotate the kernel; state that convention if a question explicitly requests it. **Question.** Apply a 2 × 2 kernel Matrix 1 shown below to the 3 × 3 image Matrix 2 shown below, with stride 1, no padding and zero bias. ### Given Input X = $\begin{bmatrix}1 & 2 & 3\\4 & 5 & 6\\7 & 8 & 9\end{bmatrix}$ . Kernel K = $\begin{bmatrix}1 & 0\\0 & -1\end{bmatrix}$ . Stride = 1; padding = 0; bias = 0; no kernel flip. ### To find Find the 2 × 2 output map. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_cd39cabe48.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Matrix 1 | 1 | 0 | 0 | −1 | Matrix 2 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 Given matrices for this question Tap the diagram to enlarge. $$ X=\begin{bmatrix}1&2&3\\4&5&6\\7&8&9\end{bmatrix},\quad K=\begin{bmatrix}1&0\\0&-1\end{bmatrix} $$ **Step 1. Top-left window.** 1(1) + 2(0) + 4(0) + 5(−1) = 1 − 5 = −4. **Step 2. Move right by one.** 2(1) + 3(0) + 5(0) + 6(−1) = 2 − 6 = −4. **Step 3. Move down and return left.** 4(1) + 5(0) + 7(0) + 8(−1) = 4 − 8 = −4. **Step 4. Bottom-right window.** 5(1) + 6(0) + 8(0) + 9(−1) = 5 − 9 = −4. Output is $\begin{bmatrix}-4 & -4\\-4 & -4\end{bmatrix}$ . If ReLU follows, that next layer changes all four entries to zero; do not silently apply it when it was not requested. ![Four windows produce four output cells](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: Four windows produce four output cells. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. Four windows produce four output cells Tap the diagram to enlarge. ## Drill 34 Convolution with a negative coefficient ## Question **Question.** Use the input A and kernel K shown below, stride 1, padding 0. Compute all outputs. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | --- | --- | | 1 | −1 | | 0 | 1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel K are printed with the question. Stride S = 1; padding P = 0; bias = 0. Use the kernel without flipping it. ### To find Calculate all nine output cells. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−2) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(-1)+(1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 3 = 3** **2 + 3 = 5** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(-1)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(3)=5 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 2 = 2** **(−1) + 2 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(-1)+(2)=1 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 1 = 1** **(−1) + 1 = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(-1)+(1)=0 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | −3 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + (−3) = −2** Row 2**0 + 0 = 0** **(−2) + 0 = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(-1)=-2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(-2)+(0)=-2 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | −2 | | 0 | 1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + (−2) = 1** Row 2**0 + 1 = 1** **1 + 1 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(1)+(1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | −1 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + (−1) = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(1)+(0)=1 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 2 = 2** **1 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(-1)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(2)=3 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | −1 | | 0 | 1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | −1 | | 0 | 3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + (−1) = −1** Row 2**0 + 3 = 3** **(−1) + 3 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | 5 | 1 | | 0 | −2 | 2 | | 1 | 3 | 2 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(-1)=-1 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(1)=3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(-1)+(3)=2 $$ **Final output:** $\begin{bmatrix}0 & 5 & 1\\0 & -2 & 2\\1 & 3 & 2\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ## Drill 46 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. ## Max, min and average pooling Pooling reduces each window to one value. Max pooling keeps the largest entry; average pooling adds all included entries and divides by their count. Pooling is performed independently in each channel. $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{max}=\max_{(i,j)\in W}x_{ij} $$ $$ y_{avg}=\frac1{|W|}\sum_{(i,j)\in W}x_{ij} $$ ## Pooling needs comparisons or averages **Definition.** Pooling is a local aggregation operation that summarizes each spatial window, usually independently per channel. Max pooling selects the largest value; average pooling calculates a mean. Standard pooling has no learned weights or biases. ### Explanation and working Max pooling selects the largest value inside each window. Average pooling adds window entries and divides by their count. Ordinary max and average pooling have no learned weights or biases. Pool each input channel independently, so the number of channels is unchanged. **Question.** Pool this 4 × 4 map with 2 × 2 windows, stride 2 and no padding: ### Given The 4 × 4 input map is printed below. Window = 2 × 2; stride = 2; padding = 0. ### To find Find the pooled output by the operation(s) requested. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step $$ A=\begin{bmatrix}21&11&15&10\\12&9&10&11\\11&13&18&12\\7&4&15&12\end{bmatrix} $$ ### Top-left window **Step 1: select the four entries.** [21, 11, 12, 9] **Step 2: choose the largest entry.** $$ y_{\max}=21 $$ **Step 3: add all entries.** $$ 21+11+12+9=53 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{53}4=13.25 $$ ### Top-right window **Step 1: select the four entries.** [15, 10, 10, 11] **Step 2: choose the largest entry.** $$ y_{\max}=15 $$ **Step 3: add all entries.** $$ 15+10+10+11=46 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{46}4=11.5 $$ ### Bottom-left window **Step 1: select the four entries.** [11, 13, 7, 4] **Step 2: choose the largest entry.** $$ y_{\max}=13 $$ **Step 3: add all entries.** $$ 11+13+7+4=35 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{35}4=8.75 $$ ### Bottom-right window **Step 1: select the four entries.** [18, 12, 15, 12] **Step 2: choose the largest entry.** $$ y_{\max}=18 $$ **Step 3: add all entries.** $$ 18+12+15+12=57 $$ **Step 4: divide the sum by four for the average.** $$ y_{\mathrm{avg}}=\frac{57}4=14.25 $$ Outputs are max $\begin{bmatrix}21 & 15\\13 & 18\end{bmatrix}$ and average $\begin{bmatrix}13.25 & 11.5\\8.75 & 14.25\end{bmatrix}$ . ![Each pooling window reduces several values to one](https://deep-learning-study-room.vercel.app/assets/figures/pooling.svg) Diagram: Each pooling window reduces several values to one. Labels: INPUT 4 × 4 | 21 | 11 | 15 | 10 | 12 | 9 | 10 | 11 | 11 | 13 | 18 | 12 | 7 | 4 | 15 | 12 | MAX POOL | 21 | 15 | 13 | 18 | AVERAGE POOL | 13.25 | 11.5 | 8.75 | 14.25 | 2 × 2 windows | stride 2 Each pooling window reduces several values to one Tap the diagram to enlarge. Pooling reduces spatial size and later computation and can provide limited robustness to small shifts. It can lose exact position information. It does not guarantee complete translation invariance. ## Max, min and average pooling on the same lecture grid **Definition.** Pooling summarizes each local window separately within each channel. Max pooling keeps its largest value, min pooling its smallest value, and average pooling its arithmetic mean. Ordinary versions have no trainable weights or biases. **Question. Apply 2 × 2 max, min and average pooling with stride 2 and no padding to the following 4 × 4 input. Give all three output matrices. (Practice: 3 marks)** ### Given X = $\begin{bmatrix}12 & 20 & 30 & 0\\8 & 12 & 2 & 0\\34 & 70 & 37 & 4\\112 & 100 & 25 & 12\end{bmatrix}$ . Window = 2 × 2; stride = 2; padding = 0. ### To find The shared output shape and each of the four cells for each pooling method. ### Formula — which one and why Each output summarizes four input numbers. Use max for the largest, min for the smallest, and sum divided by 4 for the average. The window moves two cells, so these four windows do not overlap. $$ y_{\max}=\max(x_1,x_2,x_3,x_4) $$ $$ y_{\min}=\min(x_1,x_2,x_3,x_4) $$ $$ y_{\mathrm{avg}}=\frac{x_1+x_2+x_3+x_4}{4} $$ ![The four colours identify four separate pooling windows. Each window creates one cell in each output map.](https://deep-learning-study-room.vercel.app/assets/figures/u4_pooling_windows.svg) Diagram: The four colours identify four separate pooling windows. Each window creates one cell in each output map.. Labels: u4 pooling windows | Same four windows, three different summaries | Input: 4 × 4 | 12 | 20 | 30 | 0 | 8 | 12 | 2 | 0 | 34 | 70 | 37 | 4 | 112 | 100 | 25 | 12 | Max | 20 | 30 | 112 | 37 | Min | 8 | 0 | 34 | 4 | Average | 13 | 8 | 79 | 19.5 | Bottom-right average: (37 + 4 + 25 + 12) ÷ 4 = 19.5 | Pooling changes values and shape; ordinary pooling learns no parameters. The four colours identify four separate pooling windows. Each window creates one cell in each output map. Tap the diagram to enlarge. ### Solution — step by step **Step 1: output size.** (4 −2)/2 +1 = 2. Each output has shape 2 × 2. **Step 2: top-left window [12,20,8,12].** Largest = 20; smallest = 8. Sum = 12+20+8+12 = 52. Average = 52/4 = 13. **Step 3: top-right window [30,0,2,0].** Largest = 30; smallest = 0. Sum = 30+0+2+0 = 32. Average = 32/4 = 8. **Step 4: bottom-left window [34,70,112,100].** Largest = 112; smallest = 34. Sum = 34+70+112+100 = 316. Average = 316/4 = 79. **Step 5: bottom-right window [37,4,25,12].** Largest = 37; smallest = 4. Sum = 37+4+25+12 = 78. Average = 78/4 = **19.5**. Do not silently round this to 20. **Step 6: put each result back in its window's position.** Max = $\begin{bmatrix}20 & 30\\112 & 37\end{bmatrix}$ . Min = $\begin{bmatrix}8 & 0\\34 & 4\end{bmatrix}$ . Average = $\begin{bmatrix}13 & 8\\79 & 19.5\end{bmatrix}$ . **Final answer:** Three 2 × 2 maps as above; zero trainable parameters for each pooling operation. ### Why the choice matters Max retains the largest activation in a feature window, which can still be negative if every entry is negative. Min retains the lowest response. Average retains an overall response but can dilute a sharp local peak. On raw intensity images, min can retain a dark stroke and max can retain a bright stroke. A learned CNN feature map represents filter responses; the best pooling choice is not determined solely by the original background colour. Pooling has no parameters to update, but gradients can still pass through it. For a unique max, the upstream gradient is routed to the winning input. For an average over four inputs, each receives one-fourth of that gradient. Fewer pooled activations can reduce a later dense layer's parameter count; they do not change the size of an earlier convolution's filter. **Yaad rakho:** Max = sabse bada; min = sabse chhota; average = sab jodo, count se divide. ## Drill 36 Max and average pooling ## Question **Question.** Pool the input A shown below with 2 × 2 windows and stride 2. Find max and average outputs. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_563e4ee8f7.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | A | 1 | 4 | 2 | 3 | 0 | 2 | 5 | 1 | 6 | 3 | 2 | 0 | 1 | 7 | 4 | 8 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input A is printed with the question. Window size = 2 × 2; stride = 2; no padding. ### To find Find the complete max-pooling and average-pooling outputs. ### Formula — which one and why **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. ### Solution — step by step ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. **Shape:** $\left\lfloor\frac{4-2}{2}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [1, 4, 0, 2] **Step 2: add all selected entries.** 1 + 4 + 0 + 2 = 7 **Step 3: divide by the 4 entries in this window.** 7/4 = **1.75**. Maximum = **4** ### Pooling window (1,2) **Step 1: select the entries.** [2, 3, 5, 1] **Step 2: add all selected entries.** 2 + 3 + 5 + 1 = 11 **Step 3: divide by the 4 entries in this window.** 11/4 = **2.75**. Maximum = **5** ### Pooling window (2,1) **Step 1: select the entries.** [6, 3, 1, 7] **Step 2: add all selected entries.** 6 + 3 + 1 + 7 = 17 **Step 3: divide by the 4 entries in this window.** 17/4 = **4.25**. Maximum = **7** ### Pooling window (2,2) **Step 1: select the entries.** [2, 0, 4, 8] **Step 2: add all selected entries.** 2 + 0 + 4 + 8 = 14 **Step 3: divide by the 4 entries in this window.** 14/4 = **3.5**. Maximum = **8** **Average output:** $\begin{bmatrix}1.75 & 2.75\\4.25 & 3.5\end{bmatrix}$ . **Max output:** $\begin{bmatrix}4 & 5\\7 & 8\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. ## Drill 46 Mixed convolution pool dense and loss ## Question **Question.** A is shown below, kernel Matrix 2 shown below, stride 1, valid. Apply ReLU, then 2 × 2 max pool stride 1. Flatten rowwise, use weight vector [0.1,0.2,0.3,0.4], bias 0, sigmoid output, target 1. Find all stages and BCE. Given matrices Input A Input A | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Kernel K · Matrix 2 Kernel K · Matrix 2 | | | | --- | --- | | 1 | 0 | | 0 | −1 | Stride 1; P = 0 (no padding). ## Answer — step by step ### Given Input A and kernel Matrix 2 are printed with the question. Conv: stride 1, valid padding; follow with ReLU. Max pool: 2 × 2, stride 1; flatten row by row. Dense weights = [0.1,0.2,0.3,0.4]; bias = 0; sigmoid output. Binary target = 1. ### To find Find convolution, activated map, pooled map, flattened values, prediction and BCE. ### Formula — which one and why **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. **Pooling.** Max pooling selects the largest window entry. Average pooling adds its entries and divides by their count. Each channel is pooled separately. V = current pooling window; n = number of entries in that window. Pooling learns no weights or biases. $$ y_{\max}=\max V $$ $$ y_{\mathrm{avg}}=\frac1n\sum_{v\in V}v $$ **Yaad rakhne ka cue:** Max mein biggest; average mein total ÷ window ka count. **Weighted sum and activation.** Each input is multiplied by its connection weight. Add these contributions, add the bias once, then apply the stated activation. xᵢ = input i; wᵢ = its weight; b = bias; z = sum before activation; a = output after activation; f = the specified activation function. $$ z=\sum_iw_ix_i+b $$ $$ a=f(z) $$ **Yaad rakhne ka cue:** Input × weight, sabka sum, bias once, phir activation. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Binary cross-entropy.** For a binary target, choose the probability of the correct outcome and take its negative natural log. Sigmoid plus BCE has combined logit derivative p−y. y is 0 or 1; p is the predicted probability of class 1; z is the sigmoid logit. The p−y derivative already includes the sigmoid derivative. $$ L=-[y\ln p+(1-y)\ln(1-p)] $$ $$ \frac{\partial L}{\partial z}=p-y $$ **Yaad rakhne ka cue:** Binary target 1 ho toh −ln(p); target 0 ho toh −ln(1−p). ### Solution — step by step ### Formulas and the operation we are performing A filter window produces one output value. Multiply entries at matching positions and add the products. In this deep-learning convention, keep the filter in its printed orientation. $$ Y_{ij}=\sum_{u=0}^{K_h-1}\sum_{v=0}^{K_w-1}X_{iS+u,jS+v}K_{uv}+b $$ Here i and j identify an output position; u and v count positions inside its window. S is stride, and b is the filter bias, zero in this question. The equation uses a padded input if padding is present and zero-based indices; the working below labels output positions from 1 for readability. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is input width/height, P is padding per side, and K is filter width/height. Floor means take the integer below the result, then add one. ### Formulas and meaning Each 2 × 2 window contains four numbers. Max pooling selects the largest of those four. Average pooling adds all four and divides by 4. Move the window by the given stride after producing one answer. $$ p_{\mathrm{max}}=\max(a_1,a_2,a_3,a_4) $$ $$ p_{\mathrm{avg}}=\frac{a_1+a_2+a_3+a_4}{4} $$ $$ N_{\mathrm{pool}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Use the input size at the pooling layer. The output keeps the same channel count. ### Completing the network after pooling Flatten row by row. Multiply the resulting entries by the dense weights and add bias. Apply sigmoid to produce p. For target 1, binary cross-entropy is minus ln(p). $$ z=\sum_iw_ix_i+b,\qquad p=\frac1{1+e^{-z}},\qquad L=-\ln p $$ Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. **Step 1 shape:** $\left\lfloor\frac{4+2 \times 0-2}{1}\right\rfloor+1$ = 3 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–2**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−1) = −1** **1 + (−1) = 0** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 1. **Window row 1.** $$ R_1=(1)\times(1)+(2)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(-1)=0 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–2**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + (−3) = −3** **2 + (−3) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(0)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(2)+(-3)=-1 $$ ### Output position (1, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 3) into one output cell Input rows **1–2**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−2) = −2** **0 + (−2) = −2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 1, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(3)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,3}=(0)+(-2)=-2 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **2–3**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−1) = −1** **0 + (−1) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 1. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(0)+(-1)=-1 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **2–3**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + 0 = 0** **1 + 0 = 1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(3)\times(0)=1 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(1)+(0)=1 $$ ### Output position (2, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 3) into one output cell Input rows **2–3**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 3 | 0 | | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 = 3** Row 2**0 + (−1) = −1** **3 + (−1) = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 2, column 3. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)=3 $$ **Window row 2.** $$ R_2=(0)\times(0)+(1)\times(-1)=-1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,3}=(3)+(-1)=2 $$ ### Output position (3, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 1) into one output cell Input rows **3–4**, columns **1–2**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 2 | 0 | | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 = 2** Row 2**0 + 0 = 0** **2 + 0 = 2** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 1. **Window row 1.** $$ R_1=(2)\times(1)+(1)\times(0)=2 $$ **Window row 2.** $$ R_2=(1)\times(0)+(0)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{3,1}=(2)+(0)=2 $$ ### Output position (3, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 2) into one output cell Input rows **3–4**, columns **2–3**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 1 | 0 | | 0 | −2 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 = 1** Row 2**0 + (−2) = −2** **1 + (−2) = −1** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)=1 $$ **Window row 2.** $$ R_2=(0)\times(0)+(2)\times(-1)=-2 $$ **Add the rows to obtain one output cell.** $$ Y_{3,2}=(1)+(-2)=-1 $$ ### Output position (3, 3) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (3, 3) into one output cell Input rows **3–4**, columns **3–4**. The window moves **1** cell per output step. 1. Select the input window 1. Select the input window | | | | | | --- | --- | --- | --- | | 1 | 2 | 0 | 1 | | 0 | 1 | 3 | 2 | | 2 | 1 | 0 | 1 | | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | --- | --- | | 1 | 0 | | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | --- | --- | | 0 | 0 | | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**0 + 0 = 0** Row 2**0 + (−3) = −3** **0 + (−3) = −3** 5. Place it in the output 5. Place it in the output | | | | | --- | --- | --- | | 0 | −1 | −2 | | −1 | 1 | 2 | | 2 | −1 | −3 | Output row 3, column 3. **Window row 1.** $$ R_1=(0)\times(1)+(1)\times(0)=0 $$ **Window row 2.** $$ R_2=(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{3,3}=(0)+(-3)=-3 $$ **Final output:** $\begin{bmatrix}0 & -1 & -2\\-1 & 1 & 2\\2 & -1 & -3\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ReLU turns negative entries into 0: $\begin{bmatrix}0 & 0 & 0\\0 & 1 & 2\\2 & 0 & 0\end{bmatrix}$ . **Shape:** $\left\lfloor\frac{3-2}{1}\right\rfloor+1$ = 2 on each axis; channels stay unchanged. ### Pooling window (1,1) **Step 1: select the entries.** [0, 0, 0, 1] **Step 2: add all selected entries.** 0 + 0 + 0 + 1 = 1 **Step 3: divide by the 4 entries in this window.** 1/4 = **0.25**. Maximum = **1** ### Pooling window (1,2) **Step 1: select the entries.** [0, 0, 1, 2] **Step 2: add all selected entries.** 0 + 0 + 1 + 2 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,1) **Step 1: select the entries.** [0, 1, 2, 0] **Step 2: add all selected entries.** 0 + 1 + 2 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** ### Pooling window (2,2) **Step 1: select the entries.** [1, 2, 0, 0] **Step 2: add all selected entries.** 1 + 2 + 0 + 0 = 3 **Step 3: divide by the 4 entries in this window.** 3/4 = **0.75**. Maximum = **2** **Average output:** $\begin{bmatrix}0.25 & 0.75\\0.75 & 0.75\end{bmatrix}$ . **Max output:** $\begin{bmatrix}1 & 2\\2 & 2\end{bmatrix}$ . Pooling normally has zero trainable parameters and reduces spatial dimensions. Use MAX output and flatten rowwise: [1, 2, 2, 2]. Dense z = 0.1 × 1 + 0.2 × 2 + 0.3 × 2 + 0.4 × 2 = 1.9. p = σ(1.9) = 0.869891526. BCE = −ln(p) = **0.139386758**. Average pool values were computed for contrast; the question requires max pool. ## Convolution parameter count A filter has one learned weight for each spatial position in each input channel. Count the weights in one filter, add its one bias, then multiply by the number of filters. Sliding a filter to many positions reuses these weights; it does not create new parameters. $$ P_{conv}=(K_hK_wC_{in}+1)C_{out} $$ ## Output size and parameter count are different **Definition.** Output shape specifies the size of a layer's computed tensor. Parameter count specifies the number of independent trainable coefficients. Spatial weight sharing allows a convolutional layer to produce many activations from comparatively few parameters. ### Explanation and working **Question.** A 7 × 7 × 3 input passes through two 3 × 3 filters, stride 2 and padding 1. Find output shape and parameter count, including bias. ### Given Input = 7 × 7 × 3; two 3 × 3 filters. Stride = 2; padding = 1; include biases. ### To find Find the output shape and trainable parameter count. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step **Step 1 spatial size:** padding adds two, giving 7 + 2 − 3 = 6. Divide by stride: 6 ÷ 2 = 3. Apply floor and add the first position: 3 + 1 = 4. There are two filters, so output is 4 × 4 × 2. ![Output size counts produced values; parameter count counts adjustable weights and biases.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Output size counts produced values; parameter count counts adjustable weights and biases.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Output size counts produced values; parameter count counts adjustable weights and biases. Tap the diagram to enlarge. **Step 2 parameters:** one filter has 3 × 3 × 3 = 27 weights and one bias, giving 28. Two filters have 56 parameters. The 4 × 4 output positions reuse those weights; they do not create 16 new sets of parameters. ## Model answer: shape, activations and parameters ## Question **Question.** Distinguish a CNN layer's output shape, number of output activations and number of trainable parameters. Explain the formulas used to calculate them. ## Answer — step by step Output shape tells how the produced values are arranged. The activation count tells how many values were produced for one input. Parameter count tells how many learned weights and biases define the layer. These are three different questions. ![Keep output shape, activation count and parameter count as three separate quantities.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: Keep output shape, activation count and parameter count as three separate quantities.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. Keep output shape, activation count and parameter count as three separate quantities. Tap the diagram to enlarge. ### Spatial output shape For an ordinary convolution with dilation 1: $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ N is the incoming spatial size, P is padding on each side, K is kernel size and S is stride. Apply the rule independently to height and width. The number of output channels is the number of filters. ### Output activations $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ This is the number of output values for one example, excluding any batch dimension. ### Trainable parameters $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Each filter spans every incoming channel. The +1 is one bias per filter. Remove it if biases are disabled. Spatial output dimensions do not appear here because filters share their weights across positions. ### Small example A 7 × 7 × 3 input, two 3 × 3 filters, stride 2 and padding 1 gives output 4 × 4 × 2. It contains 32 activations but has (3 × 3 × 3 + 1) × 2 = 56 parameters. These counts do not need to match. In a multilayer question, the current layer's output channels become the next convolution's input channels. ## Faculty CNN table: shapes, activations and 62958 parameters **Question. A CNN receives a 32 × 32 × 3 image. It uses Conv1: eight 5 × 5 filters, stride 1, no padding; Pool1: 2 × 2, stride 2; Conv2: sixteen 5 × 5 filters, stride 1, no padding; Pool2: 2 × 2, stride 2; flatten; dense 120; dense 84; dense 10 followed by softmax. Include one bias per filter or receiving dense neuron. Find every output shape, activation count and trainable-parameter count. (Practice: 10 marks)** ### Given Input = 32 × 32 × 3. Both convolutions are ordinary full-channel convolutions. Pooling has no padding. Dense widths are 120, 84 and 10. No batch-normalization layer is present in this particular question. ### To find Output shape and number of output values for each layer, its trainable parameters, and the total parameters. ### Formula — which one and why Use the spatial-size formula for each convolution/pool. Use one full-depth kernel plus one bias per convolutional filter. Flattening counts all remaining values before the first dense layer. Each dense neuron then receives one weight per input plus one bias. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ A=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}} $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ A counts produced activations. P\_conv and P\_dense count independent learned coefficients. They are not the same quantity. Pooling, flattening and softmax itself add zero learned parameters here. ### Solution — step by step **Step 1: input.** Shape = 32 × 32 × 3. Values = 32×32×3 = 3,072. Trainable parameters = 0. **Step 2: Conv1 spatial size.** (32−5)/1 +1 = 28. Eight filters give shape 28 × 28 × 8. Activations = 28×28×8 = 6,272. **Step 3: Conv1 parameters.** One filter contains 5×5×3 = 75 weights. Add one bias: 76. Multiply by eight filters: 76×8 = **608**. **Step 4: Pool1.** (28−2)/2 +1 = 14. Shape = 14 × 14 × 8. Activations = 14×14×8 = 1,568. Parameters = 0; channel count remains eight. **Step 5: Conv2 spatial size.** (14−5)/1 +1 = 10. Sixteen filters give shape 10 × 10 × 16. Activations = 1,600. **Step 6: Conv2 parameters.** Its input has **eight** channels. Each filter has 5×5×8 = 200 weights, then one bias. Total = 201×16 = **3,216**. **Step 7: Pool2 and flatten.** (10−2)/2 +1 = 5. Pool shape = 5 × 5 × 16. Flattened length = 5×5×16 = **400**. Neither operation adds parameters. **Step 8: dense 120.** Weights = 400×120 = 48,000. Biases = 120. Parameters = **48,120**. Output contains 120 activations. **Step 9: dense 84.** Weights = 120×84 = 10,080. Biases = 84. Parameters = **10,164**. Output contains 84 activations. **Step 10: dense 10 and softmax.** Weights = 84×10 = 840. Biases = 10. Dense parameters = **850**. Softmax converts the ten logits into ten probabilities and adds **zero** parameters. A table naming the combined classifier row “Softmax” attributes these 850 parameters to its preceding dense transformation. **Step 11: add only the trainable counts.** 608+3,216 = 3,824; +48,120 = 51,944; +10,164 = 62,108; +850 = **62,958**. | Layer | Output shape | Activations | Parameters | | --- | --- | --- | --- | | Input | 32 × 32 × 3 | 3,072 | 0 | | Conv1 | 28 × 28 × 8 | 6,272 | 608 | | Pool1 | 14 × 14 × 8 | 1,568 | 0 | | Conv2 | 10 × 10 × 16 | 1,600 | 3,216 | | Pool2 / flatten | 5 × 5 × 16 / 400 | 400 | 0 | | Dense 120 | 120 | 120 | 48,120 | | Dense 84 | 84 | 84 | 10,164 | | Dense 10 → softmax | 10 | 10 | 850 + 0 | **Final answer:** The network has **62,958 trainable parameters**. This is the specified classroom CNN, not a universal parameter total for every LeNet-like network. **Yaad rakho:** Shape aage carry karo; weights aur activations ko alag columns mein rakho. ## Parameters are different from activation values **Definition.** Parameters are learned coefficients shared across evaluations. Activations are input-dependent values computed during a forward pass. The number of activation values therefore differs from the number of trainable parameters. ### Explanation and working An activation is a number produced for a particular input. A parameter is a learned number reused across inputs. A 100 × 100 feature map can be produced by a filter with only a few weights. ![The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.](https://deep-learning-study-room.vercel.app/assets/figures/v_parameters_activations.svg) Diagram: The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations.. Labels: Learned parameters are not output activations | Learned parameters are not output activations | Example: a 5 × 5 × 1 input with one 3 × 3 filter, stride 1, no padding. | Learned filter | 3 × 3 × 1 = 9 weights | Plus one bias | Total parameters = 10 | Output feature map | (5 − 3)/1 + 1 = 3 | Shape = 3 × 3 × 1 | Total activations = 9 | The same ten parameters are reused to calculate all nine output values. The ten learned parameters are reused at nine spatial positions. The resulting nine numbers are activations. Tap the diagram to enlarge. $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ The +1 is one bias per filter. If bias is explicitly disabled, remove it. Height and width of the image do not enter the parameter formula; they enter the number of output activations and the amount of computation. ### One filter, then all filters **Question. A 3 × 3 convolution receives 3 input channels and has 16 filters, each with one bias. Find its trainable parameters and output-channel count.** ### Given Kernel = 3 × 3; input channels = 3; filters = 16; one bias per filter. ### To find Weights, biases, total trainable parameters and output channels. ### Formula — which one and why Each filter spans every input channel and owns one bias. Spatial reuse does not create extra weights. Kₕ,Kw are kernel dimensions; Cᵢₙ is input depth; Cₒᵤₜ is the number of filters. $$ P=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Ek filter ka count banao, phir filters se multiply. ### Solution — step by step **Step 1: count weights in one filter.** $$ 3(3)(3)=27 $$ **Step 2: include its one bias.** $$ 27+1=28 $$ **Step 3: count all weights.** $$ 27(16)=432 $$ **Step 4: count all biases.** $$ 1(16)=16 $$ **Step 5: add and state channel count.** $$ P=432+16=448 $$ Each filter produces one channel, so output channels = 16. **Final answer:** 432 weights + 16 biases = 448 trainable parameters; 16 output channels. $$ P_{\mathrm{pool}}=0,\quad P_{\mathrm{flatten}}=0,\quad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flatten rearranges H × W × C activation values into one vector of length HWC. It does not learn anything. A following dense layer connects to every one of those values. ## Drill 37 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. ## Flatten and dense parameter count Flattening rearranges all activation values into a list. It learns no new numbers. A dense layer then connects every value in that list to every receiving neuron; each receiving neuron also has one bias. $$ N_{flat}=HWC $$ $$ P_{dense}=(N_{in}+1)N_{out} $$ ## Counting parameters in a dense network **Definition.** A dense layer connects every input component to every receiving neuron. Its trainable parameters are these connection weights and the receiving neurons' independently trainable biases. ### Deriving the parameter count For m inputs and n receiving neurons, each receiver needs m weights. There are therefore mn weights. With one bias per receiver, there are n additional parameters: $$ P_{\mathrm{dense}}=mn+n=(m+1)n $$ Activation values are computed from data and parameters; they are not additional trainable parameters. The input layer supplies data and does not acquire an extra bias merely because it appears in a diagram. ![Count all input-to-neuron connections and then one bias per receiving neuron.](https://deep-learning-study-room.vercel.app/assets/figures/v_dense_count.svg) Diagram: Count all input-to-neuron connections and then one bias per receiving neuron.. Labels: Count connections separately from biases | Count connections separately from biases | A dense layer with 3 inputs and 2 receiving neurons. | x1 | x2 | x3 | a1 | + b1 | a2 | + b2 | 3 × 2 = 6 weights | Two receiving neurons also have 2 biases. Total = 6 + 2 = 8 parameters. Count all input-to-neuron connections and then one bias per receiving neuron. Tap the diagram to enlarge. ### Complete numerical **Question.** A network has 10 input features, three hidden layers with 5 neurons each, and 4 outputs. Every receiving neuron has a bias. Count its trainable parameters. ### Given Layer sizes: 10 inputs → 5 hidden → 5 hidden → 5 hidden → 4 outputs. Every hidden/output neuron has one bias. ### To find Count weights and biases at each receiving layer, then the total. ### Formula — which one and why **Dense-layer parameters.** Every output neuron receives every input, so multiply inputs by outputs for weights. Add one bias per receiving neuron. Sum separate layers. n\_in = incoming features; n\_out = receiving neurons. The input layer itself has no learned receiving weights or biases. $$ P=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ **Yaad rakhne ka cue:** Har output ke liye saare inputs plus ek bias. ### Solution — step by step **Step 1: identify the consecutive layer widths.** $$ 10\longrightarrow5\longrightarrow5\longrightarrow5\longrightarrow4 $$ There are four trainable dense transformations. A connection between two hidden layers is counted once, as part of the receiving layer's parameters. **Step 2: count weights and biases for each transformation.** | Transformation | Weights | Biases | Layer total | | --- | --- | --- | --- | | Input → hidden 1 | 10 × 5 = 50 | 5 | 55 | | Hidden 1 → hidden 2 | 5 × 5 = 25 | 5 | 30 | | Hidden 2 → hidden 3 | 5 × 5 = 25 | 5 | 30 | | Hidden 3 → output | 5 × 4 = 20 | 4 | 24 | **Step 3: add the layer totals.** $$ P=55+30+30+24=139 $$ Equivalently, the network has 120 weights and 19 biases, so 120 + 19 = 139. **Final answer: 139 trainable parameters.** If a question explicitly removes biases, count only the 120 weights. Optimizer states such as Adam's moments are extra training storage, not extra parameters in this predictive network. ## One complete CNN pipeline **Question.** Input 128 × 128 × 3 → convolution 3 × 3, 16 filters, stride 1, no padding → pooling 2 × 2, stride 2 → convolution 3 × 3, 32 filters, stride 1, no padding → pooling 2 × 2, stride 2 → dense 240 → dense 120 → dense 5 with softmax. Find every output shape and trainable parameter count. Assume one bias per receiving filter or dense neuron. ### Given Input = 128 × 128 × 3. Conv: 3 × 3,16 filters,stride 1,valid → pool 2,stride 2. Conv: 3 × 3,32 filters,stride 1,valid → pool 2,stride 2. Flatten → dense 240 → dense 120 → dense 5. One bias per learned filter/dense neuron. ### To find Find every output shape and layer parameter count, then the network total. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step **Step 1. Conv 1 size.** (128 − 3)/1 + 1 = 126; output 126 × 126 × 16. Parameters = (3 × 3 × 3 + 1) × 16 = 448. ![Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.](https://deep-learning-study-room.vercel.app/assets/figures/v_cnn_pipeline.svg) Diagram: Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count.. Labels: Follow the complete CNN question, one layer at a time | Follow the complete CNN question, one layer at a time | Start with 128 × 128 × 3. Shape and learned-parameter count are separate. | LAYER | OUTPUT SHAPE / LENGTH | PARAMETERS | 1. Convolution 1 | 126 × 126 × 16 | 448 | 2. Pooling 1 | 63 × 63 × 16 | 0 | 3. Convolution 2 | 61 × 61 × 32 | 4,640 | 4. Pooling 2 | 30 × 30 × 32 | 0 | 5. Flatten | 28,800 | 0 | 6. Dense 240 | 240 | 6,912,240 | 7. Dense 120 | 120 | 28,920 | 8. Classifier | 5 | 605 | Total learned parameters = 6,946,853 | Each output is the next layer’s input. Pooling and flattening add no learned | parameters. Follow all eight layers from the question. Each row keeps the output shape separate from its learned-parameter count. Tap the diagram to enlarge. **Step 2. Pool 1 size.** $\left\lfloor\frac{126 - 2}{2}\right\rfloor+1$ = 63; output 63 × 63 × 16. Parameters = 0. **Step 3. Conv 2 size.** (63 − 3)/1 + 1 = 61; output 61 × 61 × 32. Input channels are now 16, not the original 3. Parameters = (3 × 3 × 16 + 1) × 32 = 4,640. **Step 4. Pool 2 size.** $\left\lfloor\frac{61 - 2}{2}\right\rfloor+1$ = $\left\lfloor 29.5\right\rfloor$ + 1 = 30; output 30 × 30 × 32. Parameters = 0. **Step 5. Flatten.** 30 × 30 × 32 = 28,800 values. Parameters = 0. **Step 6. Dense 240.** 28,800 weights per neuron plus one bias, multiplied by 240: (28,800 + 1) × 240 = 6,912,240. **Step 7. Dense 120.** (240 + 1) × 120 = 28,920. **Step 8. Classifier.** (120 + 1) × 5 = 605. Softmax itself adds no extra parameters; these 605 belong to the dense class-score layer. **Step 9. Total.** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853**. Most parameters come from the first dense layer. ## Drill 37 Complete small CNN ## Question **Question.** Input 10 × 10 × 1 → Conv 3 × 3,4 filters, stride 1, valid → Pool 2 stride 2 → Flatten → Dense 3. Find all shapes and parameters. ## Answer — step by step ### Given Input = 10 × 10 × 1. Conv: 3 × 3, 4 filters, stride 1, valid padding. Pool: 2 × 2, stride 2; then flatten and dense layer with 3 outputs. Include one bias per filter/dense neuron. ### To find Find all output shapes, layer parameter counts and their sum. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. ### Solution — step by step ### Formulas and assumptions “Valid” means zero padding. The convolution and dense layers each have one bias per receiving filter or neuron. Pooling and flattening have no learned parameters. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Step 1: convolution shape $$ N_{\mathrm{conv}}=\frac{10+0-3}{1}+1=7+1=8 $$ Four filters produce four channels: **8 × 8 × 4**. ### Step 2: convolution parameters $$ \text{weights}=3\times3\times1\times4=36 $$ $$ \text{biases}=4,\qquad P_{\mathrm{conv}}=36+4=40 $$ ### Step 3: pooling shape Pooling now receives size 8, not the original size 10. $$ N_{\mathrm{pool}}=\left\lfloor\frac{8-2}{2}\right\rfloor+1=3+1=4 $$ Channels remain 4: **4 × 4 × 4**. Pooling parameters = 0. ### Step 4: flatten $$ n_{\mathrm{flat}}=4\times4\times4=64 $$ Flatten creates a vector of 64 numbers. It rearranges values, so its parameter count is 0. ### Step 5: dense layer with 3 outputs $$ \text{weights}=64\times3=192 $$ $$ \text{biases}=3,\qquad P_{\mathrm{dense}}=192+3=195 $$ ### Step 6: add learned parameters $$ P_{\mathrm{total}}=40+0+0+195=235 $$ **Final answer:** shapes 8 × 8 × 4 → 4 × 4 × 4 → 64 → 3; total 235 parameters. ## Drill 49 Mixed CNN dimensions softmax and loss ## Question **Question.** Input 8 × 8 × 3 → Conv 3, stride 1, padding 1, 2 filters → Pool 2, stride 2 → Flatten → Dense 3. Logits [0,1,−1], true class is the second class (one-hot target [0,1,0]). Find shapes, parameters, probabilities and CCE. ## Answer — step by step ### Given Input = 8 × 8 × 3. Conv: kernel 3, stride 1, padding 1, 2 filters. Pool: kernel 2, stride 2; flatten; dense layer with 3 outputs. Logits = [0,1,−1]; true class is the second class (one-hot target [0,1,0]). ### To find Find shapes, parameter counts, probabilities and CCE. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Stable softmax.** Exponentiate each logit and divide by the sum. Subtracting the largest logit first keeps the exponentials numerically manageable and leaves the probabilities unchanged. zⱼ = class j score; M = largest score; pⱼ = class j probability. All probabilities sum to 1. $$ M=\max_j z_j $$ $$ p_j=\frac{e^{z_j-M}}{\sum_k e^{z_k-M}} $$ **Yaad rakhne ka cue:** Largest score ghatao, exp nikalo, total se sabko divide karo. **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step ### Formulas used Separate shape from learned parameter count. H, W and C mean height, width and channels. K, P and S mean filter size, padding per side and stride. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ The +1 counts one bias per receiving filter or neuron. Pooling and flattening add no learned parameters. Softmax converts scores to probabilities; CCE uses the probability of the correct class. $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\quad c=\max_jz_j,\quad L=-\ln p_{\mathrm{correct}} $$ Conv size = (8+2−3)+1 = 8; output 8 × 8 × 2; parameters = (3 × 3 × 3+1) × 2 = 56. Pool 4 × 4 × 2; parameters 0. Flatten 32. Dense 3 parameters = (32+1) × 3 = 99. Total = **155**. **Step 1 stabilize:** subtract the largest logit 1 from every value: [-1, 0, -2]. This does not change softmax. **Step 2 exponentiate:** exp values = [0.367879441, 1, 0.135335283]. **Step 3 denominator:** sum = 1.503214724. **Step 4 divide each value by this same sum:** probabilities = **[0.244728471, 0.665240956, 0.090030573]**. **Check:** all are positive and their sum is 1 (apart from rounding); the largest logit gets the largest probability. Correct class probability = 0.665240956; CCE = −ln(0.665240956) = **0.407605964**. ## Convolution multiply-accumulate count A parameter count counts learned numbers. A multiply-accumulate count counts how many times input–weight products are used. One filter is reused at every output position, so computation includes output height and width even though its parameter count does not. $$ MAC=H_{out}W_{out}K_hK_wC_{in}C_{out} $$ ## Why one by one convolution saves work **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts. ![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg) Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications. Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge. **Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.** ### Given Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size. ### To find Direct MACs, total two-stage bottleneck MACs and percentage saved. ### Formula — which one and why Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs. ### Solution — step by step **Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400 $$ **Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels. $$ \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448 $$ **Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels. $$ \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200 $$ **Step 4: add the two bottleneck costs.** $$ \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648 $$ **Step 5: calculate the saving.** $$ \mathrm{saving}=120422400-12443648=107978752 $$ $$ \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\% $$ **Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption. ### A second bottleneck with different dimensions **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1. ### To find Direct MACs, bottleneck MACs and saving. ### Formula — which one and why Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers. ### Solution — step by step **Step 1: direct spatial convolution.** $$ 8(8)(8)(3)(3)(16)=73728 $$ **Step 2: 1 × 1 reduction.** $$ 8(8)(4)(1)(1)(16)=4096 $$ **Step 3: reduced-depth 3 × 3 convolution.** $$ 8(8)(8)(3)(3)(4)=18432 $$ **Step 4: add the stages and subtract from the direct cost.** $$ 4096+18432=22528 $$ $$ 73728-22528=51200 $$ **Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values. ## The faculty's 480-channel bottleneck numerical ![Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.](https://deep-learning-study-room.vercel.app/assets/figures/u4_bottleneck_480.svg) Diagram: Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers.. Labels: u4 bottleneck 480 | Same output shape, different computation | Input | 14 × 14 × 480 | → | Direct 5 × 5 | 48 filters | → | Output | 14 × 14 × 48 | 112,896,000 MACs | 1 × 1 reduction | 480 → 16 channels | → | Reduced 5 × 5 | 16 → 48 channels | → | Output | 14 × 14 × 48 | 1,505,280 + 3,763,200 = 5,268,480 MACs | The lower path also starts from 14 × 14 × 480. Count both layers. Both paths start from 14 × 14 × 480 and end at 14 × 14 × 48. The bottleneck cost includes both layers. Tap the diagram to enlarge. **Question (practice, 3 marks). Compare a direct same-padded 5 × 5 convolution with 48 filters on a 14 × 14 × 480 input against a 1 × 1 reduction to 16 channels followed by the same 5 × 5, 48-filter layer. Use stride 1. Count MACs, ignore bias additions, and find the percentage reduction.** ### Given Both paths produce 14 × 14 × 48. Direct spatial filters span 480 channels; reduced spatial filters span 16. The reduction has 16 output filters. ### To find MACs for the direct path, both bottleneck stages, their total and percentage saved. ### Formula — which one and why $$ M=H_oW_oC_oK^2C_i $$ M is the MAC count. H\_oW\_oC\_o counts output values; K²C\_i is the products used for each output. One MAC means one multiply-accumulate pair. Count both layers of the bottleneck. $$ \mathrm{saving}\%=100\frac{M_{\mathrm{direct}}-M_{\mathrm{bottleneck}}}{M_{\mathrm{direct}}} $$ ### Solution — step by step **Step 1: direct output count.** 14 × 14 × 48 = 9,408. **Step 2: products per direct output.** 5 × 5 × 480 = 12,000. **Step 3: direct MACs.** $$ 9408(12000)=112896000 $$ **Step 4: reduction-layer MACs.** Each of 14 × 14 × 16 outputs combines 480 values. $$ 14(14)(16)(480)=1505280 $$ **Step 5: reduced spatial-layer MACs.** Each of 9,408 outputs now combines 5 × 5 × 16 = 400 values. $$ 9408(400)=3763200 $$ **Step 6: add both bottleneck costs.** $$ 1505280+3763200=5268480 $$ **Step 7: calculate the saving.** $$ 112896000-5268480=107627520 $$ $$ 100\frac{107627520}{112896000}\approx95.333333\% $$ **Final answer:** direct = **112.896 million MACs**; bottleneck = **5.26848 million MACs**; saving ≈ **95.33%**. The lecture's 112.9M and 5.3M are rounded versions. This arithmetic proves reduced computation; equal prediction accuracy is not guaranteed by the count alone. ## Drill 39 Bottleneck cost with new values ## Question **Question.** A 16 × 16 × 64 map feeds 32 same-padded 3 × 3 filters. Compare direct MACs with an 8-channel 1 × 1 bottleneck first. ## Answer — step by step ### Given Input = 16 × 16 × 64. Direct branch: 32 same-padded 3 × 3 filters. Alternative: 1 × 1 reduction to 8 channels, then the same 32-filter 3 × 3 layer. Spatial size remains 16 × 16; exclude bias additions. ### To find Find both MAC counts, absolute saving and percentage saving. ### Formula — which one and why **Multiply–accumulate count.** Each output position and output channel uses K²C\_in products. A bottleneck requires two separate counts, one for the 1×1 reduction and one for the following spatial convolution. H\_out,W\_out = output map size; C\_in,C\_out = incoming/outgoing channels; K = kernel width. Count one multiply–accumulate as one MAC; bias additions are excluded. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ **Yaad rakhne ka cue:** Har output cell ka kaam × kitne output cells. ### Solution — step by step ### Meaning and formula A MAC is one multiply–accumulate operation. For the convention used here, count the weight–input products over every output location. Bias additions are omitted. A 1 × 1 bottleneck reduces channels before the expensive 3 × 3 layer. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ ### Step 1: direct 3 × 3 convolution Each output uses 3 × 3 × 64 = 576 weight–input pairs. There are 16 × 16 × 32 = 8,192 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 reduction from 64 to 8 channels $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ ### Step 3: 3 × 3 convolution now receives only 8 channels $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: add the two bottleneck stages $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: calculate the saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct 4,718,592 MACs; bottleneck 720,896 MACs; approximately 84.72% fewer. The output shape stays 16 × 16 × 32. ### Bottleneck computation on an 8 × 8 input **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8 in both stages. Input channels = 16; output filters = 8; main kernel = 3 × 3. Bottleneck uses 4 filters of 1 × 1. ### To find Direct MACs, two-stage bottleneck MACs and savings. ### Formula — which one and why Each output scalar requires one multiply-accumulate per incoming kernel entry. A bottleneck reduces the input depth of the expensive spatial convolution. One MAC is one multiply-accumulate pair. If counting multiplication and addition separately, the operation count is approximately twice the MAC count. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K_hK_wC_{\mathrm{in}} $$ **Yaad rakhne ka cue:** Spatial kernel se pehle channels kam; dono stages ka cost add. ### Solution — step by step **Step 1: count output positions per channel.** $$ 8(8)=64 $$ **Step 2: count direct work per output scalar.** $$ 3(3)(16)=144 $$ **Step 3: count the full direct layer.** $$ 64(8)(144)=73728 $$ **Step 4: count the 1 × 1 reduction.** $$ 64(4)(1)(1)(16)=4096 $$ **Step 5: count the reduced-depth 3 × 3 layer.** $$ 64(8)(3)(3)(4)=18432 $$ **Step 6: sum both bottleneck stages.** $$ 4096+18432=22528 $$ **Step 7: subtract to obtain savings.** $$ 73728-22528=51200 $$ **Final answer:** Direct = 73,728 MACs; bottleneck = 22,528 MACs; savings = 51,200 MACs. ## Concatenation and residual addition Concatenation and addition combine branches differently. Concatenation places channel groups next to each other and increases the channel count. Residual addition adds matching entries, so the two full shapes must be compatible. $$ C_{concat}=\sum_b C_b $$ $$ y_{res}=F(x)+x $$ ## Inception concatenation and residual addition **Definition.** Channel concatenation joins feature tensors while preserving their separate channels. Residual addition sums corresponding entries. These operations impose different shape constraints and produce different output depths. ### Explanation and working Concatenation joins feature channels; elementwise addition sums matching entries. They are different operations. ![Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.](https://deep-learning-study-room.vercel.app/assets/figures/v_join_add.svg) Diagram: Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape.. Labels: Concatenation and addition produce different shapes | Concatenation and addition produce different shapes | Use height and width 14 × 14 throughout. | CONCATENATE four branches | Channels: 16 + 32 + 8 + 8 | Result: 14 × 14 × 64 | Stack all 64 channels. | ADD two matching tensors | 14 × 14 × 64 + 14 × 14 × 64 | Result: 14 × 14 × 64 | Add each matching pair. | A 14 × 14 × 32 shortcut needs a projection before addition to a 14 × 14 × 64 | branch. Concatenation stacks channels. Residual addition combines matching values and preserves the matched shape. Tap the diagram to enlarge. Four Inception branches of spatial size 14 × 14 have 16, 32, 8 and 8 channels. First add the channel counts: 16 + 32 = 48; 48 + 8 = 56; 56 + 8 = 64. Height and width stay 14. The concatenated output is 14 × 14 × 64. A residual addition requires matching shapes. A 14 × 14 × 64 tensor cannot be directly added to a 14 × 14 × 32 tensor. A projection, often a 1 × 1 convolution, can transform one path to the matching shape before addition. ## ResNet adds a shortcut **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x. $$ y=F(x)+x $$ If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch. ![The two paths must have equal output shape before elementwise addition.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge. ### Effect on gradient propagation For the identity-shortcut formulation, the derivative includes a direct identity contribution. $$ \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I $$ This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients. ### Projection when dimensions differ If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride. $$ y=F(x)+W_sx $$ An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition. **Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum. ### Given x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3]. ### To find Find the three components of x+F(x). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. ### Solution — step by step **Step 1: add the first matching pair.** $$ 1+0.1=1.1 $$ **Step 2: add the second matching pair.** $$ 2+(-0.2)=1.8 $$ **Step 3: add the third matching pair.** $$ 3+0.3=3.3 $$ **Step 4: assemble in the original order.** $$ y=[1.1,1.8,3.3] $$ The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum. ## Drill 51 Inception concatenation and parameter count ## Question **Question.** Input 10 × 10 × 8. Parallel branches: A uses 1 × 1 conv 4 filters ; B uses 3 × 3 same Conv 6 filters; C max pool 3 same, stride 1 then 1 × 1 conv 2 filters. Find combined shape and trainable parameters. ## Answer — step by step ### Given Input = 10 × 10 × 8. Branch A: 1 × 1 convolution, 4 filters. Branch B: 3 × 3 same convolution, 6 filters. Branch C: 3 × 3 same max pool with stride 1, then 1 × 1 convolution, 2 filters. ### To find Find each branch shape and parameters, then the concatenated shape and total parameters. ### Formula — which one and why **Inception channel concatenation.** Parallel branches retain their separate features. Add their channel counts while keeping the common height and width. Add trainable parameters across branches. C₁,C₂,… = branch output channels. Concatenation requires matching spatial dimensions. $$ C_{\mathrm{out}}=\sum_k C_k $$ $$ P_{\mathrm{total}}=\sum_k P_k $$ **Yaad rakhne ka cue:** Channels jodo; height-width same rakho. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Each parallel branch receives the same original 10 × 10 × 8 input. “Same” with stride 1 preserves height and width. Concatenation joins the channel lists without adding the feature values. $$ C_{\mathrm{combined}}=C_A+C_B+C_C $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ Assume one bias per filter. Pooling has no learned parameters. **Step 1: Branch A.** Its 4 filters produce shape 10 × 10 × 4. $$ P_A=(1\times1\times8+1)\times4=9\times4=36 $$ **Step 2: Branch B.** Its 6 filters produce shape 10 × 10 × 6. $$ P_B=(3\times3\times8+1)\times6=73\times6=438 $$ **Step 3: Branch C.** Pooling preserves 10 × 10 × 8 here because stride is 1 and padding is same. Its parameter count is zero. The following 2-filter projection creates 10 × 10 × 2. $$ P_C=0+(1\times1\times8+1)\times2=18 $$ **Step 4: Join the shapes and add the parameters.** $$ C_{\mathrm{combined}}=4+6+2=12 $$ $$ P_{\mathrm{total}}=36+438+18=492 $$ **Final answer:** combined shape 10 × 10 × 12; 492 parameters. Branch B does not take branch A's four channels as input: they are parallel branches. ## Drill 52 Projection shortcut ## Question **Question.** A residual main path outputs 8 × 8 × 16 but input is 16 × 16 × 8. Design a 1 × 1 projection shortcut with stride 2, zero padding. Find its output and parameters. ## Answer — step by step ### Given Residual input = 16 × 16 × 8; main-path output = 8 × 8 × 16. Shortcut kernel = 1 × 1; stride 2; padding 0. Use one bias per projection filter. ### To find Choose the number of filters and find projection output shape and parameters. ### Formula — which one and why **Convolution/pooling spatial size.** Padding adds P positions at both ends. Subtract kernel width to find the last valid start, divide by stride, take the floor and include the initial placement with +1. N = current height or width; K = kernel/window size; S = stride; P = padding per side. Apply separately to height and width; dilation is 1 here. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ **Yaad rakhne ka cue:** Input + dono sides ki padding − kernel; stride se divide; floor + 1. **Convolution parameters.** Every filter spans all input channels. Count its spatial coefficients across those channels, add one bias and multiply by the number of filters. K\_h,K\_w = filter height/width; C\_in = input channels; C\_out = number of filters and output channels. Spatial output positions reuse these weights. $$ N_w=K_hK_wC_{\mathrm{in}}C_{\mathrm{out}} $$ $$ N_b=C_{\mathrm{out}} $$ $$ P=N_w+N_b $$ **Yaad rakhne ka cue:** Filter area × input channels × filters; phir ek bias per filter. ### Solution — step by step ### Meaning and formulas Residual addition needs identical shapes on both paths. The projection must reduce height and width from 16 to 8 and increase channels from 8 to 16. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{projection}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ ### Step 1: choose 16 filters One filter creates one channel, so 16 filters give the required 16 output channels. ### Step 2: use the specified 1 × 1 kernel and stride 2 $$ N_{\mathrm{out}}=\left\lfloor\frac{16+0-1}{2}\right\rfloor+1 $$ $$ N_{\mathrm{out}}=\lfloor7.5\rfloor+1=7+1=8 $$ The floor operation drops the fractional part; it does not round 7.5 to 8 before adding 1. ### Step 3: count weights and biases $$ \text{weights}=1\times1\times8\times16=128 $$ $$ \text{biases}=16,\qquad \text{total}=128+16=144 $$ **Final answer:** projection output 8 × 8 × 16 and 144 parameters with biases. The projected shortcut now matches the main path, so elementwise addition is valid. ## Drill 40 Residual addition ## Question **Question.** x = [2,−3,1], F(x) = [−1,1,0.5]. Calculate residual output before and after ReLU and compare the plain block. ## Answer — step by step ### Given x = [2,−3,1]; F(x) = [−1,1,0.5]. ### To find Find x+F(x), apply ReLU, and compare with ReLU(F(x)). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step ### Meaning and formulas The main branch produces F(x). The shortcut carries x. A residual join adds matching entries from these equally sized vectors. ReLU is a separate operation after that join. $$ H(x)=x+F(x),\qquad a_j=\max(0,H_j) $$ ### Step 1: add each pair $$ H_1=2+(-1)=1 $$ $$ H_2=-3+1=-2 $$ $$ H_3=1+0.5=1.5 $$ The residual output before ReLU is **[1, −2, 1.5]**. ### Step 2: apply ReLU $$ [\max(0,1),\max(0,-2),\max(0,1.5)]=[1,0,1.5] $$ ### Step 3: compare with the plain branch Without a shortcut, the raw output is just F(x) = [−1, 1, 0.5]. Its ReLU output is **[0, 1, 0.5]**. Residual addition keeps three entries. Concatenation would join the vectors into six entries and is a different operation. ## Vanilla RNN A recurrent state uses two sources: the current input and the previous hidden state. Transform each with its own weight, add the contributions and bias, and apply tanh. The activated state—not its raw sum—is carried into the next time step. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h) $$ $$ o_t=W_{hy}h_t+b_y $$ ## The vanilla RNN equations from scratch **Definition.** A vanilla recurrent neural network updates a hidden state from the current sequence input and the previous hidden state, using the same trainable parameters at every time step. An output transformation converts that state into the task's prediction. ### Hidden-state and output equations For a tanh recurrent layer: $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b_h $$ $$ h_t=\tanh(a_t) $$ aₜ is the preactivation and hₜ is the activated hidden state. Wₓₕ transforms the new input; Wₕₕ transforms the previous state. Both contributions have the same hidden dimension before they are added. bₕ has one component per hidden unit. For classification at time t, an output layer can produce logits and probabilities: $$ z_t=W_{hy}h_t+b_y,\qquad p_t=\operatorname{softmax}(z_t) $$ A regression output may instead be linear. A question asking only for hidden states does not require an output-layer prediction, and such a prediction cannot be calculated without the output transformation. ![The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.](https://deep-learning-study-room.vercel.app/assets/figures/v_rnn_state.svg) Diagram: The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state.. Labels: Combine the current input and the previous state | Combine the current input and the previous state | The new state depends on both paths. | Previous state | h at time t − 1 | Current input | x at time t | Combine | Weighted sum + | bias | Activate | New state h at t | The same recurrent weight matrices are used again at the next time step. The current input and previous activated state enter different weight transformations. Their sum is activated to obtain the next state. Tap the diagram to enlarge. ### Complete numerical: two recurrent time steps **Question.** Let h₀ = 0, Wₓₕ = 0.5, Wₕₕ = 0.2, bₕ = 0, x₁ = 1 and x₂ = 2. Calculate h₁ and h₂ using tanh activation. ### Given h₀ = 0; Wxh = 0.5; Whh = 0.2; b = 0. x₁ = 1; x₂ = 2; activation = tanh. ### To find Find h₁ and h₂ in order. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step **Step 1: Time step 1 — calculate the preactivation.** $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ **Apply the recurrent activation.** $$ h_1=\tanh(0.5)\approx0.462117157 $$ This activated value, not a₁ = 0.5, is passed to the next time step. **Step 2: Time step 2 — combine the new input and previous state.** $$ a_2=0.5(2)+0.2h_1+0 $$ $$ a_2\approx1+0.2(0.462117157)\approx1.092423431 $$ **Apply tanh to the complete new preactivation.** $$ h_2=\tanh(a_2)\approx0.797760954 $$ | Time step | Current input | Previous state | Preactivation | New state | | --- | --- | --- | --- | --- | | 1 | 1 | 0 | 0.5 | 0.462117157 | | 2 | 2 | 0.462117157 | 1.092423431 | 0.797760954 | **Final answer:** h₁ ≈ 0.462117 and h₂ ≈ 0.797761. Keep the full calculator value of h₁ while computing the second step. Although a₂ is greater than 1, h₂ is between −1 and 1 because tanh is applied after the affine sum. Both time steps use the same weights; this calculation contains no parameter update. ## A complete vector RNN calculation **Question from R26 Q5a.** xₜ = [0,1], hₜ₋₁ = [0,1], no bias is supplied, Wxh is shown below, Whh is shown below. Use tanh and rows as receiving neurons. ### Given Input x = [0,1]ᵀ; previous h = [0,1]ᵀ. Wxh = $\begin{bmatrix}0.6 & 0.4\\0.3 & 0.7\end{bmatrix}$ . Whh = $\begin{bmatrix}0.5 & 0.3\\0.2 & 0.4\end{bmatrix}$ ; bias = 0. Apply tanh separately to the two sums. ### To find Find the new two-component hidden state; output-layer parameters are not supplied. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_deac39f14c.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | Wxh | 0.6 | 0.4 | 0.3 | 0.7 | Whh | 0.5 | 0.3 | 0.2 | 0.4 Given matrices for this question Tap the diagram to enlarge. **Step 1 new input:** Wxh x = [0.6 × 0 + 0.4 × 1, 0.3 × 0 + 0.7 × 1] = [0.4,0.7]. **Step 2 old memory:** Whh hprevious = [0.5 × 0 + 0.3 × 1, 0.2 × 0 + 0.4 × 1] = [0.3,0.4]. **Step 3 add:** a = [0.4+0.3, 0.7+0.4] = [0.7,1.1]. **Step 4 activate each element:** h = [tanh(0.7), tanh(1.1)] = **[0.604368,0.800499]**. An output-layer prediction cannot be numerically calculated without its weights and bias. ## Drill 41 Two RNN steps ## Question **Question.** h₀ = 0, Wxh = 0.5, Whh = 0.2, b = 0, x₁ = 1, x₂ = 2, tanh activation. Find h₁, h₂. ## Answer — step by step ### Given Initial h₀ = 0; input weight Wxh = 0.5; recurrent weight Whh = 0.2; b = 0. Inputs: x₁ = 1, x₂ = 2; activation: tanh. ### To find Calculate h₁, then use it to calculate h₂. ### Formula — which one and why **Recurrent forward pass.** At each time step, combine the current input contribution with the previous activated hidden state, add the bias and apply tanh. The same weights are reused at every time step. x\_t = current input; h\_(t−1) = previous hidden state; W\_xh = input weights; W\_hh = recurrent weights; b = bias. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b $$ $$ h_t=\tanh(a_t) $$ **Yaad rakhne ka cue:** Naya input + pichhla activated state; phir activation. ### Solution — step by step ### Meaning and formulas At time t, the RNN combines the current input xₜ and the previous hidden state hₜ₋₁. The same two weights are reused at both times. $$ a_t=W_{xh}x_t+W_{hh}h_{t-1}+b,\qquad h_t=\tanh(a_t) $$ The raw sum a is not the hidden state. Tanh converts that sum into the state that is carried forward. $$ \tanh(a)=\frac{e^{2a}-1}{e^{2a}+1} $$ ### Step 1: calculate the first raw sum $$ a_1=0.5(1)+0.2(0)+0=0.5 $$ ### Step 2: activate it to obtain the first state $$ h_1=\tanh(0.5)=\frac{e^1-1}{e^1+1} $$ $$ h_1=\frac{2.718281828-1}{2.718281828+1}=0.462117157 $$ ### Step 3: use that state in the second raw sum $$ a_2=0.5(2)+0.2(0.462117157)+0 $$ $$ a_2=1+0.092423431=1.092423431 $$ ### Step 4: activate the second sum $$ h_2=\tanh(1.092423431)=0.797760954 $$ **Final answer:** h₁ ≈ 0.462117157; h₂ ≈ 0.797760954. Pass h₁ to the next step, not the unactivated value 0.5. ## Perceptron update The perceptron uses a threshold to choose a class and an error-based learning rule to change weights. Its threshold convention must be stated, especially at zero. This discrete learning rule is different from sigmoid backpropagation. $$ \widehat y=\operatorname{step}(w^Tx+b) $$ $$ w_{new}=w+\eta(y-\widehat y)x $$ $$ b_{new}=b+\eta(y-\widehat y) $$ ## Perceptrons and the threshold A perceptron is a simple neuron used for classification with a threshold decision. If a question says output 1 for z ≥ 0 and 0 otherwise, then z = 0 gives 1. If it says output 1 for z > 0 and 0 otherwise, then z = 0 gives 0. That single equality sign can change an entire training table. ![The weighted sum is a real number. The threshold turns that number into a hard decision.](https://deep-learning-study-room.vercel.app/assets/figures/v_threshold.svg) Diagram: The weighted sum is a real number. The threshold turns that number into a hard decision.. Labels: A perceptron makes a threshold decision | A perceptron makes a threshold decision | Example threshold: return 1 when the weighted sum is greater than 0. | Inputs | x₁ and x₂ | Weighted sum | z = w₁x₁ + w₂x₂ + b | Threshold | z > 0: 1; otherwise: | 0 | Unlike sigmoid, this rule makes a hard 0-or-1 decision. The weighted sum is a real number. The threshold turns that number into a hard decision. Tap the diagram to enlarge. **Question.** Implement binary AND with w₁ = w₂ = 0.6, b = −0.8 and output 1 when z ≥ 0. ### Given w₁ = w₂ = 0.6; bias b = −0.8. Binary inputs; output 1 when z ≥ 0. ### To find Calculate the output for 00,01,10,11 and show that the result is AND. ### Formula — which one and why **Threshold logic.** Compute the weighted sum for every binary input combination and apply the threshold rule exactly. Hidden neurons can provide separate threshold decisions before the output combines them. step(z) returns 1 at or above zero unless the question specifies a strict inequality; otherwise it returns 0. $$ z=\sum_iw_ix_i+b $$ $$ \operatorname{step}(z)=\begin{cases}1&z\ge0\\0&z<0\end{cases} $$ **Yaad rakhne ka cue:** Har input row ka sum nikalo, phir threshold lagao. ### Solution — step by step ### Input (0, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0+0=0 $$ **Step 4: add the bias once.** $$ z=0+(-0.8)=-0.8 $$ **Step 5: apply the stated threshold.** z = -0.8; output = 0 because output is 1 exactly when z ≥ 0. ### Input (0, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(0)=0 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0+0.6=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 0) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(0)=0 $$ **Step 3: add the two products.** $$ s=0.6+0=0.6 $$ **Step 4: add the bias once.** $$ z=0.6+(-0.8)=-0.2 $$ **Step 5: apply the stated threshold.** z = -0.2; output = 0 because output is 1 exactly when z ≥ 0. ### Input (1, 1) **Step 1: multiply input 1 by its weight.** $$ 0.6(1)=0.6 $$ **Step 2: multiply input 2 by its weight.** $$ 0.6(1)=0.6 $$ **Step 3: add the two products.** $$ s=0.6+0.6=1.2 $$ **Step 4: add the bias once.** $$ z=1.2+(-0.8)=0.4 $$ **Step 5: apply the stated threshold.** z = 0.4; output = 1 because output is 1 exactly when z ≥ 0. **Final answer:** outputs are [0, 0, 0, 1] for inputs 00, 01, 10, 11: exactly AND. ### Summary of the four cases | Inputs | Full net calculation | Net z | Output | AND target | | --- | --- | --- | --- | --- | | 0, 0 | 0.6(0) + 0.6(0) − 0.8 | −0.8 | 0 | 0 | | 0, 1 | 0.6(0) + 0.6(1) − 0.8 | −0.2 | 0 | 0 | | 1, 0 | 0.6(1) + 0.6(0) − 0.8 | −0.2 | 0 | 0 | | 1, 1 | 0.6(1) + 0.6(1) − 0.8 | 0.4 | 1 | 1 | The neuron fires only when both inputs are 1. No weight update was requested, so this is a forward evaluation, not a training problem. For binary targets, a common learning rule is w ← w + η(t − a)x and b ← b + η(t − a). For bipolar targets −1 and +1, the class notes use a different rule: on a mismatch, w ← w + η tx and b ← b + ηt; otherwise keep them. The notes give output 0 at zero threshold, +1 above it and −1 below it. Never combine the activation from one convention with an unexplained update from another. **Training table routine:** copy current weights → calculate z → apply the stated threshold → compare output with target → compute every weight change → update bias → carry new values to the next row. Reset nothing between rows or epochs unless instructed. ### Binary perceptron: target 1 **Question. A binary perceptron has x = [1,2], weights [-0.2, 0.1], bias 0 and target 1. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.** ### Given x = [1,2]; w = [-0.2, 0.1]; b = 0; target = 1; η = 0.1. ### To find Net, threshold output, correction and new weights/bias. ### Formula — which one and why This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias. t is target; o is threshold output; Δ means the amount added to the current parameter. $$ z=\sum_iw_ix_i+b $$ $$ \Delta w_i=\eta(t-o)x_i $$ $$ \Delta b=\eta(t-o) $$ **Yaad rakhne ka cue:** Target minus output; correct prediction ho toh correction zero. ### Solution — step by step **Step 1: calculate the net input.** $$ z=(-0.2)(1)+(0.1)(2)+0=0 $$ **Step 2: apply the threshold.** Net = 0; output = 1 under the stated ≥ 0 rule. **Step 3: calculate the correction.** $$ t-o=1-1=0 $$ **Step 4: update weight 1.** $$ w_1^{\mathrm{new}}=-0.2+0.1(0)(1)=-0.2 $$ **Step 5: update weight 2.** $$ w_2^{\mathrm{new}}=0.1+0.1(0)(2)=0.1 $$ **Step 6: update the bias.** $$ b_{\mathrm{new}}=0+0.1(0)=0 $$ **Final answer:** New weights = [-0.2,0.1]; new bias = 0. ### Binary perceptron: target 0 **Question. A binary perceptron has x = [1,2], weights [0.2, 0.1], bias 0 and target 0. Use output 1 for net ≥ 0 and learning rate 0.1. Perform one update.** ### Given x = [1,2]; w = [0.2, 0.1]; b = 0; target = 0; η = 0.1. ### To find Net, threshold output, correction and new weights/bias. ### Formula — which one and why This binary learning rule uses target minus output. Carry the same correction into every input weight and the bias. t is target; o is threshold output; Δ means the amount added to the current parameter. $$ z=\sum_iw_ix_i+b $$ $$ \Delta w_i=\eta(t-o)x_i $$ $$ \Delta b=\eta(t-o) $$ **Yaad rakhne ka cue:** Target minus output; correct prediction ho toh correction zero. ### Solution — step by step **Step 1: calculate the net input.** $$ z=(0.2)(1)+(0.1)(2)+0=0.4 $$ **Step 2: apply the threshold.** Net = 0.4; output = 1 under the stated ≥ 0 rule. **Step 3: calculate the correction.** $$ t-o=0-1=-1 $$ **Step 4: update weight 1.** $$ w_1^{\mathrm{new}}=0.2+0.1(-1)(1)=0.1 $$ **Step 5: update weight 2.** $$ w_2^{\mathrm{new}}=0.1+0.1(-1)(2)=-0.1 $$ **Step 6: update the bias.** $$ b_{\mathrm{new}}=0+0.1(-1)=-0.1 $$ **Final answer:** New weights = [0.1,-0.1]; new bias = -0.1. ## Drill 11 One perceptron learning update ## Question **Question.** Binary perceptron, w = [0,0], b = 0, x = [1,1], target 1, η = 0.5. Use output 1 if z>0 else 0. Perform one update. ## Answer — step by step ### Given w = [0,0]; b = 0; x = [1,1]; target t = 1. Learning rate η = 0.5. Output 1 only when z > 0; otherwise output 0. ### To find Perform one update and report new weights and bias. ### Formula — which one and why **Perceptron learning rule.** Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters. t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question. $$ z=\sum_iw_ix_i+b $$ $$ w_i^{\mathrm{new}}=w_i+\eta(t-o)x_i $$ $$ b^{\mathrm{new}}=b+\eta(t-o) $$ **Yaad rakhne ka cue:** Prediction nikalo; target−prediction se correction nikalo. ### Solution — step by step ### Meaning and formula A perceptron first computes a weighted sum z and then applies the exact threshold rule in the question. This question uses **strictly greater than zero**, so z = 0 produces output 0. $$ z=w_1x_1+w_2x_2+b,\qquad o=\begin{cases}1&z>0\\0&z\le0\end{cases} $$ The perceptron learning rule uses target minus output, written e below. This is a discrete perceptron rule, not the sigmoid backpropagation derivative. $$ e=t-o,\qquad w_{j,\mathrm{new}}=w_j+\eta e x_j,\qquad b_{\mathrm{new}}=b+\eta e $$ ### Step 1: obtain the old output $$ z=0\times1+0\times1+0=0,\qquad o=0 $$ ### Step 2: compare it with the target $$ e=1-0=1 $$ ### Step 3: update each weight separately $$ w_{1,\mathrm{new}}=0+0.5\times1\times1=0.5 $$ $$ w_{2,\mathrm{new}}=0+0.5\times1\times1=0.5 $$ $$ b_{\mathrm{new}}=0+0.5\times1=0.5 $$ ### Step 4: interpret the updated prediction $$ z_{\mathrm{new}}=0.5\times1+0.5\times1+0.5=1.5 $$ Since 1.5 > 0, the new output is 1. **Final parameters: w = [0.5, 0.5], b = 0.5.** ## BPTT hidden delta At an RNN hidden state, a loss gradient can arrive from the current output and from the next time step. Add these two contributions, then multiply by the local activation derivative. The displayed output term p−y assumes softmax with categorical cross-entropy. $$ \delta_t=\left[W_{hy}^T(p_t-y_t)+W_{hh}^T\delta_{t+1}\right]\odot(1-h_t^2) $$ ## How an RNN is trained **Definition.** Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters. ### Explanation and working Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT. ![Unroll forward through time, then accumulate backward contributions for the shared parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_bptt.svg) Diagram: Unroll forward through time, then accumulate backward contributions for the shared parameters.. Labels: One shared weight receives contributions from many times | One shared weight receives contributions from many times | Unrolling draws repeated uses of one cell, not independent copies of its | weights. | Time 1 | Same weights W | Time 2 | Same weights W | Time 3 | Same weights W | Gradients flow backward through the state dependencies | Add shared-weight contributions | Total gradient = contribution 1 + contribution 2 + | contribution 3 | After summing the required contributions, update each shared parameter once. Unroll forward through time, then accumulate backward contributions for the shared parameters. Tap the diagram to enlarge. For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero. $$ \delta_t=\left(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1}\right)\odot(1-h_t^2) $$ $$ \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T},\quad \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^{T},\quad \frac{\partial L}{\partial b_h}=\sum_t\delta_t $$ The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters. ### One BPTT delta at h = 0.5 **Question. In a scalar tanh RNN, h\_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.** ### Given Activated h\_t = 0.5; q\_t = 0.2; δ\_(t+1) = 0.3; recurrent weight u = 0.4. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.4(0.3)=0.12 $$ **Step 2: add the current output contribution.** $$ 0.2+0.12=0.32 $$ **Step 3: calculate the current tanh slope.** $$ 1-(0.5)^2=0.75 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.32)(0.75)=0.24 $$ **Final answer:** δ\_t = 0.24. ### One BPTT delta at h = -0.2 **Question. In a scalar tanh RNN, h\_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.** ### Given Activated h\_t = -0.2; q\_t = 0.1; δ\_(t+1) = 0; recurrent weight u = 0.5. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.5(0)=0 $$ **Step 2: add the current output contribution.** $$ 0.1+0=0.1 $$ **Step 3: calculate the current tanh slope.** $$ 1-(-0.2)^2=0.96 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.1)(0.96)=0.096 $$ **Final answer:** δ\_t = 0.096. **Question. In a scalar tanh RNN, the current hidden state is hₜ = 0.6. The current output path contributes 0.1 to the derivative with respect to hₜ. The next hidden preactivation has delta 0.4 and recurrent weight 0.5. Find the current hidden preactivation delta.** ### Given Current hₜ = 0.6; local output-path derivative = 0.1. Next-step delta = 0.4; recurrent weight = 0.5; current activation = tanh. ### To find Find the current hidden preactivation delta. ### Formula — which one and why **Current and future recurrent paths.** Add the local-output contribution and the weighted next-step delta, then multiply by the current tanh slope. hₜ = current activated state; qₜ = current output-path derivative; u = recurrent weight; δₜ₊₁ = next-step preactivation delta. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Current path plus future path, phir current activation slope. ### Solution — step by step The next-step contribution must travel through the recurrent weight. Multiply that weight by the supplied next delta. $$ \text{future contribution}=0.5(0.4)=0.2 $$ Add the current output-path contribution. These are two routes to the same hidden state, so addition is required. $$ \frac{\partial L}{\partial h_t}=0.1+0.2=0.3 $$ The current activation is tanh. Its derivative uses the current activated value hₜ. $$ 1-h_t^2=1-(0.6)^2=1-0.36=0.64 $$ Multiply the combined gradient by this local derivative. $$ \delta_t=0.3(0.64)=0.192 $$ **Final answer: δₜ = 0.192.** This is the derivative with respect to the current hidden raw score. At the last step, there is no future-state contribution unless the question connects another computation there. ### One BPTT delta at h = 0.5 **Question. In a scalar tanh RNN, h\_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.** ### Given Activated h\_t = 0.5; q\_t = 0.2; δ\_(t+1) = 0.3; recurrent weight u = 0.4. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.4(0.3)=0.12 $$ **Step 2: add the current output contribution.** $$ 0.2+0.12=0.32 $$ **Step 3: calculate the current tanh slope.** $$ 1-(0.5)^2=0.75 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.32)(0.75)=0.24 $$ **Final answer:** δ\_t = 0.24. ### One BPTT delta at h = -0.2 **Question. In a scalar tanh RNN, h\_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.** ### Given Activated h\_t = -0.2; q\_t = 0.1; δ\_(t+1) = 0; recurrent weight u = 0.5. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.5(0)=0 $$ **Step 2: add the current output contribution.** $$ 0.1+0=0.1 $$ **Step 3: calculate the current tanh slope.** $$ 1-(-0.2)^2=0.96 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.1)(0.96)=0.096 $$ **Final answer:** δ\_t = 0.096. ## BPTT recurrent-weight gradient The same recurrent weight is reused at several times. Each use contributes to its gradient, so sum all required time-step contributions before updating it. For vectors, delta is a column and the previous state is transposed into a row: their outer product has the same shape as the weight matrix. $$ g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right) $$ $$ \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^T $$ $$ \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^T $$ ## Vanishing and exploding gradients **Definition.** Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment. ### Explanation and working In BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity. ![Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.](https://deep-learning-study-room.vercel.app/assets/figures/v_vanishing.svg) Diagram: Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.. Labels: Repeated multiplication changes gradient size | Repeated multiplication changes gradient size | These scalar examples illustrate one part of the BPTT mechanism. | Repeated factors below 1 | 0.5 × 0.5 × … × 0.5 | Ten factors: 0.0009765625 | Earlier learning signal is tiny. | Repeated large factors | 2 × 2 × … × 2 | Ten factors: 1,024 | Updates can become unstable. | Gradient clipping limits explosion. It does not rebuild a gradient that has | already vanished. Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates. Tap the diagram to enlarge. **Numerical intuition:** 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs. Useful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory. ### Why the effect grows with sequence length For a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians: $$ \frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(1-h_t^2)W_{hh} $$ $$ \frac{\partial h_T}{\partial h_t}=\prod_{k=T}^{t+1}\frac{\partial h_k}{\partial h_{k-1}} $$ The product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone. ### Gradient norm clipping: a complete numerical Norm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient. $$ g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right) $$ If g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation. **Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.** ### Given Gradient g = [6,8]ᵀ; norm threshold τ = 5. Old weights w = [1,1]ᵀ; η = 0.1. ### To find Find the clipped gradient and updated weights. ### Formula — which one and why **Gradient norm clipping.** If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction. g = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate. $$ \lVert g\rVert_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min\left(1,\frac\tau{\lVert g\rVert_2}\right) $$ $$ w_{\mathrm{new}}=w-\eta g_{\mathrm{clip}} $$ **Yaad rakhne ka cue:** Bada gradient? Sab components ko ek hi factor se chhota karo. ### Solution — step by step **Step 1: calculate the original norm.** $$ \lVert g\rVert_2=\sqrt{6^2+8^2}=\sqrt{36+64}=10 $$ **Step 2: find the scale.** Since 10 > 5, scale the vector by 5/10 = 0.5. $$ g_{\mathrm{clipped}}=0.5\begin{bmatrix}6\\8\end{bmatrix}=\begin{bmatrix}3\\4\end{bmatrix} $$ **Step 3: verify the clipped norm.** $$ \sqrt{3^2+4^2}=\sqrt{9+16}=5 $$ **Step 4: update each parameter with the clipped gradient.** $$ w_{1,\mathrm{new}}=1-0.1(3)=0.7 $$ $$ w_{2,\mathrm{new}}=1-0.1(4)=0.6 $$ **Final answer:** clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture. ### Gradient norm clipping: [3, 4] **Question. Clip gradient g = [3, 4] to maximum L2 norm 10. Use global norm clipping.** ### Given g = [3, 4]; threshold τ = 10. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 3^2+4^2=25 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{25}=5 $$ **Step 3: choose the shared scale.** $$ c=\min(1,10/5)=1 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=3(1)=3 $$ $$ g_2^{\mathrm{clip}}=4(1)=4 $$ **Final answer:** Clipped gradient = [3, 4]. ### Gradient norm clipping: [0, 12] **Question. Clip gradient g = [0, 12] to maximum L2 norm 3. Use global norm clipping.** ### Given g = [0, 12]; threshold τ = 3. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 0^2+12^2=144 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{144}=12 $$ **Step 3: choose the shared scale.** $$ c=\min(1,3/12)=0.25 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=0(0.25)=0 $$ $$ g_2^{\mathrm{clip}}=12(0.25)=3 $$ **Final answer:** Clipped gradient = [0.0, 3.0]. ## Drill 45 BPTT shared weights ## Question **Question.** Linear recurrence hₜ = uxₜ+vhₜ₋₁+b, output h₂, target 1, half squared loss. u = 0.2, v = 0.5, b = 0, h₀ = 0, x₁ = 1, x₂ = 2, η = 0.1. Find both states and one update. ## Answer — step by step ### Given Linear recurrence hₜ = uxₜ+vhₜ₋₁+b. u = 0.2; v = 0.5; b = 0; h₀ = 0. x₁ = 1; x₂ = 2; target for h₂ = 1. Half squared loss; learning rate η = 0.1. ### To find Find both states, all three shared-parameter gradients and one simultaneous update. ### Formula — which one and why **Backpropagation through time.** Unroll the recurrence, send the loss derivative backward through its links and add contributions for each shared parameter before updating it once. For h\_t=ux\_t+vh\_(t−1)+b with loss only at the last state: δ\_t = loss derivative at that state. The linear recurrence has activation slope 1. $$ \delta_T=h_T-y $$ $$ \delta_t=v\delta_{t+1} $$ $$ g_u=\sum_t\delta_tx_t $$ $$ g_v=\sum_t\delta_th_{t-1} $$ $$ g_b=\sum_t\delta_t $$ **Yaad rakhne ka cue:** Time ke har use ka gradient jodo; shared weight ko ek baar update karo. **Gradient descent.** Evaluate the gradient at the current parameter, multiply by the learning rate and subtract. If the objective is supplied, differentiate it and recompute the gradient after each update. θ (theta) = trainable parameter; g = derivative of the stated objective J; η = learning rate; t labels the update number. $$ g_t=J\prime(\theta_{t-1}) $$ $$ \theta_t=\theta_{t-1}-\eta g_t $$ **Yaad rakhne ka cue:** Naya = purana − learning rate × current gradient. ### Solution — step by step ### Meaning and formulas The same u, v and b appear at both times. u multiplies the input, v multiplies the previous state, and b is a shared bias. The loss is only at the final state h₂. Because the recurrence is linear, its local activation derivative is 1. $$ h_t=ux_t+vh_{t-1}+b,\qquad L=\frac12(h_2-y)^2 $$ $$ \delta_2=h_2-y,\qquad \delta_1=v\delta_2 $$ Each shared parameter receives a contribution from both of its uses. $$ \frac{\partial L}{\partial u}=\delta_1x_1+\delta_2x_2 $$ $$ \frac{\partial L}{\partial v}=\delta_1h_0+\delta_2h_1,\qquad \frac{\partial L}{\partial b}=\delta_1+\delta_2 $$ ### Step 1: forward through time 1 $$ h_1=0.2(1)+0.5(0)+0=0.2 $$ ### Step 2: forward through time 2 $$ h_2=0.2(2)+0.5(0.2)+0=0.4+0.1=0.5 $$ $$ L=\frac12(0.5-1)^2=0.125 $$ ### Step 3: carry the gradient backwards $$ \delta_2=0.5-1=-0.5 $$ $$ \delta_1=0.5(-0.5)=-0.25 $$ ### Step 4: add contributions for each shared parameter $$ \frac{\partial L}{\partial u}=(-0.25)(1)+(-0.5)(2)=-0.25-1=-1.25 $$ $$ \frac{\partial L}{\partial v}=(-0.25)(0)+(-0.5)(0.2)=0-0.1=-0.1 $$ $$ \frac{\partial L}{\partial b}=-0.25-0.5=-0.75 $$ ### Step 5: update once after summing $$ u_{\mathrm{new}}=0.2-0.1(-1.25)=0.325 $$ $$ v_{\mathrm{new}}=0.5-0.1(-0.1)=0.51 $$ $$ b_{\mathrm{new}}=0-0.1(-0.75)=0.075 $$ **Final answer:** states 0.2 and 0.5; new parameters u = 0.325, v = 0.51, b = 0.075. ## LSTM gates and candidate An LSTM gate is a multiplier between zero and one. The forget gate chooses the fraction of old cell content retained; the input gate chooses the fraction of candidate content admitted. Each gate is calculated from its own weighted sum and sigmoid. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f) $$ $$ i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ ## LSTM a memory path with three gates **Definition.** Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell. ### Explanation and working Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference. ![Follow the cell state from left to right then identify forget input and output gates](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: Follow the cell state from left to right then identify forget input and output gates. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. Follow the cell state from left to right then identify forget input and output gates Tap the diagram to enlarge. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c) $$ $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\qquad h_t=o_t\odot\tanh(C_t) $$ The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state. ## Reading an LSTM equation one operation at a time ### 1. Two memories on the page do not mean the same value The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other. ![Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_split.svg) Diagram: Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.. Labels: Calculate cell memory before the hidden output | Calculate cell memory before the hidden output | Example: old cell 0.8, forget gate 0.25, input gate 0.5, candidate −0.4. | Retain old content | 0.25 × 0.8 = 0.2 | Admit new content | 0.5 × (−0.4) = −0.2 | New cell | 0.2 + (−0.2) = 0 | Hidden output: 0.7 × tanh(0) = 0 | The output gate is 0.7 here. Apply it to tanh of the NEW cell, after the cell | update. Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output. Tap the diagram to enlarge. ### 2. Gate values are multipliers For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention. An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive. ### 3. Add old and new contributions With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel. $$ c_t=f_tc_{t-1}+i_t\widetilde c_t $$ This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update. ### 4. Calculate the hidden state afterward Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0. $$ h_t=o_t\tanh(c_t) $$ If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c. ### 5. Supplied gates and calculated gates are different numerical types When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition. ## LSTM numerical when gates must first be calculated **Question.** At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state. ### Given x = 1; old hidden state = 0; old cell = 0.2. Forget/input/output preactivations are all 0. Candidate preactivation = 0.5. ### To find Activate all gates and candidate, then find C and h. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Distinguish preactivations from gate outputs The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate. $$ \sigma(a)=\frac1{1+e^{-a}} $$ $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ ![A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_gates_first.svg) Diagram: A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.. Labels: When the question supplies gate raw scores | When the question supplies gate raw scores | A gate raw score is not yet a gate value. | Gate scores | f, i, o scores | = 0 | Apply sigmoid | Each gate | becomes 0.5 | Candidate | score | 0.5 → tanh → | 0.462117 | Then update | Compute cell, | then hidden | output | Sigmoid(0) = 0.5. Do not replace a zero raw score with a closed gate. A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update. Tap the diagram to enlarge. **Step 1: activate the three gates.** $$ f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5 $$ **Step 2: activate the candidate preactivation.** $$ \widetilde C_t=\tanh(0.5)\approx0.462117157 $$ **Step 3: calculate the old and new cell contributions separately.** $$ f_tC_{t-1}=0.5(0.2)=0.1 $$ $$ i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579 $$ **Step 4: calculate the updated cell.** $$ C_t=0.1+0.5\tanh(0.5)\approx0.331058579 $$ **Step 5: calculate the hidden output from the updated cell.** $$ h_t=0.5\tanh(C_t)\approx0.159735818 $$ **Final answer:** fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely. ### LSTM gates from raw score 0 **Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 0. Old cell C\_old = 1; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(0)=\frac1{1+e^{-0}}\approx0.500000000 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(0)\approx0.000000000 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.462117157 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579 $$ **Final answer:** f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579. ### LSTM gates from raw score 1 **Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 1. Old cell C\_old = -0.5; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(1)=\frac1{1+e^{-1}}\approx0.731058579 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(1)\approx0.761594156 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.188942847 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289 $$ **Final answer:** f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289. ## Drill 43 Calculate gates before the cell ## Question **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h. ## Answer — step by step ### Given Raw gate scores: zf = zi = zo = 0. Raw candidate score = 0.5; old cell C = 0.2. ### To find Activate gates/candidate, then calculate the new cell and hidden state. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C) $$ $$ \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C) $$ ### Step 1: turn each zero score into a gate $$ \sigma(0)=\frac1{1+e^0}=\frac12=0.5 $$ Thus f = i = o = 0.5. A zero score is not a zero gate. ### Step 2: calculate candidate content $$ \widetilde C=\tanh(0.5)=0.462117157 $$ ### Step 3: combine retained memory and new content $$ fC_{\mathrm{previous}}=0.5\times0.2=0.1 $$ $$ i\widetilde C=0.5\times0.462117157=0.231058579 $$ $$ C=0.1+0.231058579=0.331058579 $$ ### Step 4: compute the visible hidden state $$ \tanh(C)=\tanh(0.331058579)\approx0.319471636 $$ $$ h=0.5\times0.319471636=0.159735818 $$ **Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818. ## LSTM candidate and output gate The candidate is proposed new cell content, so it may be negative. The output gate is a positive sigmoid multiplier that controls how much of the activated cell is exposed. Candidate content and gate values therefore have different roles. $$ \widetilde C_t=\tanh(W_C[h_{t-1},x_t]+b_C) $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o) $$ ## LSTM numerical when gates must first be calculated **Question.** At an LSTM time step, xₜ = 1, hₜ₋₁ = 0 and Cₜ₋₁ = 0.2. The affine preactivations of the forget, input and output gates are all 0. The candidate preactivation is 0.5. Calculate the activated gates, candidate cell content, new cell state and new hidden state. ### Given x = 1; old hidden state = 0; old cell = 0.2. Forget/input/output preactivations are all 0. Candidate preactivation = 0.5. ### To find Activate all gates and candidate, then find C and h. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Distinguish preactivations from gate outputs The affine transformations have already been evaluated in this question. Their outputs are preactivations, so activation functions are still required. Sigmoid is used for f, i and o; tanh is used for the candidate. $$ \sigma(a)=\frac1{1+e^{-a}} $$ $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ ![A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_gates_first.svg) Diagram: A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update.. Labels: When the question supplies gate raw scores | When the question supplies gate raw scores | A gate raw score is not yet a gate value. | Gate scores | f, i, o scores | = 0 | Apply sigmoid | Each gate | becomes 0.5 | Candidate | score | 0.5 → tanh → | 0.462117 | Then update | Compute cell, | then hidden | output | Sigmoid(0) = 0.5. Do not replace a zero raw score with a closed gate. A gate preactivation of zero becomes a sigmoid gate output of 0.5. Use that activated gate value in the cell update. Tap the diagram to enlarge. **Step 1: activate the three gates.** $$ f_t=i_t=o_t=\sigma(0)=\frac1{1+e^0}=0.5 $$ **Step 2: activate the candidate preactivation.** $$ \widetilde C_t=\tanh(0.5)\approx0.462117157 $$ **Step 3: calculate the old and new cell contributions separately.** $$ f_tC_{t-1}=0.5(0.2)=0.1 $$ $$ i_t\widetilde C_t=0.5\tanh(0.5)\approx0.231058579 $$ **Step 4: calculate the updated cell.** $$ C_t=0.1+0.5\tanh(0.5)\approx0.331058579 $$ **Step 5: calculate the hidden output from the updated cell.** $$ h_t=0.5\tanh(C_t)\approx0.159735818 $$ **Final answer:** fₜ = iₜ = oₜ = 0.5; C̃ₜ ≈ 0.462117157; Cₜ ≈ 0.331058579; hₜ ≈ 0.159735818. Retain unrounded calculator values between steps. A gate preactivation of 0 is different from a supplied gate output of 0: the former becomes 0.5 under sigmoid, while the latter would suppress its contribution completely. ### LSTM gates from raw score 0 **Question. An LSTM has forget, input and output gate raw scores all equal to 0, candidate raw score 0, and old cell 1. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 0. Old cell C\_old = 1; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(0)=\frac1{1+e^{-0}}\approx0.500000000 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(0)\approx0.000000000 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.500000000)(1)\approx0.500000000 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.500000000)(0.000000000)\approx0.000000000 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=0.500000000+(0.000000000)\approx0.500000000 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.462117157 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.500000000)(0.462117157)\approx0.231058579 $$ **Final answer:** f=i=o≈0.500000000; candidate≈0.000000000; cell≈0.500000000; hidden≈0.231058579. ### LSTM gates from raw score 1 **Question. An LSTM has forget, input and output gate raw scores all equal to 1, candidate raw score 1, and old cell -0.5. Calculate all gates, candidate, new cell and hidden state.** ### Given Raw scores z\_f = z\_i = z\_o = z\_candidate = 1. Old cell C\_old = -0.5; gate activation sigmoid; candidate activation tanh. ### To find f,i,o,candidate,C\_new,h\_new. ### Formula — which one and why Raw gate scores require sigmoid first. The candidate uses tanh; do not apply sigmoid to it. f retains old memory, i admits candidate, o controls exposed hidden output; C is cell memory. $$ f=i=o=\sigma(z) $$ $$ \widetilde C=\tanh(z) $$ $$ C_{\mathrm{new}}=fC_{\mathrm{old}}+i\widetilde C $$ $$ h_{\mathrm{new}}=o\tanh(C_{\mathrm{new}}) $$ **Yaad rakhne ka cue:** Gates sigmoid; candidate tanh; cell mein add; hidden pe output gate. ### Solution — step by step **Step 1: calculate the shared gate value.** $$ \sigma(1)=\frac1{1+e^{-1}}\approx0.731058579 $$ **Step 2: calculate the candidate.** $$ \widetilde C=\tanh(1)\approx0.761594156 $$ **Step 3: retain the old-cell part.** $$ fC_{\mathrm{old}}=(0.731058579)(-0.5)\approx-0.365529289 $$ **Step 4: admit the new content.** $$ i\widetilde C=(0.731058579)(0.761594156)\approx0.556769941 $$ **Step 5: add the memory contributions.** $$ C_{\mathrm{new}}=-0.365529289+(0.556769941)\approx0.191240652 $$ **Step 6: apply tanh to the cell.** $$ \tanh(C_{\mathrm{new}})\approx0.188942847 $$ **Step 7: apply the output gate.** $$ h_{\mathrm{new}}=(0.731058579)(0.188942847)\approx0.138128289 $$ **Final answer:** f=i=o≈0.731058579; candidate≈0.761594156; cell≈0.191240652; hidden≈0.138128289. ## Drill 43 Calculate gates before the cell ## Question **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h. ## Answer — step by step ### Given Raw gate scores: zf = zi = zo = 0. Raw candidate score = 0.5; old cell C = 0.2. ### To find Activate gates/candidate, then calculate the new cell and hidden state. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C) $$ $$ \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C) $$ ### Step 1: turn each zero score into a gate $$ \sigma(0)=\frac1{1+e^0}=\frac12=0.5 $$ Thus f = i = o = 0.5. A zero score is not a zero gate. ### Step 2: calculate candidate content $$ \widetilde C=\tanh(0.5)=0.462117157 $$ ### Step 3: combine retained memory and new content $$ fC_{\mathrm{previous}}=0.5\times0.2=0.1 $$ $$ i\widetilde C=0.5\times0.462117157=0.231058579 $$ $$ C=0.1+0.231058579=0.331058579 $$ ### Step 4: compute the visible hidden state $$ \tanh(C)=\tanh(0.331058579)\approx0.319471636 $$ $$ h=0.5\times0.319471636=0.159735818 $$ **Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818. ## Drill 50 Three step LSTM retention and replacement ## Question **Question.** C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step. ## Answer — step by step ### Given Initial C₀ = 0; output gate o = 0.7 at all three steps. Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6). ### To find Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas At each time step use the previous **cell state C**, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step. $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ **Step 1: Time 1: admit new content.** $$ f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4 $$ $$ C_1=0+0.4=0.4 $$ $$ h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274 $$ **Step 2: Time 2: retain memory and admit no new content.** $$ f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0 $$ $$ C_2=0.36+0=0.36 $$ $$ h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824 $$ **Step 3: Time 3: retain a small part and add negative content.** $$ f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3 $$ $$ C_3=0.072-0.3=-0.228 $$ $$ h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772 $$ **Final cell states:** 0.4, 0.36, −0.228. **Final hidden states:** approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given. ## LSTM cell and hidden state First calculate the retained old contribution. Separately calculate the admitted new contribution. Add them to obtain the cell state. Only afterward apply tanh and the output gate to obtain the hidden state. $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ h_t=o_t\odot\tanh(C_t) $$ ## Reading an LSTM equation one operation at a time ### 1. Two memories on the page do not mean the same value The cell state c carries the recurrent cell memory. The hidden state h is an exposed output computed from that memory. An LSTM passes both to the next step. A question may give the previous cell and hidden states separately; do not substitute one for the other. ![Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.](https://deep-learning-study-room.vercel.app/assets/figures/v_lstm_split.svg) Diagram: Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output.. Labels: Calculate cell memory before the hidden output | Calculate cell memory before the hidden output | Example: old cell 0.8, forget gate 0.25, input gate 0.5, candidate −0.4. | Retain old content | 0.25 × 0.8 = 0.2 | Admit new content | 0.5 × (−0.4) = −0.2 | New cell | 0.2 + (−0.2) = 0 | Hidden output: 0.7 × tanh(0) = 0 | The output gate is 0.7 here. Apply it to tanh of the NEW cell, after the cell | update. Old and new contributions are calculated separately, added to form the cell, and only then transformed into the hidden output. Tap the diagram to enlarge. ### 2. Gate values are multipliers For a previous cell value of 0.8, a forget gate of 0.25 keeps 0.25 × 0.8 = 0.2. A gate of 0 keeps none of that contribution; a gate of 1 keeps the whole contribution. Intermediate sigmoid values give partial retention. An input gate does the same kind of multiplication to the new candidate. If the candidate is −0.4 and the input gate is 0.5, the admitted new contribution is −0.2. The negative sign belongs to the candidate; sigmoid gates themselves are positive. ### 3. Add old and new contributions With those values, the new cell is 0.2 + (−0.2) = 0. A retained positive contribution and an admitted negative contribution can cancel. $$ c_t=f_tc_{t-1}+i_t\widetilde c_t $$ This example also explains why “input gate” does not mean simply multiply the raw input x by i. It multiplies the candidate calculated for the cell update. ### 4. Calculate the hidden state afterward Apply tanh to the updated cell and multiply by the output gate. If o = 0.7 and c = 0, tanh(0) = 0 and h = 0.7 × 0 = 0. $$ h_t=o_t\tanh(c_t) $$ If c were 1 instead, tanh(1) ≈ 0.761594 and h ≈ 0.533116 for the same output gate. The output gate controls exposure of the transformed cell, not the earlier addition that produced c. ### 5. Supplied gates and calculated gates are different numerical types When f, i and o are supplied, begin directly with their multiplications. When weights and inputs are supplied, first calculate each gate's affine input, apply sigmoid and retain those gate values. Calculate the candidate using tanh. Only then update c and h. The separate solved examples below cover both types and multi-step repetition. ## Drill 42 LSTM with a negative candidate ## Question **Question.** f = 0.8, i = 0.2, o = 0.6, candidate = −0.5, Cprevious = 0.3. Find C, h. ## Answer — step by step ### Given Activated gates: f = 0.8, i = 0.2, o = 0.6. Candidate = −0.5; old cell C = 0.3. ### To find Find the new cell and hidden state. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas An LSTM keeps a cell memory C and a visible hidden state h. The forget gate f controls old memory, the input gate i controls new candidate content, and the output gate o controls the visible state. These gate values are already supplied. $$ C=fC_{\mathrm{previous}}+i\widetilde C,\qquad h=o\tanh(C) $$ The candidate, written C with a tilde, may be negative. A negative candidate subtracts content when admitted. ### Step 1: retained old memory $$ fC_{\mathrm{previous}}=0.8\times0.3=0.24 $$ ### Step 2: admitted new content $$ i\widetilde C=0.2\times(-0.5)=-0.1 $$ ### Step 3: add them to obtain the new cell memory $$ C=0.24+(-0.1)=0.14 $$ ### Step 4: apply tanh to the cell memory $$ \tanh(0.14)=\frac{e^{0.28}-1}{e^{0.28}+1}=0.139092448 $$ ### Step 5: apply the output gate $$ h=0.6\times0.139092448=0.083455469 $$ **Final answer:** cell memory 0.14; hidden state approximately 0.083455469. C and h are different quantities. ## Drill 50 Three step LSTM retention and replacement ## Question **Question.** C₀ = 0, output gate 0.7 at all steps. Supplied(f, i, candidate) are(1,.5,.8), (.9,0,−.2), (.2,.5,−.6). Find C and h at each step. ## Answer — step by step ### Given Initial C₀ = 0; output gate o = 0.7 at all three steps. Supplied (f,i,candidate): (1,0.5,0.8), (0.9,0,−0.2), (0.2,0.5,−0.6). ### To find Calculate C₁,h₁,C₂,h₂,C₃,h₃ in order. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas At each time step use the previous **cell state C**, not the previous visible output h, in the memory equation. The output gate is 0.7 at every step. $$ C_t=f_tC_{t-1}+i_t\widetilde C_t,\qquad h_t=o_t\tanh(C_t) $$ **Step 1: Time 1: admit new content.** $$ f_1C_0=1(0)=0,\qquad i_1\widetilde C_1=0.5(0.8)=0.4 $$ $$ C_1=0+0.4=0.4 $$ $$ h_1=0.7\tanh(0.4)=0.7(0.379948962)=0.265964274 $$ **Step 2: Time 2: retain memory and admit no new content.** $$ f_2C_1=0.9(0.4)=0.36,\qquad i_2\widetilde C_2=0(-0.2)=0 $$ $$ C_2=0.36+0=0.36 $$ $$ h_2=0.7\tanh(0.36)=0.7(0.345214034)=0.241649824 $$ **Step 3: Time 3: retain a small part and add negative content.** $$ f_3C_2=0.2(0.36)=0.072,\qquad i_3\widetilde C_3=0.5(-0.6)=-0.3 $$ $$ C_3=0.072-0.3=-0.228 $$ $$ h_3=0.7\tanh(-0.228)=0.7(-0.224129674)=-0.156890772 $$ **Final cell states:** 0.4, 0.36, −0.228. **Final hidden states:** approximately 0.265964274, 0.241649824, −0.156890772. The supplied gate value 1 is an idealized limit, used exactly as given. ## Drill 43 Calculate gates before the cell ## Question **Question.** Gate preactivations zf = zi = zo = 0, candidate preactivation 0.5, Cprevious = 0.2. Find gates, C, h. ## Answer — step by step ### Given Raw gate scores: zf = zi = zo = 0. Raw candidate score = 0.5; old cell C = 0.2. ### To find Activate gates/candidate, then calculate the new cell and hidden state. ### Formula — which one and why **Activate raw gate scores first.** A preactivation is an unprocessed score. Convert forget/input/output scores with sigmoid and candidate scores with tanh before using the cell-update equations. z\_f,z\_i,z\_o = gate preactivations; z\_c = candidate preactivation. An already activated gate must not be passed through sigmoid again. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o) $$ $$ \widetilde C=\tanh(z_c) $$ **Yaad rakhne ka cue:** Raw score pehle activate; activated gate ko dobara sigmoid nahi. **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step ### Meaning and formulas The question supplies raw gate scores, called preactivations. Convert them to gates using sigmoid first. Convert the candidate's score using tanh. $$ f=\sigma(z_f),\quad i=\sigma(z_i),\quad o=\sigma(z_o),\quad \widetilde C=\tanh(z_C) $$ $$ \sigma(z)=\frac1{1+e^{-z}},\quad C=fC_{\mathrm{previous}}+i\widetilde C,\quad h=o\tanh(C) $$ ### Step 1: turn each zero score into a gate $$ \sigma(0)=\frac1{1+e^0}=\frac12=0.5 $$ Thus f = i = o = 0.5. A zero score is not a zero gate. ### Step 2: calculate candidate content $$ \widetilde C=\tanh(0.5)=0.462117157 $$ ### Step 3: combine retained memory and new content $$ fC_{\mathrm{previous}}=0.5\times0.2=0.1 $$ $$ i\widetilde C=0.5\times0.462117157=0.231058579 $$ $$ C=0.1+0.231058579=0.331058579 $$ ### Step 4: compute the visible hidden state $$ \tanh(C)=\tanh(0.331058579)\approx0.319471636 $$ $$ h=0.5\times0.319471636=0.159735818 $$ **Final answer:** gates 0.5 each; candidate ≈ 0.462117157; C ≈ 0.331058579; h ≈ 0.159735818. ## GRU gates, candidate and update A GRU has a reset gate and an update gate. Reset changes how earlier state enters the candidate calculation; update controls the final old/new mixture. The numerical below states its convention explicitly: z is the fraction of the new candidate admitted. $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z) $$ $$ r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ ## GRU two gates and one state **Definition.** A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state. ### Gate equations and the convention used here The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state: $$ r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z) $$ The candidate equation used in these notes applies reset before the recurrent candidate transformation: $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h) $$ These notes define z as the fraction of **new candidate** admitted: $$ h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture. ![First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention. Tap the diagram to enlarge. ### Complete numerical with supplied gate values **Question.** A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ. ### Given Old hidden state = 0.4; r = 0.5; z = 0.25. x = 1; W = 0.2; U = 0.5; b = 0. Here z weights the candidate in the final mixture. ### To find Find candidate and new hidden state. ### Formula — which one and why **GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate. r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state. $$ \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b) $$ $$ h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h $$ **Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control. ### Solution — step by step **Step 1: apply the reset gate to the previous hidden state.** $$ r_th_{t-1}=0.5(0.4)=0.2 $$ The gates are already supplied as outputs, so no further sigmoid calculation is needed. **Step 2: calculate the candidate preactivation.** $$ a_{\mathrm{candidate}}=W_hx_t+U_h(r_th_{t-1})+b_h $$ $$ a_{\mathrm{candidate}}=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh to obtain the candidate.** $$ \widetilde h_t=\tanh(0.3)\approx0.291312612 $$ **Step 4: calculate the two mixture contributions.** $$ (1-z_t)h_{t-1}=(1-0.25)(0.4)=0.3 $$ $$ z_t\widetilde h_t=0.25\tanh(0.3)\approx0.072828153 $$ **Step 5: add the contributions.** $$ h_t=0.3+0.25\tanh(0.3)\approx0.372828153 $$ **Final answer:** candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture. ### If the question uses the other update-gate convention Some texts define z as the fraction of the **old state retained**: $$ h_t=z_th_{t-1}+(1-z_t)\widetilde h_t $$ Under that explicitly different convention, the same numerical value z = 0.25 gives: $$ h_t=0.25(0.4)+0.75\tanh(0.3)\approx0.318484459 $$ This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values. ### Relationship to LSTM A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task. ### Retain-old GRU with gate 0 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = 0.4; z = 0; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: retain the old-state fraction.** $$ 0(0.4)=0 $$ **Step 5: calculate the candidate fraction.** $$ (1-0)(0.291312612)\approx0.291312612 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612 $$ **Final answer:** Candidate ≈ 0.291312612; new state ≈ 0.291312612. ### Retain-old GRU with gate 0.8 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = -0.2; z = 0.8; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(-0.2)=-0.1 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(-0.1)+0=0.15 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.15)\approx0.148885034 $$ **Step 4: retain the old-state fraction.** $$ 0.8(-0.2)=-0.16 $$ **Step 5: calculate the candidate fraction.** $$ (1-0.8)(0.148885034)\approx0.029777007 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993 $$ **Final answer:** Candidate ≈ 0.148885034; new state ≈ -0.130222993. ## Drill 44 GRU reset then mix ## Question **Question.** hprevious = 0.4, r = 0.5, z = 0.25, x = 1, W = 0.2, U = 0.5, b = 0. Use candidate = tanh(Wx+U(r × hprevious)) and h = (1−z) hprevious+z candidate. ## Answer — step by step ### Given Old hidden state = 0.4; reset r = 0.5; update z = 0.25. x = 1; W = 0.2; U = 0.5; b = 0. Use h = (1−z)h\_old + z×candidate. ### To find Find reset contribution, candidate and new hidden state. ### Formula — which one and why **GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate. r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state. $$ \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b) $$ $$ h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h $$ **Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control. ### Solution — step by step ### Meaning and given convention The reset gate r scales the old state while making the candidate. The update gate z controls the amount of candidate used in the final mixture **under this question's convention**. Some books reverse the roles of z and 1 − z, so copy the supplied formula. $$ \widetilde h=\tanh\left(Wx+U(rh_{\mathrm{previous}})+b\right) $$ $$ h=(1-z)h_{\mathrm{previous}}+z\widetilde h $$ ### Step 1: reset the old state for candidate construction $$ rh_{\mathrm{previous}}=0.5\times0.4=0.2 $$ ### Step 2: form the candidate's raw score $$ Wx=0.2\times1=0.2 $$ $$ U(rh_{\mathrm{previous}})=0.5\times0.2=0.1 $$ $$ a=0.2+0.1+0=0.3 $$ ### Step 3: apply tanh $$ \widetilde h=\tanh(0.3)=0.291312612 $$ ### Step 4: mix the original old state with the candidate $$ 1-z=1-0.25=0.75 $$ $$ h=0.75(0.4)+0.25(0.291312612) $$ $$ h=0.3+0.072828153=0.372828153 $$ **Final answer: h ≈ 0.372828153.** The final old-state term uses 0.4, not the reset value 0.2. ## GRU state under the retain-old convention Some GRU questions use the opposite naming convention: z is the fraction of the OLD state retained. The formula below uses that convention. Do not combine its formula with an explanation that says z retains the new candidate. $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h) $$ $$ h_t=z_t\odot h_{t-1}+(1-z_t)\odot\widetilde h_t $$ **Question. Use the retain-old GRU convention. Given hprevious = 0.4, reset gate r = 0.5, update gate z = 0.25, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and the next hidden state.** ### Given h\_old = 0.4; r = 0.5; z = 0.25; x = 1. W = 0.2; U = 0.5; b = 0; z retains the old state. ### To find Find the candidate and new hidden state. ### Formula — which one and why **GRU with a retain-old update gate.** Form the candidate using reset memory; then z retains the old state and 1−z admits the candidate. Follow this question-specific convention. r = reset gate; z = retain-old gate; h\_old = previous state; h̃ = candidate. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Question mein z kisko multiply karta hai wahi meaning; yahan old ko retain. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ r h_{\mathrm{previous}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate's raw input.** Its new-input contribution is 0.2 × 1; its reset-memory contribution is 0.5 × 0.2. $$ a=0.2(1)+0.5(0.2)+0=0.2+0.1=0.3 $$ **Step 3: apply tanh to this raw input.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: calculate how much old state is retained.** In this question, z = 0.25 multiplies the original old state 0.4. $$ z h_{\mathrm{previous}}=0.25(0.4)=0.1 $$ **Step 5: calculate how much candidate is admitted.** The other fraction is 1 − z = 0.75. $$ 1-z=1-0.25=0.75 $$ $$ (1-z)\widetilde h=0.75(0.291312612)\approx0.218484459 $$ **Step 6: add the two final contributions.** $$ h=0.1+0.218484459=0.318484459 $$ **Final answer: h ≈ 0.318484459.** The candidate was 0.291312612; the final hidden state is a mixture of that candidate and the original old state. Under the admit-new convention with the same numeric z, the result would instead be 0.372828153. ### Retain-old GRU with gate 0 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = 0.4; z = 0; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: retain the old-state fraction.** $$ 0(0.4)=0 $$ **Step 5: calculate the candidate fraction.** $$ (1-0)(0.291312612)\approx0.291312612 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612 $$ **Final answer:** Candidate ≈ 0.291312612; new state ≈ 0.291312612. ### Retain-old GRU with gate 0.8 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = -0.2; z = 0.8; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(-0.2)=-0.1 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(-0.1)+0=0.15 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.15)\approx0.148885034 $$ **Step 4: retain the old-state fraction.** $$ 0.8(-0.2)=-0.16 $$ **Step 5: calculate the candidate fraction.** $$ (1-0.8)(0.148885034)\approx0.029777007 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993 $$ **Final answer:** Candidate ≈ 0.148885034; new state ≈ -0.130222993. ## Batch normalization and learned scale/shift Batch normalization first uses batch statistics, then applies a learned scale and shift. Epsilon belongs inside the square root. Inference ordinarily uses stored running statistics. $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ ## Batch normalization inside a residual network **Definition.** Batch normalization normalizes a feature using training-batch statistics, then applies a learned scale and shift. It operates on intermediate activations and is different from normalizing the original dataset once before training. For one feature with m values in a training batch: $$ \mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i $$ $$ \sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2 $$ $$ \hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}} $$ $$ y_i=\gamma\hat{x}_i+\beta $$ The batch mean μ\_B is the centre; batch variance σ²\_B measures squared spread. Epsilon ε is a small positive stability constant **inside the square root**. Gamma γ is a learned scale and beta β is a learned shift. For convolutional BN, statistics are usually computed separately for each channel across batch and spatial positions. Ordinary inference uses stored running statistics rather than the current test batch. **Question (practice, 3 marks). For one feature, a training batch contains [2,4]. Use ε = 0.00001, γ = 2 and β = 0.5. Calculate both batch-normalized outputs using population variance.** ### Given x₁ = 2; x₂ = 4; m = 2; ε = 0.00001; γ = 2; β = 0.5. ### To find Batch mean, variance, normalized values and scale-and-shift outputs. ### Formula — which one and why Use the four formulas above in order. Population variance divides by m = 2, not m − 1. Scale and shift follow normalization because the model can learn the useful final feature scale. ### Solution — step by step **Step 1: mean.** $$ \mu_B=\frac{2+4}{2}=3 $$ **Step 2: deviations.** 2 − 3 = −1 and 4 − 3 = 1. **Step 3: squared deviations and variance.** $$ \sigma_B^2=\frac{(-1)^2+1^2}{2}=\frac{1+1}{2}=1 $$ **Step 4: stable denominator.** $$ d=\sqrt{1+0.00001}\approx1.00000499999 $$ **Step 5: normalize separately.** $$ \hat{x}_1=\frac{-1}{d}\approx-0.99999500004 $$ $$ \hat{x}_2=\frac{1}{d}\approx0.99999500004 $$ **Step 6: scale the first normalized value, then shift it.** $$ y_1=2(-0.99999500004)+0.5\approx-1.49999000008 $$ **Step 7: scale the second normalized value, then shift it.** $$ y_2=2(0.99999500004)+0.5\approx2.49999000008 $$ **Final answer:** output ≈ **[−1.49999,2.49999]**. Its mean is β = 0.5; the final output need not have zero mean after the learned shift. The normalized values themselves have mean zero here. **Yaad rakho:** centre hatao → spread se divide → learned scale lagao → shift add karo. ## Global average pooling Global average pooling takes one mean per complete spatial channel. The worked example compares the resulting dense classifier with a classifier after flattening. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ ## Global average pooling: one average for each channel ![Average each channel separately: these are the exact values used in the worked example.](https://deep-learning-study-room.vercel.app/assets/figures/u4_gap.svg) Diagram: Average each channel separately: these are the exact values used in the worked example.. Labels: u4 gap | Global average pooling keeps channels separate | Channel A | 1 | 3 | 5 | 7 | → | 4 | 16 ÷ 4 | Channel B | 2 | 4 | 6 | 8 | → | 5 | 20 ÷ 4 | 2 × 2 × 2 input → two channel means → [4, 5] Average each channel separately: these are the exact values used in the worked example. Tap the diagram to enlarge. **Definition.** Global average pooling, or GAP, replaces every complete spatial feature map by its mean. An H × W × C tensor becomes C numbers. The channels remain separate; we do not average all channels into one number. $$ g_c=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}a_{i,j,c} $$ The letter c selects a channel; i and j select its row and column; a is the value at that location. Add all H × W entries in that channel and divide by H × W. GAP has no learned weights or biases. A classifier after GAP can still have trainable parameters. In GoogLeNet, 7 × 7 × 1,024 becomes 1 × 1 × 1,024 before the final 1,000-class linear classifier. **Question (practice, 3 marks). A 2 × 2 × 2 tensor has channel A = $\begin{bmatrix}1 & 3\\5 & 7\end{bmatrix}$ and channel B = $\begin{bmatrix}2 & 4\\6 & 8\end{bmatrix}$ . Apply GAP, then find the parameters of a three-output dense classifier receiving the result, including one bias per output.** ### Given H = 2, W = 2, C = 2. Each of the two channels contains four values. The classifier has three outputs. ### To find The GAP result and dense-head parameter count. ### Formula — which one and why Use one mean per channel. The dense layer receives two numbers, so each of its three outputs needs two weights and one bias. $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ ### Solution — step by step **Step 1: add channel A.** 1 + 3 + 5 + 7 = 16. **Step 2: average channel A.** 16 ÷ 4 = 4. **Step 3: add channel B.** 2 + 4 + 6 + 8 = 20. **Step 4: average channel B.** 20 ÷ 4 = 5. Thus GAP outputs **[4,5]**. **Step 5: count the classifier's weights.** 2 inputs × 3 outputs = 6 weights. **Step 6: add three biases.** 6 + 3 = **9 parameters**. GAP itself contributes zero. **Final answer:** GAP = [4,5]; GAP shape = 1 × 1 × 2; classifier parameters = 9. If we flattened all eight inputs instead, that head would require (8 + 1) × 3 = 27 parameters. **Yaad rakho:** ek channel ka poora average; channels ko aapas mein mat milao. ## Top-k error percentage Top-k error counts examples whose correct class is absent from the highest-scoring k predictions. Success counts must not be added twice. $$ E_k=\frac{N-C_k}{N}\times100\% $$ ## Top-1 and top-5 error: what an ImageNet result means **Top-1 prediction** is the class with the highest score. A top-1 mistake occurs when that class is wrong. **Top-5 prediction** is the list of the five highest-scoring classes. A top-5 mistake occurs only when the correct class is absent from that list. It is not the average error of five separate models. For an image of a cat, the ordered list dog, cat, fox, horse, bird is a top-1 error but a top-5 success. The ordered list dog, fox, horse, bird, car is an error under both measures. **Question (practice, 2 marks). A classifier evaluates 100 images. The correct class is ranked first for 78 images and appears somewhere in the first five positions for 92 images. Calculate top-1 error and top-5 error.** ### Given Total images = 100; top-1 successes = 78; top-5 successes = 92. The 78 first-place successes are already included in the 92 top-five successes. ### To find The percentage of mistakes under each rule. ### Formula — which one and why Subtract successful images from the total to obtain mistakes. Divide mistakes by the total and multiply by 100 to express a percentage. $$ E_k=\frac{N-C_k}{N}\times100\% $$ N means total evaluated images; C\_k means images whose correct class appears among the top k predictions. ### Solution — step by step **Step 1: top-1 mistakes.** 100 − 78 = 22. **Step 2: top-1 error percentage.** $$ E_1=\frac{22}{100}\times100\%=22\% $$ **Step 3: top-5 mistakes.** 100 − 92 = 8. **Step 4: top-5 error percentage.** $$ E_5=\frac{8}{100}\times100\%=8\% $$ **Final answer:** top-1 error = 22%; top-5 error = 8%. A broader acceptable list makes top-5 error no greater than top-1 error for the same evaluated predictions. **Yaad rakho:** top-five mein sahi naam mil gaya toh top-five success. --- ## TEE · Calculator sequence without model guesswork Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c19-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/19/tee-c19-s2 The reported model name is “Casio FX ES 990”; the exact model is not confirmed. Use your calculator’s actual function labels. On common ES/ES Plus models, exp(x) is usually the shifted ln key, while the EXP/ × 10ˣ key enters scientific notation and is NOT the same as exp(x). Check the exact key labeling on your device. For sigmoid at 2.2, type 1÷(1+exp(−2.2)). Keep the negative sign inside the exponential. Expected 0.900249511. For a negative gradient update, type old−rate × (negative gradient); use parentheses around the negative value. For cross-entropy, use −ln(probability), not log 10. For tanh, use the hyperbolic function or its exponential fraction if that function is difficult to find. Retain 6–9 digits in intermediate results or use stored values; round final answers to 4–6 decimal places unless the paper says otherwise. Work line by line on paper even if the calculator can evaluate the whole expression. Check exp(0) = 1, ln(1) = 0, sigmoid(0) = 0.5, tanh(0) = 0 before relying on the chosen key sequence. --- ## TEE · What to write if the exam page feels blank Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c19-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/19/tee-c19-s3 Pause briefly, put both feet down and take one slow breath. Then write only the GIVEN values and what is ASKED. Classify the question using the method map. Write its general formula before inserting numbers. One correct first line is a smaller task than “solve the whole question”. For CNN, draw the shape chain. For back prop, draw arrows and label the loss. For an optimizer, draw columns for step, gradient, state, new parameter. For LSTM, write old part, new part, cell, hidden. If you remain stuck after a short attempt, leave visible working space and move to a question you can start. Return after collecting those marks. The goal is to reduce the next action to something concrete and familiar. Practice it during mocks so it is available under time pressure. --- ## TEE · Answer lengths and partial credit habits Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c19-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/19/tee-c19-s4 For 2 marks, give a precise definition plus the key distinction/example. For 5 marks, give the definition, a labeled diagram or formula, working/principle and a relevant implication. For 10 marks, answer every subpart: architecture or setup, main equations, stepwise explanation, required numerical working, and limitations or justification where asked. Do not pad a numerical with unrelated theory. Always label assumptions when inputs, loss, threshold or matrix orientation are missing. Box the requested final result, include shapes or units, and retain working. An unsupported final number makes it harder to award method credit. Never say an optimizer guarantees global convergence or an architecture always outperforms another unless a specific theorem and its conditions actually apply. --- ## TEE · Methods to revise before the examination Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c19-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/19/tee-c19-s5 ### Last-day revision: read in this order This is a revision digest after studying the lessons. Full solutions remain beside each topic and under Solved Questions. If a step here is unclear, return to that worked example; the final day is for recovering the method, not memorizing unexplained final numbers. ### 1. Learning and the neuron Machine learning estimates a model from data. Deep learning uses multiple trainable layers to learn successive representations. Inputs describe a sample; targets are the required answers; predictions are the model's answers. Weights and biases are learned parameters. Learning rate, batch size and architecture are chosen hyperparameters. Training changes parameters using training examples. Validation selects settings/stopping time. Testing evaluates the final choice on held-out data. Overfitting means strong training performance with weak generalization; underfitting means the model has not captured enough of the relationship. For one neuron, multiply each input by its own weight, add all products, add the bias once and apply the activation. A layer repeats this for several receiving neurons. Each weight-matrix row belongs to one receiving neuron. Transpose swaps rows and columns; it does not change signs. $$ z=\sum_iw_ix_i+b,\qquad a=f(z) $$ **Memory cue:** multiply → sum → bias → activate. In a full network, finish one complete layer before the next. ### 2. Activation and loss choices Sigmoid maps to (0,1) and is used for binary-class probabilities. Tanh maps to (−1,1), centred on zero. Both can saturate, producing small derivatives. ReLU retains positive values and returns zero on the negative side; its negative-side gradient is zero. Leaky ReLU retains a specified small negative-side slope. Softmax normalizes a complete list of logits into mutually exclusive class probabilities. $$ \sigma(z)=\frac1{1+e^{-z}},\qquad\sigma\prime(z)=\sigma(z)[1-\sigma(z)] $$ $$ \tanh\prime(z)=1-\tanh^2(z) $$ $$ p_i=\frac{e^{z_i-c}}{\sum_j e^{z_j-c}},\qquad c=\max_jz_j $$ For squared errors, SSE sums squares, half SSE divides that sum by 2, and MSE divides it by the number of scalar predictions. Their derivatives differ. For one-hot categorical cross-entropy, only the correct-class probability contributes. For binary cross-entropy, p means probability of class 1, not automatically the true class. $$ L_{\mathrm{half}}=\frac12\sum_i(\hat y_i-y_i)^2 $$ $$ \mathrm{MSE}=\frac1n\sum_i(\hat y_i-y_i)^2 $$ $$ L_{\mathrm{BCE}}=-[y\ln p+(1-y)\ln(1-p)] $$ $$ L_{\mathrm{CCE}}=-\ln p_{\mathrm{correct}} $$ **Memory cue:** loss ka denominator dekho before differentiating. Sigmoid output is not its derivative. ### 3. Backpropagation: the repeatable method Write the given loss and activation first. Compute all forward sums/activations with the original parameters. Differentiate loss with respect to each output, multiply by the output activation derivative, and call the result the output delta. $$ \delta_o=\frac{\partial L}{\partial o}f_o\prime(z_o) $$ For a hidden neuron, add the outgoing weighted deltas first and then multiply by the hidden activation derivative. Each connection gradient is receiving delta × sending activation. The bias gradient is the receiving delta. Use old weights for all gradients before making the simultaneous update. $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i,\qquad\frac{\partial L}{\partial b_j}=\delta_j $$ $$ \theta_{\mathrm{new}}=\theta-\eta\frac{\partial L}{\partial\theta} $$ **Memory cue:** path mein multiply; paths milne par add; update sab gradients ke baad. For sigmoid+BCE or softmax+CCE, the logit delta simplifies to prediction−target for the stated single-example sum convention. ### 4. Optimizers: identify the stored state Full-batch GD uses the entire dataset for one update; SGD uses one sample; mini-batch uses a subset. An epoch is one pass through the data. With a final incomplete batch included, updates per epoch are ceiling(N/B). | Method | State and calculation order | Detail that changes the answer | | --- | --- | --- | | GD | Current gradient → subtract ηg | Recompute at the new parameter unless gradients are explicitly supplied | | Momentum | Retain βv → subtract ηg → add new velocity to parameter | Signed velocity and gradient-average conventions must not be mixed | | Nesterov | Compute lookahead θ+βv → gradient there → new velocity → update actual θ | Do not add new velocity to the lookahead point | | AdaGrad | Add g² to all-past sum G → divide gradient by √G+ε | G never decreases | | RMSProp | Decay old squared state → add recent g² → normalize | Use the specified decay factor | | Adam | Update signed mean m and squared mean v → correct both → normalize | Carry raw states forward; correction powers use iteration t | $$ v_t=\beta v_{t-1}-\eta g_t,\qquad\theta_t=\theta_{t-1}+v_t $$ $$ G_t=G_{t-1}+g_t^2 $$ $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t $$ $$ v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\qquad\hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ Here v means signed velocity in momentum and squared-gradient state in Adam; the method defines the symbol. **Memory cue:** AdaGrad sab squares; RMSProp recent squares; Adam direction + squares + correction. ### 5. Regularization and generalization L1 adds λΣ|w| and a nonzero-weight derivative λsign(w), encouraging sparsity. L2 with the half convention adds (λ/2)Σw² and derivative λw. Without that half, the derivative is 2λw. Regularization gradients add to data gradients before updating the weight. Inverted dropout uses keep probability q: retain a sampled activation with scale 1/q, or replace it by zero. Inference uses the full activation without that training-time rescaling. Drop probability p gives q = 1−p. Early stopping tracks validation performance and restores the best checkpoint under the stated patience rule. $$ a'_i=\frac{m_i a_i}{q},\qquad m_i\in\{0,1\} $$ **Memory cue:** dropout probability aur keep probability alag; L2 mein half hai ya nahi, pehle dekho. ### 6. CNN calculation sequence Use the deep-learning cross-correlation convention unless the question explicitly requires kernel flipping. At each output position, multiply matching input-window and kernel entries, sum over spatial entries and all input channels, and add that filter's bias once. A new filter creates a new output channel. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K_hK_wC_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC,\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Apply the shape formula separately to height and width. Pooling changes spatial dimensions but preserves channels; ordinary pooling and flattening have zero learned parameters. Maximum pooling chooses the largest window value; average pooling adds its values and divides by the number of entries. A filter's spatial reuse changes computation, not its parameter count. For a complete architecture, make one row per layer: input shape → operation → output shape → weights → biases → total. Flatten uses the size after the final convolution/pool. **Memory cue:** shape count aur parameter count ko kabhi mix mat karo. ### 7. CNN theory and architectures CNNs use local connectivity and weight sharing, reducing parameters compared with unrestricted dense image mappings. Convolution learns features; activations introduce nonlinearity; pooling summarizes local responses; dense or global-pooling/output stages make the final task prediction. AlexNet: stacked convolutions with ReLU, pooling and dense classification; dropout in dense layers and data augmentation help control overfitting. Inception: parallel spatial scales, 1 × 1 reductions and channel concatenation. ResNet: learns a residual F(x) and adds a shortcut x; use projection when shapes differ. Concatenation adds channels; residual addition requires equal shapes and retains their count. Transfer learning reuses a pretrained feature extractor, replaces the task head, trains the head and optionally fine-tunes selected pretrained layers at a suitable smaller learning rate. Explain classification, detection and segmentation as different outputs, with an appropriate concrete application. ### 8. Recurrent models An RNN combines the current input with the previous activated hidden state using weights shared across time. Unroll the sequence and calculate one step at a time. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b) $$ For TEE, revise every recurrent-model item below. The faculty MT2 update received on 12 October focuses MT2 on Units 3 and 4; these Unit 5 RNN topics belong to the complete TEE preparation route. BPTT applies the chain rule backward through the unrolled network. A shared weight receives contributions from all relevant time steps; sum them before the update. Repeated small derivative products can vanish; repeated large products can explode. Gradient clipping limits large updates; gated memory helps preserve useful long-range dependencies. LSTM has a cell memory C and hidden state h. Sigmoid gates control retain/write/read proportions; the candidate uses tanh. Calculate activated gates first when raw gate scores are supplied. $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ h_t=o_t\odot\tanh(C_t) $$ GRU uses reset and update gates with one hidden state. In the candidate-weighting convention used here: $$ \widetilde h_t=\tanh(Wx_t+U(r_t\odot h_{t-1})+b) $$ $$ h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ If the question instead uses z to retain the old state, follow its reversed mixing convention. **Memory cue:** LSTM = old memory retain + new content write, then output gate. GRU = reset for candidate, update for final mixture. ### The final answer-writing routine For every numerical: full question → Given → To find → governing formula and why → separate numbered steps → final answer. Carry extra calculator precision; round at the end. Label each layer, iteration and connection so the examiner can follow the calculation. For a definition question, state the technical meaning and the requested distinction/example. For a longer mechanism question, include a labelled diagram or equations, explain their sequence, then give the relevant benefits, limitations or comparison. Marks printed beside a historical question are its actual marks; individual steps here are not assigned invented official marks. For a 180-minute TEE, a workable allocation is 10 minutes to read/select, about 30 minutes per attempted 20-mark main question, and 20 minutes to complete or review. Adjust to the paper. If stuck, write the givens and governing formula first, complete the solvable parts, then return to the unfinished calculation. --- ## TEE · Learning and the numbers we adjust Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c20-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/20/tee-c20-s0 | Term | Meaning | Tiny example | | --- | --- | --- | | Input / feature | A number given to the model | Image pixel intensity | | Sample | One example processed by the model | One labeled image | | Dataset | A collection of examples | 1,000 training images | | Target / label | Correct answer supplied for training | Class 2 is correct | | Prediction | Model's current answer | Class probabilities | | Weight | Learned multiplier on a connection | 0.4 multiplying an input | | Bias | Learned value added to a weighted sum | z = wx + b | | Parameter | A value learned during training | Weight or bias | | Hyperparameter | A setting chosen for training | Learning rate or batch size | | Activation | Transformation of a neuron's raw score | ReLU or sigmoid | | Loss | Numerical measure of prediction error | Mean squared error | | Gradient | Local rate of loss change for parameters | Positive g means locally uphill | | Optimizer | Rule using gradients to update parameters | Momentum or Adam | | Epoch | One pass through the training dataset | All 100 samples processed | | Iteration | One parameter update in these exercises | One mini-batch update | | Inference | Applying the trained model | Predicting a new image's class | --- ## TEE · Images and sequences Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c20-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/20/tee-c20-s1 | Term | Meaning | What to keep separate | | --- | --- | --- | | Tensor | Array of numbers with one or more axes | Its shape and its values | | Channel | One feature map within an image-like tensor | Channels versus spatial size | | Kernel / filter | Learned weights sliding over a map | Kernel size versus filter count | | Stride | Step between adjacent window positions | Stride versus kernel width | | Padding | Extra border cells around an input | Per-side P versus total 2P | | Feature map | Responses from a filter across positions | Activations versus parameters | | Pooling | Window-based reduction of feature maps | Maximum versus average | | Flatten | Rearrange H × W × C entries as a vector | Rearrangement has no parameters | | Logit | Raw score before an output activation | Score versus probability | | Hidden state | Recurrent features carried through time | Previous state versus current input | | Shared weights | One learned rule reused at many positions/times | Reuse versus extra parameters | --- ## TEE · Reading a question precisely Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c20-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/20/tee-c20-s2 “Calculate” asks for arithmetic and a result. Write the formula and substitutions before the final number. “Explain” asks how and why a method works. Give its definition, describe the steps, and connect each step to its purpose. “Compare” asks for matched differences. Discuss the same attributes for both methods, such as stored state, update rule, benefit and limitation. “Draw” asks for a labeled structure. Label inputs, layers or time steps, arrows, activations and the output. A diagram and explanation should describe the same computation. “Design” asks for a justified arrangement. State the task, input and output, chosen layers, activation/loss pair and training procedure. State any assumption needed because the question leaves a value unspecified. --- ## TEE · R25 Q1a · ML versus DL and architecture choice Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s0 ## Question **Question. Compare ML and DL. Choose architectures for five-class diabetic-retinopathy retinal images, alphabet-image recognition, and weather prediction from N days of history.** (5 marks) ## Answer — step by step **Definition.** Deep learning is a subfield of machine learning that learns representations through multiple trainable processing layers. Each layer transforms the preceding representation, allowing the complete model to learn a composition of functions from data. ### Position within artificial intelligence Artificial intelligence studies systems that perform tasks such as perception, reasoning and decision-making. Machine learning develops models whose parameters or behavior are estimated from data. Deep learning uses layered neural models within machine learning. $$ \text{Deep learning}\subset\text{Machine learning}\subset\text{Artificial intelligence} $$ ![AI is the broad field. Machine learning learns from data; deep learning learns through multiple representation layers.](https://deep-learning-study-room.vercel.app/assets/figures/v_ai_ml_dl.svg) Diagram: AI is the broad field. Machine learning learns from data; deep learning learns through multiple representation layers.. Labels: Deep learning belongs inside machine learning | Deep learning belongs inside machine learning | These are nested areas, not three unrelated technologies. | ARTIFICIAL INTELLIGENCE | MACHINE LEARNING | DEEP LEARNING | Neural networks with many layers | AI is the broad field. ML learns from data. DL uses deep neural networks. AI is the broad field. Machine learning learns from data; deep learning learns through multiple representation layers. Tap the diagram to enlarge. ### Motivation: why use multiple learned layers? **Representation learning.** Hand-designing every useful image or speech feature is difficult. A deep model can learn intermediate representations together with the final prediction objective. In a CNN, early layers may respond to local edges, while later layers combine information over larger regions. These patterns depend on training and are not guaranteed to have a particular human meaning. **Nonlinear relationships.** Real tasks often depend on combinations of input variables. Composing affine transformations with nonlinear activations can represent interactions that a single linear decision boundary cannot. Depth supplies repeated transformations; increasing only the number of neurons in one layer changes width instead. **End-to-end optimization.** The final task loss supplies gradients to the preceding trainable layers. The feature extractor and prediction head can therefore be adjusted for the same task. Preprocessing and human decisions about architecture, data and evaluation still remain necessary. ### Factors that made deep learning practical | Factor | How it contributes | | --- | --- | | Larger datasets | Provide more examples of the variations a complex model must learn | | GPUs and parallel computation | Accelerate the large matrix and convolution operations used in training | | Improved training methods | Backpropagation, suitable activations, initialization and optimizers make fitting deep models more practical | | Pretrained models | Reuse learned features so a new task may require less labeled data and training effort | ### Applications in different domains An application answer should identify the input, the prediction and the role of the learned representation. | Domain | Example input | Required output | Role of deep learning | | --- | --- | --- | --- | | Computer vision | Photograph or video frame | Image class, object locations or pixel labels | Learn visual features with a CNN and an appropriate task head | | Language | Review, sentence or conversation | Sentiment label, translated words or a response | Learn representations of text and its context | | Speech | Audio sequence | Words or a speaker label | Model patterns in acoustic features over time | | Healthcare | Medical image or patient measurements | A task-specific class or numerical estimate | Learn associations from labeled examples; assess results on suitable held-out data | | Autonomous systems and robotics | Camera and other sensor measurements | Perception estimates used by a larger control system | Recognize objects and relevant environmental features | | Financial services | Transaction features or a time-ordered record | Fraud score, risk estimate or forecast | Model patterns and interactions among observations | ### Advantages, limitations and comparison with traditional ML | Aspect | Traditional machine learning often uses | Deep learning often uses | | --- | --- | --- | | Representation | Selected or engineered input features | Features learned jointly across layers | | Useful architectures | Linear models, decision trees, SVMs | MLPs, CNNs, RNNs and gated recurrent models | | Training from scratch | Can be effective with modest structured datasets | Large models often need substantial data and computation | | Interpretation | Some models have directly inspectable rules or coefficients | Learned internal representations can be harder to interpret | Feature learning, flexible nonlinear modeling and reuse of pretrained representations are major advantages. Limitations include computation and memory costs, overfitting with insufficient representative data, difficulty interpreting predictions, and reduced reliability when deployment data differs from training data. High training accuracy does not establish generalization. These are tendencies, not absolute divisions. Traditional ML can also learn representations; deep models can process tabular data. A pretrained deep model may be useful even with a small target dataset. Select the model according to the task, available data, resources and validation results. **Question. Explain the motivation for deep learning and describe applications in at least three domains.** **Answer.** Deep learning is motivated by the need to learn useful hierarchical representations and nonlinear relationships directly from complex data. Multiple trainable layers compose feature transformations, and the task loss can train those transformations jointly with the predictor. Larger datasets, GPU computation and improved optimization methods have made this practical. In computer vision, CNNs learn visual features for image classification and object localization. In language processing, neural sequence models use word representations and context for sentiment analysis or translation. In speech processing, neural models map acoustic patterns to words. The benefits must be balanced against data requirements, computation, overfitting and interpretability limitations. | Task | Suitable architecture | Why / output | | --- | --- | --- | | Retinal images,5classes | CNN | Spatial image features;5-way softmax for exclusive classes | | Alphabet images | CNN | Edges/shapes compose into characters; class-count softmax | | Weather from a temporal history | RNN/LSTM or GRU | Ordered history; regression output for numeric weather quantities, suitable classification head for categorical weather | Architecture choice is justified by data structure. A fully connected model can technically take flattened images, but ignores spatial locality; sequential models preserve temporal order. --- ## TEE · R25 Q1b · Training and validation errors Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s1 ## Question **Question. Choose best and worst models from training/validation errors: M1=0.9/1.2; M2=0.3/0.6; M3=1.5/0.5; M4=1.2/1.1. Explain the reasoning and remedies.** (5 marks) ## Answer — step by step | Model | Train | Validation | | --- | --- | --- | | 1 | 0.9 | 1.2 | | 2 | 0.3 | 0.6 | | 3 | 1.5 | 0.5 | | 4 | 1.2 | 1.1 | **For the same validation set and same loss definition, M3 is best (0.5) and M1 is worst (1.2).** Choose by validation performance, not the smallest training loss. M2 has the smallest training error but a larger validation error than M3. M3’s validation loss below training loss is unusual but not logically impossible: training-only dropout/augmentation, different regularization inclusion or an easier validation sample can cause this. First verify comparable evaluation conditions and leakage-free splits. Error values alone do not prove overfitting or underfitting without baselines and learning curves. For poor generalization, inspect data quality/splits and learning curves; if overfitting is established, use regularization, dropout, more suitable data or early stopping; if underfitting is established, improve model capacity/features or training. Do not prescribe a larger network solely from these four numbers. --- ## TEE · R25 Q1c · CNN hierarchy and stride-two convolution Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s2 ## Question **Question. Explain why CNNs use a hierarchy of convolutional layers. Apply the printed3×3kernel to the5×5input with stride2.** (5 marks) ## Answer — step by step Early layers learn local edge/texture patterns; deeper layers combine them into parts and class-relevant configurations, with a growing receptive field. Nonlinearities make stacked layers more expressive than one linear convolution. ### Given Input: $$ \begin{bmatrix}2&3&8&2&3\\2&2&4&4&6\\1&2&1&2&9\\7&5&3&3&2\\2&4&3&5&4\end{bmatrix} $$ Kernel: $$ \begin{bmatrix}1&0&1\\2&4&0\\6&4&2\end{bmatrix} $$ Stride=2, zero padding=0, no bias supplied. ### To find Valid cross-correlation output (CNN convention, no kernel flip). ### Formula: which one and why $$ O=\left\lfloor\frac{N+2P-F}{S}\right\rfloor+1 $$ Multiply corresponding cells and sum. Here O=⌊(5−3)/2⌋+1=2 in each direction. ### Step 1. Output (1,1) from input top-left (1,1) $$ \begin{bmatrix}2&3&8\\2&2&4\\1&2&1\end{bmatrix} $$ 2×1 + 3×0 + 8×1 + 2×2 + 2×4 + 4×0 + 1×6 + 2×4 + 1×2 = 2 + 0 + 8 + 4 + 8 + 0 + 6 + 8 + 2 = 38 ### Step 2. Output (1,2) from input top-left (1,3) $$ \begin{bmatrix}8&2&3\\4&4&6\\1&2&9\end{bmatrix} $$ 8×1 + 2×0 + 3×1 + 4×2 + 4×4 + 6×0 + 1×6 + 2×4 + 9×2 = 8 + 0 + 3 + 8 + 16 + 0 + 6 + 8 + 18 = 67 ### Step 3. Output (2,1) from input top-left (3,1) $$ \begin{bmatrix}1&2&1\\7&5&3\\2&4&3\end{bmatrix} $$ 1×1 + 2×0 + 1×1 + 7×2 + 5×4 + 3×0 + 2×6 + 4×4 + 3×2 = 1 + 0 + 1 + 14 + 20 + 0 + 12 + 16 + 6 = 70 ### Step 4. Output (2,2) from input top-left (3,3) $$ \begin{bmatrix}1&2&9\\3&3&2\\3&5&4\end{bmatrix} $$ 1×1 + 2×0 + 9×1 + 3×2 + 3×4 + 2×0 + 3×6 + 5×4 + 4×2 = 1 + 0 + 9 + 6 + 12 + 0 + 18 + 20 + 8 = 74 ### Final answer $$ \begin{bmatrix}38&67\\70&74\end{bmatrix} $$ **Source / convention note.** Padding is not specified; valid/no-padding is declared. CNN cross-correlation convention is used without kernel reversal. --- ## TEE · R25 Q1d · BPTT and recurrent strengths Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s3 ## Question **Question. Compare BPTT with feedforward backpropagation and state the strengths of RNNs.** (5 marks) ## Answer — step by step **Definition.** Backpropagation through time (BPTT) differentiates the loss of an RNN by applying backpropagation to its temporally unrolled computational graph and summing contributions to shared parameters. ### Explanation and working Unroll the recurrent network for the sequence length. Run the forward calculation in time order. Calculate the specified loss. Differentiate backwards through the unrolled graph. Add contributions to each shared weight from every time it was used. Update each shared weight once for the selected sequence or batch. This is backpropagation through time, or BPTT. ![Unroll forward through time, then accumulate backward contributions for the shared parameters.](https://deep-learning-study-room.vercel.app/assets/figures/v_bptt.svg) Diagram: Unroll forward through time, then accumulate backward contributions for the shared parameters.. Labels: One shared weight receives contributions from many times | One shared weight receives contributions from many times | Unrolling draws repeated uses of one cell, not independent copies of its | weights. | Time 1 | Same weights W | Time 2 | Same weights W | Time 3 | Same weights W | Gradients flow backward through the state dependencies | Add shared-weight contributions | Total gradient = contribution 1 + contribution 2 + | contribution 3 | After summing the required contributions, update each shared parameter once. Unroll forward through time, then accumulate backward contributions for the shared parameters. Tap the diagram to enlarge. For tanh, let δₜ mean the gradient with respect to the hidden preactivation. The following form allows losses at several time steps; a missing output loss contributes zero. $$ \delta_t=\left(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1}\right)\odot(1-h_t^2) $$ $$ \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T},\quad \frac{\partial L}{\partial W_{xh}}=\sum_t\delta_t x_t^{T},\quad \frac{\partial L}{\partial b_h}=\sum_t\delta_t $$ The symbol ⊙ means pairwise multiplication. The transpose T turns a column into a row. δnext carries the effect of future time steps. The summation is essential because these are shared parameters. ### One BPTT delta at h = 0.5 **Question. In a scalar tanh RNN, h\_t = 0.5, current output-path derivative = 0.2, next-step delta = 0.3 and recurrent weight = 0.4. Find the current preactivation delta.** ### Given Activated h\_t = 0.5; q\_t = 0.2; δ\_(t+1) = 0.3; recurrent weight u = 0.4. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.4(0.3)=0.12 $$ **Step 2: add the current output contribution.** $$ 0.2+0.12=0.32 $$ **Step 3: calculate the current tanh slope.** $$ 1-(0.5)^2=0.75 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.32)(0.75)=0.24 $$ **Final answer:** δ\_t = 0.24. ### One BPTT delta at h = -0.2 **Question. In a scalar tanh RNN, h\_t = -0.2, current output-path derivative = 0.1, next-step delta = 0 and recurrent weight = 0.5. Find the current preactivation delta.** ### Given Activated h\_t = -0.2; q\_t = 0.1; δ\_(t+1) = 0; recurrent weight u = 0.5. ### To find Current preactivation delta δ\_t. ### Formula — which one and why Two paths affect the current hidden state: its current output and the future recurrent step. Add both before differentiating the current tanh. q\_t is already a derivative with respect to the hidden activation; the next delta is with respect to next preactivation. $$ \delta_t=(q_t+u\delta_{t+1})(1-h_t^2) $$ **Yaad rakhne ka cue:** Future aur current dono add; tanh slope uske baad. ### Solution — step by step **Step 1: calculate the future path contribution.** $$ 0.5(0)=0 $$ **Step 2: add the current output contribution.** $$ 0.1+0=0.1 $$ **Step 3: calculate the current tanh slope.** $$ 1-(-0.2)^2=0.96 $$ **Step 4: multiply by that slope.** $$ \delta_t=(0.1)(0.96)=0.096 $$ **Final answer:** δ\_t = 0.096. | Feedforward backprop | BPTT | | --- | --- | | Propagate through layers in one acyclic graph | Unroll recurrence through time, then propagate through the resulting acyclic graph | | Layer-specific parameters usually distinct | Recurrent parameters shared across time; sum gradient contributions over all uses | | No persistent temporal state by default | Hidden state connects earlier and later steps | | Depth-related gradient issues | Long time paths add vanishing/exploding difficulty; truncation limits path length | RNN strengths: variable-length sequences, shared parameters and order-dependent context; applications include language, speech and time series. Memory is not perfect and vanilla RNNs struggle with long dependencies. --- ## TEE · R25 Q2a · Activation, ReLU and numeric outputs Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s4 ## Question **Question. Explain why activation functions are needed and sketch ReLU. Is sigmoid appropriate for an unscaled house-price output? Compute softmax([1.2,3.4,0.9]) and sigmoid(2.5).** (10 marks) ## Answer — step by step An activation function transforms a neuron's weighted sum into its output. Nonlinear activations allow successive layers to represent nonlinear relationships. If every layer is only linear or affine, composing those layers still gives one affine transformation; depth alone does not create a nonlinear decision boundary. ![Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.](https://deep-learning-study-room.vercel.app/assets/figures/v_nonlinear.svg) Diagram: Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation.. Labels: Two affine layers still make one affine rule | Two affine layers still make one affine rule | Without a nonlinear activation, these two stages can be combined. | Layer 1 and layer 2 | h = 2x + 1 | y = 3h + 4 | Substitute and simplify | y = 3(2x + 1) + 4 | y = 6x + 7 | A nonlinear activation between layers prevents this simple affine collapse. Substituting one affine rule into another still gives a single affine rule. A nonlinear activation changes this limitation. Tap the diagram to enlarge. ### Why two linear layers are still linear in the input For the first layer let h = 2x + 1. Let the second output be y = 3h + 4. Substitute the first rule into the second: $$ y=3(2x+1)+4=6x+3+4=6x+7 $$ This is one straight-line rule. A nonlinear activation between the two calculations changes what the combined model can represent. | Activation | Output range | Useful property | Main limitation | | --- | --- | --- | --- | | Sigmoid | Strictly between 0 and 1 | Useful as a binary-class probability output | Saturates at large positive or negative inputs; gradients become small | | Tanh | Strictly between −1 and 1 | Can represent signed, zero-centered activations | Also saturates and can contribute to vanishing gradients | | ReLU | From 0 upward | Simple; derivative is 1 for positive inputs | A unit that stays on the negative side receives zero local gradient | Sigmoid and tanh do not return negative and positive versions of the same scale: their ranges differ. ReLU is not restricted to values below 1. For multiclass output, softmax normalizes a group of class scores so the probabilities sum to 1; it acts on the score vector rather than independently applying sigmoid to each score. **Exam conclusion:** select the activation according to the layer's job and task. Hidden layers need useful nonlinear transformations; the output activation must match the meaning of the desired prediction. $$ \mathrm{ReLU}(z)=\max(0,z) $$ za Unscaled house prices are not restricted to(0,1), so a sigmoid cannot directly represent them. Use a linear regression output (or a justified positive-valued transform). A sigmoid could be used only with a deliberately bounded/normalized target and inverse transformation. ### Given Logits=[1.2,3.4,0.9]; separate sigmoid input=2.5. ### To find Normalized class probabilities and sigmoid value. ### Formula: which one and why $$ p_i=\frac{e^{z_i-m}}{\sum_je^{z_j-m}},\quad m=\max_j z_j;\qquad\sigma(z)=\frac1{1+e^{-z}} $$ ### Step 1. Subtract the maximum for stable softmax [−2.2,0,−2.5]. This common shift leaves softmax unchanged. ### Step 2. Exponentiate then sum [0.11080315836233387, 1.0, 0.0820849986238988]; sum=1.192888156986. ### Step 3. Divide each by the same sum | Logit | Exponential / sum | Probability | | --- | --- | --- | | 1.2 | 0.110803158362/1.192888156986 | 0.092886460 | | 3.4 | 1.000000000000/1.192888156986 | 0.838301558 | | 0.9 | 0.082084998624/1.192888156986 | 0.068811982 | ### Step 4. Compute sigmoid separately e^(−2.5)=0.082084998624;1+e^(−2.5)=1.082084998624; reciprocal=0.924141820. ### Final answer Softmax=[0.09288646, 0.838301558, 0.068811982], sum=1.000000000; sigmoid(2.5)=0.924141820. --- ## TEE · R25 Q2b · Vanishing/exploding gradients and GRU Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s5 ## Question **Question. Explain vanishing/exploding gradients in RNNs and the gates of a GRU.** (10 marks) ## Answer — step by step **Definition.** Vanishing and exploding gradients describe the attenuation or amplification of backward derivatives through repeated Jacobian products. In recurrent networks, these products extend across time and affect long-range credit assignment. ### Explanation and working In BPTT, the gradient repeatedly multiplies recurrent weights and activation derivatives. Products of many small factors can approach zero; early information then receives very little learning signal. Large repeated factors can make the gradient explode. These are about training signals, not an instruction that every hidden-state value must become zero or infinity. ![Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.](https://deep-learning-study-room.vercel.app/assets/figures/v_vanishing.svg) Diagram: Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates.. Labels: Repeated multiplication changes gradient size | Repeated multiplication changes gradient size | These scalar examples illustrate one part of the BPTT mechanism. | Repeated factors below 1 | 0.5 × 0.5 × … × 0.5 | Ten factors: 0.0009765625 | Earlier learning signal is tiny. | Repeated large factors | 2 × 2 × … × 2 | Ten factors: 1,024 | Updates can become unstable. | Gradient clipping limits explosion. It does not rebuild a gradient that has | already vanished. Follow the repeated products. Small factors can weaken the learning signal; large products can destabilize updates. Tap the diagram to enlarge. **Numerical intuition:** 0.5¹⁰ = 0.0009765625, whereas 2¹⁰ = 1,024. Sigmoid's derivative is at most 0.25 and becomes even smaller in saturation. Tanh also saturates for large magnitude inputs. Useful remedies include LSTM or GRU gates and cell paths, suitable initialization and learning rate, and gradient clipping for explosion. Clipping does not by itself solve vanishing gradients. LSTM makes long-range learning easier; it does not guarantee perfect unlimited memory. ### Why the effect grows with sequence length For a vanilla tanh RNN, each hidden-state transition contributes a Jacobian. The derivative through several transitions contains a product of these Jacobians: $$ \frac{\partial h_t}{\partial h_{t-1}}=\operatorname{diag}(1-h_t^2)W_{hh} $$ $$ \frac{\partial h_T}{\partial h_t}=\prod_{k=T}^{t+1}\frac{\partial h_k}{\partial h_{k-1}} $$ The product is ordered from the latest transition toward the earliest. Small activation derivatives and recurrent transformations can attenuate the signal; repeated amplification can make it grow. A one-dimensional product illustrates the issue, but a vector RNN depends on the full matrix products, not one weight alone. ### Gradient norm clipping: a complete numerical Norm clipping rescales an oversized gradient while retaining its direction. It limits the update size caused by an exploding gradient. $$ g_{\mathrm{clipped}}=g\min\left(1,\frac{\tau}{\lVert g\rVert_2}\right) $$ If g is the zero vector, leave it unchanged. The threshold τ is positive. This is global norm clipping for the vector being considered; elementwise clipping is a different operation. **Question. A gradient is g = [6, 8]ᵀ. Clip its Euclidean norm to at most τ = 5, then update w = [1, 1]ᵀ with learning rate η = 0.1.** ### Given Gradient g = [6,8]ᵀ; norm threshold τ = 5. Old weights w = [1,1]ᵀ; η = 0.1. ### To find Find the clipped gradient and updated weights. ### Formula — which one and why **Gradient norm clipping.** If the vector norm exceeds the allowed threshold, multiply every component by the same scale. This changes magnitude while preserving direction. g = gradient vector; τ (tau) = maximum norm; ||g||₂ = square root of the sum of squared components; η = learning rate. $$ \lVert g\rVert_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min\left(1,\frac\tau{\lVert g\rVert_2}\right) $$ $$ w_{\mathrm{new}}=w-\eta g_{\mathrm{clip}} $$ **Yaad rakhne ka cue:** Bada gradient? Sab components ko ek hi factor se chhota karo. ### Solution — step by step **Step 1: calculate the original norm.** $$ \lVert g\rVert_2=\sqrt{6^2+8^2}=\sqrt{36+64}=10 $$ **Step 2: find the scale.** Since 10 > 5, scale the vector by 5/10 = 0.5. $$ g_{\mathrm{clipped}}=0.5\begin{bmatrix}6\\8\end{bmatrix}=\begin{bmatrix}3\\4\end{bmatrix} $$ **Step 3: verify the clipped norm.** $$ \sqrt{3^2+4^2}=\sqrt{9+16}=5 $$ **Step 4: update each parameter with the clipped gradient.** $$ w_{1,\mathrm{new}}=1-0.1(3)=0.7 $$ $$ w_{2,\mathrm{new}}=1-0.1(4)=0.6 $$ **Final answer:** clipped gradient [3, 4]ᵀ and new weights [0.7, 0.6]ᵀ. Clipping does not increase a vanishing gradient and does not replace the need for an appropriate recurrent architecture. ### Gradient norm clipping: [3, 4] **Question. Clip gradient g = [3, 4] to maximum L2 norm 10. Use global norm clipping.** ### Given g = [3, 4]; threshold τ = 10. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 3^2+4^2=25 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{25}=5 $$ **Step 3: choose the shared scale.** $$ c=\min(1,10/5)=1 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=3(1)=3 $$ $$ g_2^{\mathrm{clip}}=4(1)=4 $$ **Final answer:** Clipped gradient = [3, 4]. ### Gradient norm clipping: [0, 12] **Question. Clip gradient g = [0, 12] to maximum L2 norm 3. Use global norm clipping.** ### Given g = [0, 12]; threshold τ = 3. ### To find The norm, scaling factor and clipped gradient. ### Formula — which one and why Global norm clipping scales the entire list by one common factor only if its norm exceeds the threshold. τ is the maximum permitted norm, not a per-entry bound. $$ \|g\|_2=\sqrt{\sum_i g_i^2} $$ $$ g_{\mathrm{clip}}=g\min(1,\tau/\|g\|_2) $$ **Yaad rakhne ka cue:** Norm bada ho tab sab entries ko same ratio se chhota karo. ### Solution — step by step **Step 1: calculate the squared norm.** $$ 0^2+12^2=144 $$ **Step 2: take the square root.** $$ \|g\|_2=\sqrt{144}=12 $$ **Step 3: choose the shared scale.** $$ c=\min(1,3/12)=0.25 $$ **Step 4: scale every entry.** $$ g_1^{\mathrm{clip}}=0(0.25)=0 $$ $$ g_2^{\mathrm{clip}}=12(0.25)=3 $$ **Final answer:** Clipped gradient = [0.0, 3.0]. **Definition.** A gated recurrent unit (GRU) is a recurrent architecture that uses reset and update gates to control its hidden-state transition. Unlike a standard LSTM, it does not carry a separate cell state alongside the hidden state. ### Gate equations and the convention used here The reset gate r controls how the previous hidden state enters candidate formation. The update gate z controls the mixture of the old state and the candidate. Both are calculated from the current input and previous hidden state: $$ r_t=\sigma(W_rx_t+U_rh_{t-1}+b_r) $$ $$ z_t=\sigma(W_zx_t+U_zh_{t-1}+b_z) $$ The candidate equation used in these notes applies reset before the recurrent candidate transformation: $$ \widetilde h_t=\tanh(W_hx_t+U_h(r_t\odot h_{t-1})+b_h) $$ These notes define z as the fraction of **new candidate** admitted: $$ h_t=(1-z_t)\odot h_{t-1}+z_t\odot\widetilde h_t $$ Thus z near 0 favors retention of the old hidden state, while z near 1 favors the candidate. The reset gate affects the candidate calculation; it does not directly multiply the retained old-state term in the final mixture. ![First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.](https://deep-learning-study-room.vercel.app/assets/figures/gru.svg) Diagram: First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention.. Labels: h previous | Reset r × h | Candidate = tanh(...) | (1 − z) × h previous | z × candidate | + | New hidden state | Here z means the fraction of NEW content admitted. First calculate the reset-dependent candidate. Then combine the old hidden state and the candidate using the stated update-gate convention. Tap the diagram to enlarge. ### Complete numerical with supplied gate values **Question.** A scalar GRU has hₜ₋₁ = 0.4, reset gate rₜ = 0.5, update gate zₜ = 0.25, xₜ = 1, Wₕ = 0.2, Uₕ = 0.5 and bₕ = 0. Calculate the candidate and new hidden state using hₜ = (1−zₜ)hₜ₋₁ + zₜh̃ₜ. ### Given Old hidden state = 0.4; r = 0.5; z = 0.25. x = 1; W = 0.2; U = 0.5; b = 0. Here z weights the candidate in the final mixture. ### To find Find candidate and new hidden state. ### Formula — which one and why **GRU reset and state mixture.** The reset gate controls old-state information entering the candidate. The stated update convention then mixes the original old state and the candidate. r = reset gate; z = update gate; h\_old = previous state; h\_tilde = candidate. Here z weights the candidate; some questions instead define z to weight the old state. $$ \widetilde h=\tanh(Wx+U(r\odot h_{\mathrm{old}})+b) $$ $$ h=(1-z)\odot h_{\mathrm{old}}+z\odot\widetilde h $$ **Yaad rakhne ka cue:** Reset candidate ko control; update gate old-new mixture ko control. ### Solution — step by step **Step 1: apply the reset gate to the previous hidden state.** $$ r_th_{t-1}=0.5(0.4)=0.2 $$ The gates are already supplied as outputs, so no further sigmoid calculation is needed. **Step 2: calculate the candidate preactivation.** $$ a_{\mathrm{candidate}}=W_hx_t+U_h(r_th_{t-1})+b_h $$ $$ a_{\mathrm{candidate}}=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh to obtain the candidate.** $$ \widetilde h_t=\tanh(0.3)\approx0.291312612 $$ **Step 4: calculate the two mixture contributions.** $$ (1-z_t)h_{t-1}=(1-0.25)(0.4)=0.3 $$ $$ z_t\widetilde h_t=0.25\tanh(0.3)\approx0.072828153 $$ **Step 5: add the contributions.** $$ h_t=0.3+0.25\tanh(0.3)\approx0.372828153 $$ **Final answer:** candidate ≈ 0.291313 and new hidden state ≈ 0.372828. The new scalar state lies between the old state 0.4 and the candidate 0.291313, as expected for this convex mixture. ### If the question uses the other update-gate convention Some texts define z as the fraction of the **old state retained**: $$ h_t=z_th_{t-1}+(1-z_t)\widetilde h_t $$ Under that explicitly different convention, the same numerical value z = 0.25 gives: $$ h_t=0.25(0.4)+0.75\tanh(0.3)\approx0.318484459 $$ This is not the answer to the worked question above, which states the first convention. To represent an identical mixture while switching conventions, the gate values must be complemented as well. Always identify the final mixture equation before substituting values. ### Relationship to LSTM A basic GRU has three affine transformations: reset, update and candidate. A standard LSTM has four: forget, input, output and candidate. With equal input and hidden dimensions and the same bias convention, GRU therefore has fewer recurrent parameters. This is a parameter-count comparison, not a guarantee of higher accuracy or faster execution for every task. ### Retain-old GRU with gate 0 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = 0.4, z = 0, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = 0.4; z = 0; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(0.4)=0.2 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(0.2)+0=0.3 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.3)\approx0.291312612 $$ **Step 4: retain the old-state fraction.** $$ 0(0.4)=0 $$ **Step 5: calculate the candidate fraction.** $$ (1-0)(0.291312612)\approx0.291312612 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=0+(0.291312612)\approx0.291312612 $$ **Final answer:** Candidate ≈ 0.291312612; new state ≈ 0.291312612. ### Retain-old GRU with gate 0.8 **Question. Use h\_new = z h\_old+(1−z) candidate. Given h\_old = -0.2, z = 0.8, r = 0.5, x = 1, W = 0.2, U = 0.5 and b = 0, calculate the candidate and new state.** ### Given h\_old = -0.2; z = 0.8; r = 0.5; x = 1. W = 0.2; U = 0.5; b = 0; z is the retain-old gate. ### To find Candidate and final hidden state. ### Formula — which one and why The reset gate changes old-state input to the candidate. The update gate separately mixes the original old state and candidate. The question explicitly makes z multiply the old state. r and z do different jobs. $$ \widetilde h=\tanh(Wx+U(rh_{\mathrm{old}})+b) $$ $$ h_{\mathrm{new}}=zh_{\mathrm{old}}+(1-z)\widetilde h $$ **Yaad rakhne ka cue:** Reset sirf candidate mein; update mix mein original old state. ### Solution — step by step **Step 1: reset the old state for candidate formation.** $$ rh_{\mathrm{old}}=0.5(-0.2)=-0.1 $$ **Step 2: calculate the candidate raw sum.** $$ a=0.2(1)+0.5(-0.1)+0=0.15 $$ **Step 3: apply tanh.** $$ \widetilde h=\tanh(0.15)\approx0.148885034 $$ **Step 4: retain the old-state fraction.** $$ 0.8(-0.2)=-0.16 $$ **Step 5: calculate the candidate fraction.** $$ (1-0.8)(0.148885034)\approx0.029777007 $$ **Step 6: add the two fractions.** $$ h_{\mathrm{new}}=-0.16+(0.029777007)\approx-0.130222993 $$ **Final answer:** Candidate ≈ 0.148885034; new state ≈ -0.130222993. --- ## TEE · R25 Q3a · 4–3–2 forward pass and dropout Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s6/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s6 ## Question **Question. For the printed network, inputs=[0.1,0.2,0.3,0.4], targets=[0.01,0.99], w1…w18=[0.10,0.15,…,0.95] and both layer biases=0.60. Find total forward error, then drop hidden neuron H12 and compare errors.** (10 marks) ## Answer — step by step ### Given Input→hidden rows (destinations H11,H12,H13): $$ \begin{bmatrix}0.1&0.25&0.4&0.55\\0.15&0.3&0.45&0.6\\0.2&0.35&0.5&0.65\end{bmatrix} $$ Hidden→output rows (destinations O1,O2): $$ \begin{bmatrix}0.7&0.8&0.9\\0.75&0.85&0.95\end{bmatrix} $$ 4inputs → 3sigmoid hidden units → 2sigmoid outputs → Half sum of squared errors ### To find Original and dropped forward outputs and loss. ### Formula: which one and why The paper omits activation and error convention. Declare sigmoid at both layers and half-SSE. For the given single dropped unit use an ordinary binary mask, no additional inverted-dropout scaling because no keep probability is supplied. $$ h_j=\sigma(\sum_iW_{ji}x_i+0.6),\quad o_k=\sigma(\sum_jV_{kj}h_j+0.6),\quad E=\frac12\sum_k(o_k-t_k)^2 $$ ### Step 1. Multiply and add each hidden neuron | Neuron | Products + bias | Net | Sigmoid | | --- | --- | --- | --- | | 1 | 0.1×0.1 + 0.2×0.25 + 0.3×0.4 + 0.4×0.55 +0.6 | 1.000000000 | 0.731058578630 | | 2 | 0.1×0.15 + 0.2×0.3 + 0.3×0.45 + 0.4×0.6 +0.6 | 1.050000000 | 0.740774899182 | | 3 | 0.1×0.2 + 0.2×0.35 + 0.3×0.5 + 0.4×0.65 +0.6 | 1.100000000 | 0.750260105595 | ### Step 2. Compute original output nets and activations | Output | Full substitution | Net | Output | | --- | --- | --- | --- | | 1 | 0.731058578630×0.7 + 0.740774899182×0.8 + 0.750260105595×0.9 +0.6 | 2.379595019422 | 0.915258028946 | | 2 | 0.731058578630×0.75 + 0.740774899182×0.85 + 0.750260105595×0.95 +0.6 | 2.490699698593 | 0.923487256983 | ### Step 3. Square both output errors and halve their sum | Output | Prediction−target | Square | Half square | | --- | --- | --- | --- | | 1 | 0.9152580289459775−0.01=0.905258028946 | 0.819492098971 | 0.409746049486 | | 2 | 0.9234872569833742−0.99=-0.066512743017 | 0.004423944984 | 0.002211972492 | Total=0.411958021977. ### Step 4. Drop H12 only and recompute both outputs Masked hidden activations=[0.7310585786300049, 0, 0.7502601055951177]. Other hidden activations and all weights are unchanged. | Output | Net with H12 term zero | Sigmoid | | --- | --- | --- | | 1 | 0.731058578630×0.7+0×0.8+0.750260105595×0.9+0.6=1.786975100077 | 0.856556014402 | | 2 | 0.731058578630×0.75+0×0.85+0.750260105595×0.95+0.6=1.861041034288 | 0.865418242930 | ### Step 5. Compute dropped loss and compare Half squares: 0.5×(0.856556014402−0.01)² + 0.5×(0.865418242930−0.99)² = 0.366088849857. Difference dropped−original=-0.045869172120. ### Final answer **Original E=0.411958021977; dropped E=0.366088849857.** A lower single-example masked loss is not evidence that dropout always lowers training or test loss. **Source / convention note.** Activation/loss/dropout scaling missing from original; result is conditional on explicit standard classroom assumptions. --- ## TEE · R25 Q3b · Four CNN design explanations Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s7/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s7 ## Question **Question. Explain(i)parameter sharing and sparse connections,(ii)5×5versus3×3kernel trade-offs,(iii)ReLU’s effect on learning,(iv)whether CNNs can work without pooling.** (10 marks) ## Answer — step by step | Part | Explanation | | --- | --- | | i | Local receptive fields create sparse connectivity. The same filter weights are reused across spatial positions, reducing parameters and recognizing a feature in different positions. This gives translation equivariance under appropriate boundary/stride conditions, not automatic complete invariance. | | ii | At fixed channel counts,5×5uses25weights per channel-pair,3×3uses9. Two stride1 3×3layers yield an effective5×5receptive field with an extra nonlinearity and18C²weights when all channel widths equalC, versus25C². Intermediate channels, biases and compute/memory can change the comparison. | | iii | ReLU passes positive values/gradient1 and zeros negative values/gradient0. It introduces nonlinearity and avoids positive-side sigmoid saturation, but can produce inactive/dead units. It does not eliminate all gradient problems. | | iv | Yes. Strided convolutions can downsample; global pooling or other aggregation may form the head. Removing pooling without another reduction can increase feature-map sizes/compute. Pooling is a design choice, not a mathematical necessity. | --- ## TEE · R25 Q4a · 64×64 CNN shape and parameter table Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s8/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s8 ## Question **Question. For input64×64×3: conv3×3,32filters,P1,S1; maxpool2×2,S2; conv3×3,64,P1,S1; pool2,S2; conv3×3,128,P1,S1; pool2,S2; FC256; softmax43. Find each shape and all weights/biases.** (10 marks) ## Answer — step by step ### Given Input=[64, 64, 3] | Layer | Kernel | Stride | Padding | Filters / units | | --- | --- | --- | --- | --- | | Conv | 3 | 1 | 1 | 32 | | Pool | 2 | 2 | 0 | 0 | | Conv | 3 | 1 | 1 | 64 | | Pool | 2 | 2 | 0 | 0 | | Conv | 3 | 1 | 1 | 128 | | Pool | 2 | 2 | 0 | 0 | | Dense | 0 | 0 | 0 | 256 | | Dense | 0 | 0 | 0 | 43 | ### To find Each output shape and all trainable weights plus biases. ### Formula: which one and why $$ O=\left\lfloor\frac{N+2P-F}{S}\right\rfloor+1 $$ $$ P_{conv}=(F_hF_wC_{in}+1)C_{out},\quad P_{dense}=(n_{in}+1)n_{out} $$ Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit. ### Step 1. Work layer by layer | Layer | Input | Full substitution and reason | Output | Parameters | | --- | --- | --- | --- | --- | | Conv | 64×64×3 | ⌊(64+2×1−3)/1⌋+1=64; parameters=(3×3×3+1)×32=896 | 64×64×32 | 896 | | Pool | 64×64×32 | ⌊(64−2)/2⌋+1=32; channels stay 32; no trainable weights | 32×32×32 | 0 | | Conv | 32×32×32 | ⌊(32+2×1−3)/1⌋+1=32; parameters=(3×3×32+1)×64=18496 | 32×32×64 | 18496 | | Pool | 32×32×64 | ⌊(32−2)/2⌋+1=16; channels stay 64; no trainable weights | 16×16×64 | 0 | | Conv | 16×16×64 | ⌊(16+2×1−3)/1⌋+1=16; parameters=(3×3×64+1)×128=73856 | 16×16×128 | 73856 | | Pool | 16×16×128 | ⌊(16−2)/2⌋+1=8; channels stay 128; no trainable weights | 8×8×128 | 0 | | Dense | 8×8×128 | Flatten/input length=8×8×128=8192; parameters=(8192+1)×256=2097408 | 256 | 2097408 | | Dense | 1×1×256 | Flatten/input length=1×1×256=256; parameters=(256+1)×43=11051 | 43 | 11051 | ### Step 2. Sum only trainable parameters 896 + 18496 + 73856 + 2097408 + 11051 = 2201707 ### Final answer **Total parameters=2201707.** Shape and parameter count are separate results. --- ## TEE · R25 Q4b · Bipolar AND perceptron Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s9/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s9 ## Question **Question. Implement AND with bipolar inputs/targets and learning rate1.** (10 marks) ## Answer — step by step ### Given Truth table: (−1,−1)→−1;(−1,+1)→−1;(+1,−1)→−1;(+1,+1)→+1. η=1. ### To find A trained separating perceptron. ### Formula: which one and why Initial weights/bias and zero-threshold tie rule are unspecified. Choose w=(0,0),b=0; output+1ifnet≥0,else−1. Use error update w←w+η(t−y)x,b←b+η(t−y). A sign-only misclassification update differs by a factor2 but can also learn the gate. $$ z=w^Tx+b,\quad w\leftarrow w+(t-y)x,\quad b\leftarrow b+(t-y) $$ ### Step 1. Run all examples in the declared order | Epoch | x | Target | Old parameters | Net | Output | t−y | New parameters | | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | [-1, -1] | -1 | [0, 0]; b=0 | 0 | 1 | -2 | [2, 2]; b=-2 | | 1 | [-1, 1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 | | 1 | [1, -1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 | | 1 | [1, 1] | 1 | [2, 2]; b=-2 | 2 | 1 | 0 | [2, 2]; b=-2 | | 2 | [-1, -1] | -1 | [2, 2]; b=-2 | -6 | -1 | 0 | [2, 2]; b=-2 | | 2 | [-1, 1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 | | 2 | [1, -1] | -1 | [2, 2]; b=-2 | -2 | -1 | 0 | [2, 2]; b=-2 | | 2 | [1, 1] | 1 | [2, 2]; b=-2 | 2 | 1 | 0 | [2, 2]; b=-2 | ### Step 2. Verify every truth-table row with final parameters | x | Net | Output | Target | | --- | --- | --- | --- | | [-1, -1] | -6 | -1 | -1 | | [-1, 1] | -2 | -1 | -1 | | [1, -1] | -2 | -1 | -1 | | [1, 1] | 2 | 1 | 1 | ### Final answer **One valid learned solution: w=[2, 2], b=-2.** Last full epoch makes no changes; all four rows are correct. **Source / convention note.** Initialization, order and threshold convention explicitly supplied as assumptions; multiple equivalent separating weights exist. --- ## TEE · R25 Q5a · ResNet shortcut and Inception1×1 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s10/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s10 ## Question **Question. Describe skip connections and their benefit in ResNet. Explain a1×1convolution’s role in Inception with an example.** (10 marks) ## Answer — step by step **Definition.** A residual network uses shortcut connections to learn a residual mapping relative to a block's input. A basic residual block adds the block transformation F(x) to a shortcut representation of x. $$ y=F(x)+x $$ If the desired mapping is H(x), the residual branch represents F(x) = H(x) − x. When the useful mapping is close to identity, a small residual can be easier to optimize than reconstructing identity through the entire branch. ![The two paths must have equal output shape before elementwise addition.](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The two paths must have equal output shape before elementwise addition.. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The two paths must have equal output shape before elementwise addition. Tap the diagram to enlarge. ### Effect on gradient propagation For the identity-shortcut formulation, the derivative includes a direct identity contribution. $$ \frac{\partial y}{\partial x}=\frac{\partial F}{\partial x}+I $$ This provides a path for information and derivatives that does not require passing through every weight layer in the residual branch. It helps optimize deep networks and addresses the degradation problem. It does not imply that every possible ResNet is immune to vanishing gradients. ### Projection when dimensions differ If channels or spatial dimensions change, use a projection shortcut, often a 1 × 1 convolution with suitable stride. $$ y=F(x)+W_sx $$ An identity shortcut has no learned parameters. A projection does. For Cᵢ input channels and Cₒ output channels, a biased 1 × 1 projection has CᵢCₒ + Cₒ parameters. The residual branch's output must match the projection output for addition. **Question.** Let x = [1, 2, 3] and F(x) = [0.1, −0.2, 0.3]. Find the residual sum. ### Given x = [1,2,3]; residual branch F(x) = [0.1,−0.2,0.3]. ### To find Find the three components of x+F(x). ### Formula — which one and why **Residual addition.** Add corresponding components of the input shortcut and learned branch. Apply the output activation afterward if requested. x = shortcut/input; F(x) = residual branch output. Both arrays must have matching shapes before addition. $$ y=x+F(x) $$ **Yaad rakhne ka cue:** Same position ke numbers add; shape same rehni chahiye. ### Solution — step by step **Step 1: add the first matching pair.** $$ 1+0.1=1.1 $$ **Step 2: add the second matching pair.** $$ 2+(-0.2)=1.8 $$ **Step 3: add the third matching pair.** $$ 3+0.3=3.3 $$ **Step 4: assemble in the original order.** $$ y=[1.1,1.8,3.3] $$ The vector length remains 3. Concatenating these vectors would give length 6 and would implement a different operation. Any post-addition activation specified in the architecture is applied after this sum. **Definition.** A 1 × 1 convolution combines the input channels independently at each spatial location. A bottleneck uses fewer output channels before a larger spatial convolution, reducing its parameter and multiplication counts. ![Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.](https://deep-learning-study-room.vercel.app/assets/figures/v_bottleneck.svg) Diagram: Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size.. Labels: Compare the two paths in the worked question | Compare the two paths in the worked question | Input 28 × 28 × 192. Output 28 × 28 × 32. Count multiplications (MACs). | DIRECT | One 5 × 5 convolution | 28 × 28 × 32 × 25 × 192 | = 120,422,400 multiplications | REDUCE | 1 × 1: 192 → 16 | 28 × 28 × 16 × 192 | = 2,408,448 | 5 × 5: 16 → 32 | 28 × 28 × 32 × 25 × 16 | = 10,035,200 | Total with reduction = 12,443,648 MACs | Reducing depth to 16 before the 5 × 5 filter saves about 89.67% of these | multiplications. Compare the direct path with both stages of the bottleneck. The reduction changes channel depth, not spatial size. Tap the diagram to enlarge. **Question. Input is 28 × 28 × 192. Compare a direct same-padded 5 × 5 convolution with 32 filters against a 1 × 1 reduction to 16 channels followed by the same 5 × 5 convolution. Count MACs and ignore bias additions.** ### Given Both paths produce 28 × 28 × 32. The direct layer receives 192 channels; the bottleneck's spatial layer receives 16 channels. Stride is 1 and padding preserves spatial size. ### To find Direct MACs, total two-stage bottleneck MACs and percentage saved. ### Formula — which one and why Each output scalar uses one product per input-channel kernel entry. Multiply that cost by the number of output positions and channels. The two bottleneck layers require separate counts, which are added. $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ One MAC counts one multiply–accumulate pair. A convention counting multiplication and addition separately gives approximately twice as many FLOPs. ### Solution — step by step **Step 1: direct layer.** Each output uses 5 × 5 × 192 = 4,800 products. There are 28 × 28 × 32 = 25,088 outputs. $$ \mathrm{MACs}_{\mathrm{direct}}=25088(4800)=120422400 $$ **Step 2: 1 × 1 reduction.** There are 16 receiving filters, each spanning all 192 original channels. $$ \mathrm{MACs}_{1\times1}=28(28)(16)(192)=2408448 $$ **Step 3: reduced-depth spatial layer.** Its 32 filters now span 16 channels. $$ \mathrm{MACs}_{5\times5}=28(28)(32)(25)(16)=10035200 $$ **Step 4: add the two bottleneck costs.** $$ \mathrm{MACs}_{\mathrm{bottleneck}}=2408448+10035200=12443648 $$ **Step 5: calculate the saving.** $$ \mathrm{saving}=120422400-12443648=107978752 $$ $$ \mathrm{saving\ percentage}=100\frac{107978752}{120422400}\approx89.666667\% $$ **Final answer:** direct = 120,422,400 MACs; bottleneck = 12,443,648 MACs, saving about 89.67%. Spatial size is unchanged; the saving comes from reducing the channels entering the expensive 5 × 5 layer. If the reduction width is not supplied, use a symbol for it and label any numerical choice as an assumption. ### A second bottleneck with different dimensions **Question. An 8 × 8 × 16 input must produce 8 channels using a same-padded 3 × 3 convolution. Compare direct computation with a 1 × 1 reduction to 4 channels followed by the 3 × 3 layer. Count MACs and ignore bias additions.** ### Given Spatial output = 8 × 8; original channels = 16; final filters = 8; bottleneck channels = 4; stride = 1. ### To find Direct MACs, bottleneck MACs and saving. ### Formula — which one and why Use the same MAC formula above once for the direct path and once for each bottleneck stage. Every branch must include all of its layers. ### Solution — step by step **Step 1: direct spatial convolution.** $$ 8(8)(8)(3)(3)(16)=73728 $$ **Step 2: 1 × 1 reduction.** $$ 8(8)(4)(1)(1)(16)=4096 $$ **Step 3: reduced-depth 3 × 3 convolution.** $$ 8(8)(8)(3)(3)(4)=18432 $$ **Step 4: add the stages and subtract from the direct cost.** $$ 4096+18432=22528 $$ $$ 73728-22528=51200 $$ **Final answer:** direct = 73,728 MACs; bottleneck = 22,528 MACs; saving = 51,200 MACs, about 69.44%. Both paths produce 8 × 8 × 8, but they are different learned transformations and need not produce identical feature values. --- ## TEE · R25 Q5b · ReLU RNN state with declared orientation Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s11/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s11 ## Question **Question. Sketch a one-hidden-layer RNN and state its hidden-state formula. Compute hₜ for Wxh=[[0.2,0.4],[0.1,0.3]], Whh=[[0.5,0.6],[0.3,0.4]], b=[0.1,0.2], x=[1,0], previoush=[0,1], ReLU.** (10 marks) ## Answer — step by step ### Given **Input-to-hidden weight matrix Wₓₕ:** transforms the two current input values into contributions to the two hidden units. $$ W_{xh}=\begin{bmatrix}0.2&0.4\\0.1&0.3\end{bmatrix} $$ **Hidden-to-hidden weight matrix Wₕₕ:** transforms the two previous hidden activations into recurrent contributions. $$ W_{hh}=\begin{bmatrix}0.5&0.6\\0.3&0.4\end{bmatrix} $$ | Quantity | Given value | Meaning | | --- | --- | --- | | xₜ | [1, 0] | Current input row: two input values | | hₜ₋₁ | [0, 1] | Previous hidden-state row: two stored activations | | b | [0.1, 0.2] | One bias for each hidden unit | | Activation | ReLU(z) = max(0, z) | Apply separately to each hidden weighted sum | ### To find Sketch the one-hidden-layer RNN, state its recurrence, and calculate the two components of hₜ. ### Formula: which one and why One-hidden-layer recurrent neural network with a one-step recurrent connection Input x at time t passes through input-to-hidden weights W x h. The hidden layer adds that input contribution, the previous hidden state through recurrent weights W h h, and bias b, then applies ReLU to produce two hidden activations h at time t. The recurrent arrow passes through a one-step delay: the current hidden state becomes the previous hidden state for the next input. An optional output head maps the hidden state through W h y, output bias and an output activation to prediction y hat. Output-head parameters are illustrative and are not supplied or needed for this question. Input xₜ[1, 0]two input valuesWₓₕ+ bias bOne hidden layerzₜ = input + history + biashₜ = ReLU(zₜ)two hidden activationsWₕᵧOutput headŷₜillustrative onlyWₕₕ One-step delayprevious state hₜ₋₁+ output biasthen output activationThe same hidden layer and recurrent weights are reused at each time step. **Sketch to draw in the exam:** input → one hidden layer → output, with a recurrent arrow through a one-step delay back to the hidden layer. Wₓₕ connects input to hidden; Wₕₕ carries the previous hidden state. Wₕᵧ denotes a possible hidden-to-output mapping. The paper asks for hₜ only, so no output weights or output prediction are required. The printed vectors are horizontal and the weight-index convention is not specified. Adopt row vectors multiplied on the right: $$ h_t=\operatorname{ReLU}(x_tW_{xh}+h_{t-1}W_{hh}+b) $$ ### Step 1. Multiply the input row [1,0]Wxh=[1×0.2+0×0.1,1×0.4+0×0.3]=[0.2,0.4]. ### Step 2. Multiply the previous-state row [0,1]Whh=[0×0.5+1×0.3,0×0.6+1×0.4]=[0.3,0.4]. ### Step 3. Add componentwise including bias [0.2+0.3+0.1,0.4+0.4+0.2]=[0.6,1.0]. ### Step 4. Apply ReLU separately [max(0,0.6),max(0,1.0)]=[0.6,1.0]. ### Final answer **Row-vector convention: hₜ=[0.6,1.0].** If the supplied matrices instead map column inputs by Wxh x + Whh h +b, the untransposed arrays give[0.9,0.7]. That is a different orientation; explicitly state the formula. Transposing the matrices when switching convention preserves the same network. **Source / convention note.** Square matrices and printed horizontal vectors leave orientation ambiguous; both possible readings are explained, main answer follows row-vector form. --- ## TEE · R25 Q6a · 3–2–1 forward pass and update w7,w8 Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s12/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s12 ## Question **Question. For inputs[1,0,1], weights w1…w8=[0.2,−0.3,0.4,0.1,−0.5,0.2,−0.3,−0.2], biases[−0.4,0.2,0.1], target1 and sigmoid at all neurons: find prediction,error,and update w7,w8 with η=0.9.** (10 marks) ## Answer — step by step ### Given Diagram edge mapping: x1→h1=w1,x1→h2=w2;x2→h1=w3,x2→h2=w4;x3→h1=w5,x3→h2=w6;h1→out=w7,h2→out=w8. $$ \begin{bmatrix}0.2&0.4&-0.5\\-0.3&0.1&0.2\end{bmatrix} $$ [1,0,1] → Two sigmoid hidden units → One sigmoid output → Target1 ### To find Prediction, explicit error convention, two simultaneous output-weight updates. ### Formula: which one and why $$ E=\frac12(o-t)^2,\quad\delta_o=(o-t)o(1-o),\quad\frac{\partial E}{\partial w_k}=\delta_o h_j,\quad w_k^{new}=w_k-\eta\frac{\partial E}{\partial w_k} $$ The paper says error without naming its scaling. Use half squared error and also report residual/SSE, so the gradient convention is clear. ### Step 1. Calculate each hidden weighted sum then sigmoid | Unit | Products +bias | Net | Activation | | --- | --- | --- | --- | | 1 | 1×(0.2) + 0×(0.4) + 1×(-0.5) +(-0.4) | -0.7 | 0.331812227832 | | 2 | 1×(-0.3) + 0×(0.1) + 1×(0.2) +(0.2) | 0.10000000000000003 | 0.524979187479 | ### Step 2. Output weighted sum and sigmoid z=0.331812227832×(−0.3)+0.524979187479×(−0.2)+0.1=-0.104539505845; o=1/(1+e^(−z))=0.473888898824. ### Step 3. State error and loss Residual o−t=-0.526111101176; SSE=0.276792890781; halfSSE=0.138396445390. ### Step 4. Multiply the output chain-rule factors δ=(0.473888898824−1)×0.473888898824×(1−0.473888898824)=-0.131169078214. ### Step 5. Gradient for each edge | Weight | δ × its source activation | Gradient | | --- | --- | --- | | w7 | -0.131169078214×0.331812227832 | -0.043523504065 | | w8 | -0.131169078214×0.524979187479 | -0.068861036103 | ### Step 6. Update from the old weights | Weight | Old − learning rate × gradient | New | | --- | --- | --- | | w7 | -0.3−0.9×(-0.043523504065) | -0.260828846342 | | w8 | -0.2−0.9×(-0.068861036103) | -0.138025067507 | ### Final answer **w7′=-0.260828846342; w8′=-0.138025067507.** Only the requested two weights are updated; use the same old forward pass for both gradients. With unhalved SSE the gradients double, so numerical updates would differ. **Source / convention note.** Original does not specify loss scaling; half-SSE is declared and alternatives distinguished. --- ## TEE · R25 Q6b · AdaGrad stagnation versus RMSProp Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s13/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s13 ## Question **Question. A healthcare model trained with AdaGrad becomes stagnant after many epochs. Discuss AdaGrad and RMSProp in detail and their fundamental differences.** (10 marks) ## Answer — step by step **Definition.** AdaGrad is an adaptive first-order optimizer that scales each parameter's update using the inverse square root of the sum of its previous and current squared gradients. Each parameter maintains its own accumulator. ### State, effective learning rate and update For one parameter θ, let gₜ be its current gradient and s₀ = 0: $$ s_t=s_{t-1}+g_t^2 $$ $$ \eta_{\mathrm{effective},t}=\frac{\eta}{\sqrt{s_t}+\varepsilon} $$ $$ \theta_t=\theta_{t-1}-\eta_{\mathrm{effective},t}g_t $$ s stores squared-gradient history. Both positive and negative gradients increase this nonnegative sum after squaring. The effective learning rate is the coefficient multiplying the signed gradient; the actual displacement also depends on that gradient's magnitude and sign. ![The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.](https://deep-learning-study-room.vercel.app/assets/figures/v_adagrad.svg) Diagram: The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients.. Labels: AdaGrad accumulates all past squared gradients | AdaGrad accumulates all past squared gradients | Use the worked question: initial s = 0; gradients 2, then 1. | First gradient | s₁ = 0 + 2² = 4 | Second gradient | s₂ = 4 + 1² = 5 | Scale each step | Divide g by (√s + ε) | The accumulator cannot decrease. Use the current accumulator in each parameter | update. The accumulator retains every squared gradient. A larger accumulator reduces the coefficient applied to subsequent gradients. Tap the diagram to enlarge. ### Complete numerical: supplied gradients **Question.** Start θ₀ = 1, s₀ = 0, η = 0.1. Given gradients g₁ = 2 and g₂ = 1, perform two AdaGrad updates. Ignore epsilon only for this hand calculation because neither denominator is zero. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **AdaGrad.** Add the new squared gradient to all previous squares. Divide the signed current gradient by the square root of that accumulated magnitude. G = accumulated squared gradients (also denoted s in some lessons); g = current gradient; η = base learning rate; ε = small stabilizer, set to zero only where the question says so. $$ G_t=G_{t-1}+g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ **Yaad rakhne ka cue:** Square karo, purane sum mein jodo, root se scale karo. ### Solution — step by step The gradients are supplied by the question. They are not computed from an assumed objective, and g₂ is not replaced by the updated parameter. **Iteration 1 — step 1: update the accumulator.** $$ s_1=0+2^2=4 $$ **Step 2: calculate the effective learning rate.** $$ \eta_{\mathrm{effective},1}=\frac{0.1}{\sqrt4}=0.05 $$ **Step 3: apply the signed gradient.** $$ \theta_1=1-0.05(2)=0.9 $$ **Iteration 2 — step 1: retain s₁ and add the new squared gradient.** $$ s_2=4+1^2=5 $$ **Step 2: calculate the new coefficient.** $$ \eta_{\mathrm{effective},2}=\frac{0.1}{\sqrt5}\approx0.044721360 $$ **Step 3: update from θ₁.** $$ \theta_2=0.9-\frac{0.1(1)}{\sqrt5}\approx0.855278640 $$ **Final answer:** θ₁ = 0.9 and θ₂ ≈ 0.855279, with accumulators s₁ = 4 and s₂ = 5. ### Advantage, limitation and epsilon convention Coordinate-specific scaling can be useful when some features or gradients occur infrequently. Because s never decreases, its effective learning rate is nonincreasing when η and ε are fixed, and can become too small during prolonged training. The actual displacement magnitude need not decrease monotonically because gₜ also changes. ε is normally a small positive stabilizer. These notes place it outside the square root. If a question places it inside, use that specified denominator consistently; the two placements are not numerically identical. **Definition.** RMSProp is an adaptive optimizer that normalizes the current gradient by the root of an exponentially weighted moving average of squared gradients. Its decaying history addresses AdaGrad's permanently growing accumulator. ### Recurrence and interpretation Using decay β and zero initial state: $$ s_t=\beta s_{t-1}+(1-\beta)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\varepsilon} $$ β is the fraction of the previous squared-gradient average retained, not a signed velocity. Some papers use ρ for this same decay. The square is applied to the gradient before averaging; the square root is applied to the resulting average in the update. If recent squared gradients are smaller than the old average, s can decrease. Thus older large gradients do not remain in the denominator with their original weight forever. ![Retain part of the old squared-gradient average and add the weighted square of the current gradient.](https://deep-learning-study-room.vercel.app/assets/figures/v_rmsprop.svg) Diagram: Retain part of the old squared-gradient average and add the weighted square of the current gradient.. Labels: RMSProp discounts older squared gradients | RMSProp discounts older squared gradients | Use the worked question: initial s = 0; gradients 2, then 1; decay 0.9. | Step 1 memory | 0.9 × 0 + 0.1 × 4 = | 0.4 | Step 2 memory | 0.9 × 0.4 + 0.1 × 1 = | 0.46 | Scale each step | Divide g by (√s + ε) | Keep 90% of the old memory, then add 10% of the current squared gradient. Retain part of the old squared-gradient average and add the weighted square of the current gradient. Tap the diagram to enlarge. ### Complete numerical: two iterations **Question.** θ₀ = 1, s₀ = 0, η = 0.1, β = 0.9; supplied gradients are 2 then 1. Perform two RMSProp updates, with ε = 0 for this hand calculation. ### Given θ₀ = 1; initial state(s) = 0; η = 0.1. Supplied gradients g₁ = 2, g₂ = 1. ε = 0 for this hand calculation. β = 0.9. ### To find Calculate all required states and the parameter after each of two updates. ### Formula — which one and why **RMSProp.** Use a decaying average of squared gradients, then scale the current gradient by its square root. Old contributions gradually lose influence. s = squared-gradient average; ρ (rho), sometimes written β, = retained fraction; g = current gradient; η = learning rate; ε = denominator stabilizer. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ $$ \theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{s_t}+\epsilon} $$ **Yaad rakhne ka cue:** Saare purane squares nahi—unki fading average. ### Solution — step by step **Iteration 1 — step 1: calculate the squared-gradient average.** $$ s_1=0.9(0)+0.1(2^2)=0.4 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_1}=\sqrt{0.4}\approx0.632455532 $$ $$ \frac{\eta g_1}{\sqrt{s_1}}=\frac{0.1(2)}{\sqrt{0.4}}\approx0.316227766 $$ **Step 3: update the parameter.** $$ \theta_1=1-0.316227766\approx0.683772234 $$ **Iteration 2 — step 1: use s₁, not the original zero state.** $$ s_2=0.9(0.4)+0.1(1^2)=0.46 $$ **Step 2: calculate the denominator and signed correction.** $$ \sqrt{s_2}=\sqrt{0.46}\approx0.678232998 $$ $$ \frac{\eta g_2}{\sqrt{s_2}}=\frac{0.1(1)}{\sqrt{0.46}}\approx0.147441956 $$ **Step 3: update from θ₁, retaining calculator precision.** $$ \theta_2\approx0.683772234-0.147441956\approx0.536330278 $$ **Final answer:** θ₁ ≈ 0.683772 and θ₂ ≈ 0.536330; s₁ = 0.4 and s₂ = 0.46. ### Comparison with AdaGrad and Adam AdaGrad stores an unweighted cumulative sum; RMSProp stores a decaying average. Basic RMSProp has no separate first-moment average and no Adam-style bias correction. With zero initialization, the early RMSProp state can therefore be much smaller than the first squared gradient, which explains the relatively large first correction in this example. The method still requires appropriate learning-rate and decay choices; adaptive scaling does not guarantee convergence to a global optimum. AdaGrad’s nondecreasing squared-gradient sum can shrink its effective step excessively. RMSProp’s exponential average discounts older squared gradients, so it avoids that permanent accumulation. Both adapt rates coordinatewise and use ε for numerical stability. Switching is a plausible remedy, not a guarantee: inspect learning rate, gradient scale, data and convergence before attributing every plateau to the optimizer. --- ## TEE · R25 Q7a · Long-text sentiment architecture Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s14/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s14 ## Question **Question. Choose a deep architecture for long social-media/customer-review sequences classified positive,negative or neutral. Explain its operation with equations.** (10 marks) ## Answer — step by step An LSTM is a suitable syllabus-aligned choice for long dependencies. Tokenize and embed the text, process the sequence through LSTM states, form a sequence representation and use a three-class softmax. Padding/masking must prevent padded tokens from being treated as content. Text tokens → Embedding sequence → LSTM hidden/cell recurrence → Final or pooled representation → Dense3 +softmax → Positive / negative / neutral **Definition.** Long short-term memory (LSTM) is a gated recurrent architecture with a cell state and hidden state. Forget, input and output gates regulate state retention, incorporation of candidate information and exposure of the updated cell. ### Explanation and working Long short-term memory, or LSTM, keeps a cell state Cₜ and a hidden state hₜ. A sigmoid gate produces values from 0 to 1: approximately 0 blocks a component, approximately 1 passes it. Intermediate values pass a fraction. Gates are learned from the previous hidden state and current input, not manually assigned by the user during normal inference. ![Follow the cell state from left to right then identify forget input and output gates](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: Follow the cell state from left to right then identify forget input and output gates. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. Follow the cell state from left to right then identify forget input and output gates Tap the diagram to enlarge. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c) $$ $$ C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\qquad h_t=o_t\odot\tanh(C_t) $$ The brackets [h, x] concatenate two vectors: [a, b] joined with [c, d] gives [a, b, c, d]. Concatenation is not addition. f is the forget gate: how much old cell content remains. i is the input gate: how much new candidate enters. The candidate is proposed content. o is the output gate: how much of the activated cell becomes visible as the hidden state. $$ p=\operatorname{softmax}(Wh+b),\qquad L=-\sum_{k=1}^3y_k\log p_k $$ Train on labelled examples with cross-entropy and BPTT; evaluate on held-out reviews with class-wise precision/recall/F1. LSTM gates reduce long-path gradient decay but do not guarantee perfect memory. The three outputs are mutually exclusive classes for this question. --- ## TEE · R25 Q7b · Transfer learning Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s15/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s15 ## Question **Question. Explain transfer learning and its benefits with suitable examples.** (5 marks) ## Answer — step by step **Definition.** Transfer learning reuses representations or parameters learned on a source task to support a target task. With a pretrained neural network, common strategies are fixed feature extraction and fine-tuning selected pretrained layers. ### Explanation and working Transfer learning reuses a model or representation learned on one task for a related new task. A pretrained CNN may already detect useful edges, textures and shapes, saving data and training effort. ![Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Start with learned image features. Replace the task-specific head, then choose which pretrained layers may change. Tap the diagram to enlarge. **Full scenario.** Build a face-attendance model with few images per student. Explain a training plan. **Step 1. Define and split the data.** Collect varied, permitted images and separate training/validation/test by suitable independent captures. Keep duplicates out of different splits. Apply the pretrained model’s required input size and normalization. **Step 2. Choose a suitable pretrained backbone.** For a course example, use VGG 16 or ResNet as an image-feature extractor. ImageNet-pretrained VGG 16 was trained on ImageNet, not automatically on a face-specific dataset. A face-trained model can be a more task-specific choice if available. **Step 3. Replace the classifier.** Remove the original class head. Add a new classifier matching the required student classes, or an appropriate embedding-based recognition system if that is the chosen formulation. For the exam classification example, use a softmax head and cross-entropy. **Step 4. Freeze and train.** Freeze the base weights and train the new head first. Use suitable augmentation, regularization and validation monitoring. **Step 5. Fine-tune if justified.** Unfreeze selected later layers and train at a smaller learning rate. Early layers may remain frozen. Compare validation performance and restore the best checkpoint. **Step 6. Evaluate.** Use the held-out test set once for the final estimate. Include unfamiliar faces and a rejection rule if describing a real attendance system; a closed-set classroom example alone does not solve unknown-person recognition. Feature extraction keeps the backbone fixed. Fine-tuning changes some pretrained weights. Training the entire model from scratch is a third option when enough representative data and compute are available, but it is usually harder for the stated small-dataset scenario. Early image layers commonly learn reusable edges and textures. Later layers become more specialized to source-task categories. “Earlier” means closer to the input, regardless of whether a drawing places those layers at the top or bottom. | Target dataset | Sensible starting choice | Reason | | --- | --- | --- | | Small and similar | Freeze the backbone; train a new head first | Useful features already exist, while few examples make overfitting likely | | Large and similar | Train a new head, then fine-tune more layers if validation improves | More data can support more trainable parameters | | Small and different | Try a more relevant pretrained model or fixed earlier/intermediate features; tune cautiously | Later source features may be unsuitable, but small data still makes extensive tuning risky | | Large and different | Compare broad fine-tuning with training from scratch | Sufficient data permits stronger adaptation; the value of source features must be tested | These are starting strategies, not universal rules. Fine-tuning an early layer means updating its pretrained weights, not automatically learning it “from scratch”. With very little data, simply unfreezing early layers because the task differs can overfit. Choose changes using validation performance and apply the required preprocessing consistently. **Example.** With 100 labelled microscopic images, an ImageNet model may provide reusable low-level patterns, but its animal/object classifier is inappropriate. Replace the head, test fixed representations and monitor validation before increasing the number of trainable layers. With a large representative image dataset, broader fine-tuning becomes a more reasonable candidate. **Yaad rakho:** data kam toh kam weights hilao; task alag toh features ki suitability pehle dekho. --- ## TEE · R25 Q7c · Learning rate and two losses Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c21-s16/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/21/tee-c21-s16 ## Question **Question. Explain the importance of learning rate. Give and explain one classification loss and one regression loss.** (5 marks) ## Answer — step by step **Definition.** The learning rate is a hyperparameter controlling the scale of a gradient-based update. It affects convergence speed and stability; it is not the gradient and is not the magnitude of the loss. ### Explanation and working A very small rate can make progress slow. A very large rate can overshoot, oscillate or diverge. A suitable rate permits progress without uncontrolled jumps. Feature scaling can reduce long zig-zag paths. A noisy curve alone is not proof of failure; individual mini-batch losses fluctuate. ![Illustrative loss traces show why smoothness and progress are different](https://deep-learning-study-room.vercel.app/assets/figures/optimizers.svg) Diagram: Illustrative loss traces show why smoothness and progress are different. Labels: Loss | Updates | Full batch trend | SGD fluctuations | Illustration only: smoothness alone does not prove better learning. Illustrative loss traces show why smoothness and progress are different Tap the diagram to enlarge. | Observation | Possible reason | Sensible first action | | --- | --- | --- | | Loss barely changes | Very small rate, saturated/dead units, wrong implementation | Check gradients and data; then tune rate or activation | | Loss jumps wildly | Rate too high or exploding gradients | Reduce rate; inspect gradient size; clip if appropriate | | Noisy but decreasing loss | Small-batch sampling noise | Compare a smoothed trend; consider larger batches or momentum | | Training improves but validation worsens | Overfitting | Use regularization, augmentation or early stopping | | All identical hidden neurons learn alike | Symmetric initialization | Use suitable nonidentical initial weights | Convergence means updates or the objective are settling under a criterion. A zero gradient can also occur at a saddle point or in a saturated region; it is not proof of a global optimum. $$ L_{class}=-\sum_ky_k\log p_k,\qquad L_{reg}=\frac1n\sum_i(\hat y_i-y_i)^2 $$ Categorical cross-entropy penalizes low assigned probability to the true class; for a one-hot target and true-class probability0.8, L=−ln0.8≈0.223144. MSE measures average squared numeric prediction error; errors1and−2give(1²+(-2)²)/2=2.5. These are illustrative examples; their output interpretation and loss scaling must match the model. --- ## TEE · Faculty Assignment2 Q1 · One momentum update with an omitted coefficient Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c22-s0/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/22/tee-c22-s0 ## Question **Question. For J(w)=½(w−3)²,w₀=5,learning rate0.1,perform one SGD-with-momentum iteration.** (Faculty assignment: no marks printed) ## Answer — step by step ### Given w₀=5,η=0.1. βandinitialvelocityare not supplied. ### To find One update conditional on zero initial velocity. ### Formula: which one and why $$ g=w-3,\quad v_1=\beta v_0+\eta g,\quad w_1=w_0-v_1 $$ Choose displacement convention and v₀=0; then the first step is independent of β. ### Step 1. Differentiate and evaluate g₀=5−3=2. ### Step 2. Calculate momentum displacement v₁=β×0+0.1×2=0.2. ### Step 3. Subtract the displacement w₁=5−0.2=4.8. ### Final answer **w₁=4.8 if v₀=0.** For a nonzero initial velocity, w₁=4.8−βv₀; no unique numerical value follows without those givens. **Source / convention note.** Authentic faculty Assignment2; no marks printed. β/v₀ omission explicitly identified. --- ## TEE · Faculty Assignment2 Q2 · Specified momentum update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c22-s1/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/22/tee-c22-s1 ## Question **Question. For J(w)=½(w+2)²,η=0.2,β=0.8,w₀=4,v₀=0,perform one momentum step.** (Faculty assignment: no marks printed) ## Answer — step by step ### Given All givens above. ### To find v₁andw₁. ### Formula: which one and why $$ g=w+2,\quad v_1=\beta v_0+\eta g,\quad w_1=w_0-v_1 $$ ### Step 1. Gradient g₀=4+2=6. ### Step 2. Momentum displacement v₁=0.8×0+0.2×6=1.2. ### Step 3. Weight w₁=4−1.2=2.8. ### Final answer **v₁=1.2,w₁=2.8.** A signed-velocity convention givesv₁=−1.2andw₁=w₀+v₁; the final weight is the same. --- ## TEE · Faculty Assignment2 Q3 · 28×28 CNN Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c22-s2/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/22/tee-c22-s2 ## Question **Question. Input28×28×1; conv5,6valid; pool2stride2; conv5,16valid; pool2stride2; dense120,dense84,dense10. Find layer dimensions and parameters.** (Faculty assignment: no marks printed) ## Answer — step by step ### Given Input=[28, 28, 1] | Layer | Kernel | Stride | Padding | Filters / units | | --- | --- | --- | --- | --- | | Conv | 5 | 1 | 0 | 6 | | Pool | 2 | 2 | 0 | 0 | | Conv | 5 | 1 | 0 | 16 | | Pool | 2 | 2 | 0 | 0 | | Dense | 0 | 0 | 0 | 120 | | Dense | 0 | 0 | 0 | 84 | | Dense | 0 | 0 | 0 | 10 | ### To find Each output shape and all trainable weights plus biases. ### Formula: which one and why $$ O=\left\lfloor\frac{N+2P-F}{S}\right\rfloor+1 $$ $$ P_{conv}=(F_hF_wC_{in}+1)C_{out},\quad P_{dense}=(n_{in}+1)n_{out} $$ Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit. ### Step 1. Work layer by layer | Layer | Input | Full substitution and reason | Output | Parameters | | --- | --- | --- | --- | --- | | Conv | 28×28×1 | ⌊(28+2×0−5)/1⌋+1=24; parameters=(5×5×1+1)×6=156 | 24×24×6 | 156 | | Pool | 24×24×6 | ⌊(24−2)/2⌋+1=12; channels stay 6; no trainable weights | 12×12×6 | 0 | | Conv | 12×12×6 | ⌊(12+2×0−5)/1⌋+1=8; parameters=(5×5×6+1)×16=2416 | 8×8×16 | 2416 | | Pool | 8×8×16 | ⌊(8−2)/2⌋+1=4; channels stay 16; no trainable weights | 4×4×16 | 0 | | Dense | 4×4×16 | Flatten/input length=4×4×16=256; parameters=(256+1)×120=30840 | 120 | 30840 | | Dense | 1×1×120 | Flatten/input length=1×1×120=120; parameters=(120+1)×84=10164 | 84 | 10164 | | Dense | 1×1×84 | Flatten/input length=1×1×84=84; parameters=(84+1)×10=850 | 10 | 850 | ### Step 2. Sum only trainable parameters 156 + 2416 + 30840 + 10164 + 850 = 44426 ### Final answer **Total parameters=44426.** Shape and parameter count are separate results. --- ## TEE · Faculty Assignment2 Q4 · Batch linear regression update Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c22-s3/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/22/tee-c22-s3 ## Question **Question. x=[1,2],y=[3,5],w=b=0,η=0.1;J=(1/(2m))Σ(wxᵢ+b−yᵢ)²,m=2. Compute both gradients and update.** (Faculty assignment: no marks printed) ## Answer — step by step ### Given Two examples, shared w,b, m=2. ### To find One simultaneous batch update. ### Formula: which one and why $$ \frac{\partial J}{\partial w}=\frac1m\sum_ie_ix_i,\quad\frac{\partial J}{\partial b}=\frac1m\sum_ie_i,\quad e_i=wx_i+b-y_i $$ ### Step 1. Predict and subtract targets | x | y | wx+b | e | e×x | | --- | --- | --- | --- | --- | | 1 | 3 | 0 | -3 | -3 | | 2 | 5 | 0 | -5 | -10 | ### Step 2. Average each gradient sum dw=(−3−10)/2=−6.5;db=(−3−5)/2=−4. ### Step 3. Update both from old parameters w′=0−0.1×(−6.5)=0.65;b′=0−0.1×(−4)=0.4. ### Final answer **w′=0.65,b′=0.4.** New predictions1.05and1.7; do not updatew before computing theoldbgradient. --- ## TEE · Faculty Assignment2 Q5 · 32×32 three-convolution CNN Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c22-s4/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/22/tee-c22-s4 ## Question **Question. Input32×32×1; conv3,8same; pool2stride2; conv5,16valid; pool2stride2; conv3,32same; dense64; output5. Find shapes and parameters.** (Faculty assignment: no marks printed) ## Answer — step by step ### Given Input=[32, 32, 1] | Layer | Kernel | Stride | Padding | Filters / units | | --- | --- | --- | --- | --- | | Conv | 3 | 1 | 1 | 8 | | Pool | 2 | 2 | 0 | 0 | | Conv | 5 | 1 | 0 | 16 | | Pool | 2 | 2 | 0 | 0 | | Conv | 3 | 1 | 1 | 32 | | Dense | 0 | 0 | 0 | 64 | | Dense | 0 | 0 | 0 | 5 | ### To find Each output shape and all trainable weights plus biases. ### Formula: which one and why $$ O=\left\lfloor\frac{N+2P-F}{S}\right\rfloor+1 $$ $$ P_{conv}=(F_hF_wC_{in}+1)C_{out},\quad P_{dense}=(n_{in}+1)n_{out} $$ Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit. ### Step 1. Work layer by layer | Layer | Input | Full substitution and reason | Output | Parameters | | --- | --- | --- | --- | --- | | Conv | 32×32×1 | ⌊(32+2×1−3)/1⌋+1=32; parameters=(3×3×1+1)×8=80 | 32×32×8 | 80 | | Pool | 32×32×8 | ⌊(32−2)/2⌋+1=16; channels stay 8; no trainable weights | 16×16×8 | 0 | | Conv | 16×16×8 | ⌊(16+2×0−5)/1⌋+1=12; parameters=(5×5×8+1)×16=3216 | 12×12×16 | 3216 | | Pool | 12×12×16 | ⌊(12−2)/2⌋+1=6; channels stay 16; no trainable weights | 6×6×16 | 0 | | Conv | 6×6×16 | ⌊(6+2×1−3)/1⌋+1=6; parameters=(3×3×16+1)×32=4640 | 6×6×32 | 4640 | | Dense | 6×6×32 | Flatten/input length=6×6×32=1152; parameters=(1152+1)×64=73792 | 64 | 73792 | | Dense | 1×1×64 | Flatten/input length=1×1×64=64; parameters=(64+1)×5=325 | 5 | 325 | ### Step 2. Sum only trainable parameters 80 + 3216 + 4640 + 73792 + 325 = 82053 ### Final answer **Total parameters=82053.** Shape and parameter count are separate results. --- ## TEE · Faculty Assignment2 Q6 · 64×64 two-convolution CNN Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/section/tee-c22-s5/ Reader: https://deep-learning-study-room.vercel.app/#tee/read/22/tee-c22-s5 ## Question **Question. Input64×64×3; conv5,16valid; pool2stride2; conv3,32same; pool2stride2; dense128; output10. Find shapes and parameters.** (Faculty assignment: no marks printed) ## Answer — step by step ### Given Input=[64, 64, 3] | Layer | Kernel | Stride | Padding | Filters / units | | --- | --- | --- | --- | --- | | Conv | 5 | 1 | 0 | 16 | | Pool | 2 | 2 | 0 | 0 | | Conv | 3 | 1 | 1 | 32 | | Pool | 2 | 2 | 0 | 0 | | Dense | 0 | 0 | 0 | 128 | | Dense | 0 | 0 | 0 | 10 | ### To find Each output shape and all trainable weights plus biases. ### Formula: which one and why $$ O=\left\lfloor\frac{N+2P-F}{S}\right\rfloor+1 $$ $$ P_{conv}=(F_hF_wC_{in}+1)C_{out},\quad P_{dense}=(n_{in}+1)n_{out} $$ Each filter/unit has one bias. Pooling has zero learned parameters. Flatten preserves the number of values, then a dense layer connects every input value to every output unit. ### Step 1. Work layer by layer | Layer | Input | Full substitution and reason | Output | Parameters | | --- | --- | --- | --- | --- | | Conv | 64×64×3 | ⌊(64+2×0−5)/1⌋+1=60; parameters=(5×5×3+1)×16=1216 | 60×60×16 | 1216 | | Pool | 60×60×16 | ⌊(60−2)/2⌋+1=30; channels stay 16; no trainable weights | 30×30×16 | 0 | | Conv | 30×30×16 | ⌊(30+2×1−3)/1⌋+1=30; parameters=(3×3×16+1)×32=4640 | 30×30×32 | 4640 | | Pool | 30×30×32 | ⌊(30−2)/2⌋+1=15; channels stay 32; no trainable weights | 15×15×32 | 0 | | Dense | 15×15×32 | Flatten/input length=15×15×32=7200; parameters=(7200+1)×128=921728 | 128 | 921728 | | Dense | 1×1×128 | Flatten/input length=1×1×128=128; parameters=(128+1)×10=1290 | 10 | 1290 | ### Step 2. Sum only trainable parameters 1216 + 4640 + 921728 + 1290 = 928874 ### Final answer **Total parameters=928874.** Shape and parameter count are separate results. --- ## TEE · MT2 Mock A Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/mocks/a/ Reader: https://deep-learning-study-room.vercel.app/#tee/mocks/A 60 minutes · 10 marks. Q1(a,b): 2 marks each, compulsory; attempt any two of Q2–Q4: 3 marks each. This uses the shared M2 choice pattern already supplied. The latest faculty message guides the Unit 3–4 theory/numerical mix; it does not fix individual question numbers or numerical topics. Q1 theory · two Unit 4 numericals + Unit 3 theory among the 3-mark choices ## Q1a ## Question **Q1a compulsory (2 marks). Explain the key difference between the AdaGrad and RMSProp accumulators. Why can AdaGrad's effective learning rate become very small during prolonged training?** ## Answer — step by step ### 1. AdaGrad stores a running sum Let gₜ be the current gradient and Gₜ the stored sum of squared gradients. AdaGrad updates that sum as follows: $$ G_t=G_{t-1}+g_t^2 $$ Every added square is nonnegative, so G cannot decrease. All past squared gradients remain in the sum. ### 2. RMSProp stores a decaying average Let sₜ be the stored squared-gradient average and ρ the fraction of the previous average retained. $$ s_t=\rho s_{t-1}+(1-\rho)g_t^2 $$ RMSProp gradually reduces the influence of older squared gradients. Its accumulator can rise or fall, depending on the new squared gradient. ### 3. Explain the effect on step size AdaGrad's effective learning rate is: $$ \eta_{effective}=\frac{\eta}{\sqrt{G_t}+\epsilon} $$ Here η is the base learning rate and ε is a small positive number that stabilizes division. When G becomes large, its square root becomes large, so dividing by it produces a smaller effective learning rate. RMSProp's decaying average avoids this particular permanent-accumulation problem. **Final answer:** AdaGrad uses a nondecreasing sum of all squared gradients; RMSProp uses a decaying average. AdaGrad's growing denominator can make future updates very small. **Yaad rakho:** AdaGrad sab squares jodta rehta hai; RMSProp purane squares ka asar dheere kam karta hai. ## Q1b ## Question **Q1b compulsory (2 marks). Explain local connectivity and parameter sharing in a CNN. How do they reduce the number of learned parameters compared with a fully connected mapping of an image?** ## Answer — step by step ### 1. Local connectivity Each convolutional output depends on a small spatial neighbourhood of the input, called its receptive field, rather than every pixel in the entire image. The filter spans the input channels within that neighbourhood. This limits the connections used to calculate each output. ### 2. Parameter sharing The same filter weights and bias are reused at every spatial position where that filter is applied. A CNN therefore learns one set of parameters for that filter, rather than a separate set for each image location. ### 3. Concrete image example An edge-detecting filter can identify a vertical boundary near the left of an image and then slide to look for a similar boundary near the right. Both locations use the same learned filter. The network does not need to learn an independent edge detector for every position. ### 4. Comparison with a fully connected mapping A fully connected layer generally learns a separate weight for every input-to-output connection. Convolution reduces this requirement in two ways: local connectivity restricts each output's input region, while parameter sharing reuses one filter across locations. More output positions create more activations and computation, but do not create new parameters for that shared filter. **Final answer:** local connectivity uses a limited input neighbourhood; parameter sharing reuses the filter's learned values across spatial positions. Together they can require far fewer parameters than a fully connected image mapping. **Yaad rakho:** Filter ke numbers ek baar seekhte hain; image par baar-baar wahi use hote hain. ## Q2 ## Question **Q2 optional (3 marks). An input of size 7 × 7 × 3 passes through a convolutional layer with two 3 × 3 filters, stride 2 and padding 1 on each side; then a 2 × 2 pooling layer with stride 2 and no padding; then flattening; then a dense layer with three output units. Find the output shape after every layer and the total trainable parameters. Include one bias per convolutional filter and per dense unit.** ## Answer — step by step ### Given - Input: height 7, width 7, channels 3. - Convolution: kernel 3 × 3, filters 2, stride 2, padding 1 per side. - Pooling: window 2 × 2, stride 2, padding 0. - Flattening followed by 3 dense output units. - Include all specified biases. Pooling and flattening have no learned parameters. ### To find The convolution output, pooling output, flatten length, dense output length and total trainable parameter count. ### Formula — which one and why For either spatial axis, use the output-size formula. N is that axis's input size, K the window size, S its stride and P padding per side. The floor means discard any fractional part because a full window must fit. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1 $$ Each convolution filter spans all input channels; each dense unit connects to every flattened input. Therefore: $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}},\qquad P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ Flattening simply counts all values in the preceding feature maps: $$ n_{\mathrm{flat}}=HWC $$ ### Solution — step by step ### Step 1: convolution height and width $$ N_{\mathrm{conv}}=\left\lfloor\frac{7+2(1)-3}{2}\right\rfloor+1=3+1=4 $$ The numerator is 7 + 2 − 3 = 6. Divide by stride 2 to get 3, then add 1 to get 4 positions. The same calculation applies to height and width. Two filters create two output channels. **Convolution output: 4 × 4 × 2.** ### Step 2: convolution parameters $$ \text{weights}=3\times3\times3\times2=54 $$ $$ \text{biases}=2,\qquad P_{\mathrm{conv}}=54+2=56 $$ ### Step 3: pooling shape and parameters The pooling input size is now 4, not the original 7. $$ N_{\mathrm{pool}}=\left\lfloor\frac{4-2}{2}\right\rfloor+1=1+1=2 $$ Pooling operates separately on each channel, so the two channels remain two. **Pooling output: 2 × 2 × 2. Pooling parameters: 0.** ### Step 4: flattening $$ n_{\mathrm{flat}}=2\times2\times2=8 $$ There are eight values to send to the dense layer. Flattening rearranges them into a list and learns no parameters. **Flatten output: 8 values. Flatten parameters: 0.** ### Step 5: dense output and parameters Each of the three dense units receives eight inputs and has one bias. $$ \text{weights}=8\times3=24,\quad\text{biases}=3,\quad P_{\mathrm{dense}}=27 $$ **Dense output: 3 values.** ### Step 6: total trainable parameters $$ P_{\mathrm{total}}=56+0+0+27=83 $$ **Final answer:** 4 × 4 × 2 → 2 × 2 × 2 → 8 → 3. **Total: 83 trainable parameters.** **Yaad rakho:** Pehle shape; phir weights. Ek layer ka output hi next layer ka input hai. ## Q3 ## Question **Q3 optional (3 marks). Apply the printed kernel K to input I using the deep-learning cross-correlation convention, with stride 1, no padding and zero bias. Do not flip K. Apply ReLU to the resulting map, followed by 2 × 2 max pooling with stride 2 and no padding. Show every convolution entry and find the final output.** $$ I=\begin{bmatrix}1&2&1\\0&1&3\\2&1&0\end{bmatrix},\quad K=\begin{bmatrix}1&-1\\0&1\end{bmatrix} $$ ## Answer — step by step ### Given - Input I is the printed 3 × 3 matrix; kernel K is the printed 2 × 2 matrix. - Convolution stride 1, padding 0, bias 0; use K in its printed orientation. - Activation: ReLU. - Pooling: max, 2 × 2 window, stride 2, padding 0. ### To find All four convolution values, the ReLU output matrix and the final pooled output. ### Formula — which one and why For each input window, multiply entries at matching positions, then add all products and the bias once. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ X is the input, S is stride, u and v select a kernel entry, and b is bias. This formula uses zero-based array indices; the written solution labels positions from 1 for readability. ReLU is required because the question requests activation after convolution. It replaces a negative value by zero. $$ \operatorname{ReLU}(z)=\max(0,z) $$ Max pooling selects the largest value in its current window V. $$ y_{\max}=\max V $$ ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. ### Step 1: determine the convolution shape For each axis: (3 − 2)/1 + 1 = 2. Therefore calculate a 2 × 2 output matrix. ### Step 2: top-left output, position (1,1) The window is $\begin{bmatrix}1 & 2\\0 & 1\end{bmatrix}$ . Multiply matching cells: 1 × 1 = 1; 2 × (−1) = −2; 0 × 0 = 0; 1 × 1 = 1. Add the products and bias: 1 − 2 + 0 + 1 + 0 = **0**. ### Step 3: top-right output, position (1,2) Move the window one column to the right. The window is $\begin{bmatrix}2 & 1\\1 & 3\end{bmatrix}$ . Multiply matching cells: 2 × 1 = 2; 1 × (−1) = −1; 1 × 0 = 0; 3 × 1 = 3. Add: 2 − 1 + 0 + 3 + 0 = **4**. ### Step 4: bottom-left output, position (2,1) The window is $\begin{bmatrix}0 & 1\\2 & 1\end{bmatrix}$ . Multiply matching cells: 0 × 1 = 0; 1 × (−1) = −1; 2 × 0 = 0; 1 × 1 = 1. Add: 0 − 1 + 0 + 1 + 0 = **0**. ### Step 5: bottom-right output, position (2,2) The window is $\begin{bmatrix}1 & 3\\1 & 0\end{bmatrix}$ . Multiply matching cells: 1 × 1 = 1; 3 × (−1) = −3; 1 × 0 = 0; 0 × 1 = 0. Add: 1 − 3 + 0 + 0 + 0 = **−2**. **Convolution map: $\begin{bmatrix}0 & 4\\0 & -2\end{bmatrix}$ .** ### Step 6: apply ReLU separately to every entry - max(0,0) = 0. - max(0,4) = 4. - max(0,0) = 0. - max(0,−2) = 0. **ReLU map: $\begin{bmatrix}0 & 4\\0 & 0\end{bmatrix}$ .** ### Step 7: max pooling The 2 × 2 pooling window covers all four activated entries. Compare 0, 4, 0 and 0. Their maximum is **4**. **Final answer:** convolution $\begin{bmatrix}0 & 4\\0 & -2\end{bmatrix}$ → ReLU $\begin{bmatrix}0 & 4\\0 & 0\end{bmatrix}$ → pooled output ** $\begin{bmatrix}4\end{bmatrix}$ **, a 1 × 1 map. **Yaad rakho:** Multiply-add se convolution; negative hatao ReLU; biggest uthao max pool. ## Q4 ## Question **Q4 optional (3 marks). What is overfitting? Explain how dropout and early stopping help improve generalization. State which dataset should select the stopping point.** ## Answer — step by step ### 1. Definition and identification Overfitting occurs when a model learns training-specific details or noise that do not generalize to new data. It commonly shows low training error together with substantially higher validation error. Generalization means performing well on new examples drawn from the task's data distribution. ### 2. Dropout During training, dropout randomly sets selected activations to zero. The active subnetwork changes across training passes, discouraging reliance on a fixed combination of hidden units. Under inverted dropout, retained activations are divided by the keep probability to preserve their expected scale. During ordinary inference, the masking is disabled and all units are used without that training-time rescaling. Dropout is a regularizer; it can reduce overfitting, but excessive dropout can cause underfitting. ### 3. Early stopping Train on the training set and monitor performance on a separate **validation set**. Save the model whenever validation loss improves. Stop after the chosen number of consecutive non-improving evaluations, called patience, then restore the saved best model. For example, validation losses 0.8, 0.6, 0.5, 0.53 and 0.55 have their minimum at epoch 3. With patience 2, stop after epoch 5 and restore epoch 3. The stopping epoch and saved-model epoch are different. ![Early stopping retains the model with the best validation performance, not necessarily the final trained model.](https://deep-learning-study-room.vercel.app/assets/figures/v_early_stop.svg) Diagram: Early stopping retains the model with the best validation performance, not necessarily the final trained model.. Labels: Keep the model from the best validation point | Keep the model from the best validation point | Illustrative curves: lower loss is better. | Epoch | Loss | Validation | Training | Save this best checkpoint | Patience may wait for later epochs, but restore the model with the best | validation score. Early stopping retains the model with the best validation performance, not necessarily the final trained model. Tap the diagram to enlarge. **Final answer:** dropout restricts dependence on particular training-time activations; early stopping limits training after validation performance stops improving. Choose the stopping point using validation data and reserve the test set for final evaluation. **Yaad rakho:** Training se seekho, validation se best model chuno, test se final result batao. --- ## TEE · MT2 Mock B Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/mocks/b/ Reader: https://deep-learning-study-room.vercel.app/#tee/mocks/B 60 minutes · 10 marks. Q1(a,b): 2 marks each, compulsory; attempt any two of Q2–Q4: 3 marks each. This uses the shared M2 choice pattern already supplied. The latest faculty message guides the Unit 3–4 theory/numerical mix; it does not fix individual question numbers or numerical topics. Q1 theory/numerical combination · Unit 3 theory + numerical mixed question, Unit 4 numerical, Unit 4 theory ## Q1a ## Question **Q1a compulsory (2 marks). Explain the purpose of scaling retained activations in inverted dropout. For activations [1,2,3,4], keep probability 0.5 and mask [1,0,1,0], calculate the activations used during this training pass. A mask value of 1 means keep and 0 means drop.** ## Answer — step by step ### Theory: why scale retained values? Randomly dropping units would reduce the average activation. Inverted dropout divides every retained activation by the keep probability. Averaged over the random masks, this preserves the original activation's expected value. A dropped activation is still zero. ### Given - Original activations a = [1,2,3,4]. - Keep probability q = 0.5, meaning a unit has a 50% chance of being kept. - This pass's supplied mask m = [1,0,1,0]. ### To find The four post-dropout activations used in this training pass. ### Formula — which one and why Multiply the original activation aᵢ by its mask mᵢ, then divide by keep probability q. $$ a_i^{\mathrm{train}}=\frac{m_ia_i}{q} $$ ### Solution — step by step ### Step 1: first activation Mask 1 keeps the value. Multiply 1 × 1 = 1; divide 1 by 0.5 to get **2**. ### Step 2: second activation Mask 0 drops the value. Multiply 0 × 2 = 0; divide 0 by 0.5 to get **0**. ### Step 3: third activation Mask 1 keeps the value. Multiply 1 × 3 = 3; divide 3 by 0.5 to get **6**. ### Step 4: fourth activation Mask 0 drops the value. Multiply 0 × 4 = 0; divide 0 by 0.5 to get **0**. **Final answer: [2,0,6,0].** This is the training-pass result. In ordinary inference with inverted dropout, disable the mask and its scaling. **Yaad rakho:** Mask lagao; bache hue values ko keep probability se divide karo. ## Q1b ## Question **Q1b compulsory (2 marks). Explain a ResNet shortcut. Why must the two tensors being added have matching shapes, and what can be done if their shapes differ?** ## Answer — step by step ### 1. Residual addition A residual block combines a learned transformation F(x) with a shortcut carrying the input x: $$ H(x)=F(x)+x $$ F(x) is the output of the learned residual branch. Adding x provides a direct path for information and gradients, helping optimize deep networks. If the branch produces zero, the addition stage returns x. A post-addition activation, if specified by the architecture, is applied afterward. ### 2. Why matching shapes are needed The addition is entry by entry. Each element in F(x) needs a corresponding element in x, so height, width and channel count must match. For example, a 14 × 14 × 32 tensor cannot be directly added to a 14 × 14 × 64 tensor in a standard residual shortcut. ### 3. How a projection helps A learned projection, commonly a 1 × 1 convolution with an appropriate number of filters and stride, maps the shortcut to the required shape. Then add the projected shortcut and the residual branch. This is addition, not channel concatenation. **Final answer:** a shortcut adds input information to the learned residual branch. Use an identity shortcut when shapes match and a suitable projection when they do not. **Yaad rakho:** Same-size entries jodte hain; mismatch ho toh shortcut ka size pehle match karo. ## Q2 ## Question **Q2 optional (3 marks). (a) Explain why Adam uses bias correction for its initially zero moment estimates. (1 mark) (b) Adam starts with θ₀ = 1 and m₀ = v₀ = 0. Use learning rate η = 0.1, β₁ = 0.9 and β₂ = 0.99. The supplied gradients are g₁ = 2 and g₂ = −1. Perform two bias-corrected updates. Show the raw first and second moments, corrected moments and updated parameter after each iteration. Set ε = 0 only for this nonzero-denominator hand calculation. (2 marks)** ## Answer — step by step ### Part (a): why Adam corrects the moments Adam initializes both moving averages to zero. At early iterations, too little gradient history has accumulated, so the raw estimates are biased toward zero. Dividing mₜ by 1 − β₁ᵗ and vₜ by 1 − β₂ᵗ corrects this initialization bias before the parameter update. The power uses the current iteration number t. As t increases, these denominators approach 1. ### Part (b): numerical solution ### Given - Starting parameter θ₀ = 1. - Starting first moment m₀ = 0 and second moment v₀ = 0. - Learning rate η = 0.1. - Decay factors β₁ = 0.9 and β₂ = 0.99. - Supplied gradients, in order: 2, then −1. - ε = 0 for this exercise; both denominators will be nonzero. ### To find For iterations 1 and 2, calculate m, v, their bias-corrected values and the new parameter θ. ### Formula — which one and why Adam keeps two moving averages: m stores signed gradients and v stores squared gradients. Zero initialization makes these averages initially too small, so divide by the correction factors for the current iteration t. $$ m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,\quad v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2 $$ $$ \hat m_t=\frac{m_t}{1-\beta_1^t},\quad\hat v_t=\frac{v_t}{1-\beta_2^t} $$ $$ \theta_t=\theta_{t-1}-\frac{\eta\hat m_t}{\sqrt{\hat v_t}+\varepsilon} $$ The hat marks a corrected value. Carry the **raw** m and v into the next moving-average recurrence, not the corrected values. Keep full calculator precision until the final answer. ### Solution — step by step ### Step 1: iteration 1, raw first moment The current gradient is 2. Retain 90% of the old first moment and add 10% of this gradient. $$ m_1=0.9(0)+0.1(2)=0.2 $$ ### Step 2: iteration 1, raw second moment Square the gradient first: 2² = 4. Retain 99% of the old second moment and add 1% of that square. $$ v_1=0.99(0)+0.01(2)^2=0.04 $$ ### Step 3: iteration 1, bias correction Use t = 1. The correction denominators are 1 − 0.9 = 0.1 and 1 − 0.99 = 0.01. $$ \widehat m_1=\frac{0.2}{1-0.9^{1}}=2 $$ $$ \widehat v_1=\frac{0.04}{1-0.99^{1}}=4 $$ ### Step 4: iteration 1, parameter update The square root of the corrected second moment is √4 = 2. The step is 0.1 × 2 ÷ 2 = 0.1. $$ \theta_1=1-\frac{0.1\times(2)}{\sqrt{4}}=0.9 $$ ### Step 5: iteration 2, raw first moment Use supplied gradient −1, previous raw m = 0.2 and previous raw v = 0.04. $$ m_2=0.9(0.2)+0.1(-1)=0.08 $$ The two contributions are 0.18 and −0.1; their sum is 0.08. ### Step 6: iteration 2, raw second moment The square of −1 is +1. $$ v_2=0.99(0.04)+0.01(-1)^2=0.0496 $$ The two contributions are 0.0396 and 0.01; their sum is 0.0496. ### Step 7: iteration 2, bias correction Use t = 2. The correction denominators are 1 − 0.9² = 0.19 and 1 − 0.99² = 0.0199. $$ \widehat m_2=\frac{0.08}{1-0.9^{2}}=0.421052632 $$ $$ \widehat v_2=\frac{0.0496}{1-0.99^{2}}=2.492462312 $$ ### Step 8: iteration 2, parameter update Start from the updated parameter 0.9. Take the square root of the corrected second moment, divide the corrected first moment by it, multiply by 0.1 and subtract. $$ \theta_2=0.9-\frac{0.1\times(0.421052632)}{\sqrt{2.492462312}}=0.873330057 $$ The displayed decimals are rounded; evaluate using the unrounded fractions when possible. **Final answer:** θ₁ = **0.9** and θ₂ ≈ **0.873330057**. The second supplied gradient is negative, but the first-moment estimate is still positive because it retains part of the previous positive gradient. **Yaad rakho:** Do raw memories banao → current t se hats nikalo → update; next round mein raw memories le jao. ## Q3 ## Question **Q3 optional (3 marks). A 16 × 16 × 64 feature map is processed by 32 filters of size 3 × 3 using stride 1 and same padding. Compare the direct convolution's MAC count with a bottleneck that first applies eight 1 × 1 filters, then the same 32-filter 3 × 3 convolution. Both stages keep spatial size 16 × 16. Find both costs, the absolute saving and percentage saving. Count one multiply–accumulate as one MAC and omit bias additions.** ## Answer — step by step ### Given - Input map: 16 × 16 with 64 channels. - Direct layer: 32 filters, each 3 × 3, receiving 64 channels. - Bottleneck stage 1: eight 1 × 1 filters, receiving 64 channels. - Bottleneck stage 2: 32 filters, each 3 × 3, now receiving only 8 channels. - Every layer has output height and width 16; ignore bias additions. ### To find Direct MACs, both bottleneck-stage MACs, their total, absolute saving and saving percentage. ### Formula — which one and why Each output entry uses K²Cᵢₙ weight–input pairs. There are HₒᵤₜWₒᵤₜCₒᵤₜ output entries. Multiply these counts: $$ \mathrm{MACs}=H_{\mathrm{out}}W_{\mathrm{out}}C_{\mathrm{out}}K^2C_{\mathrm{in}} $$ The bottleneck has two layers, so calculate each layer's work and add them. Saving is direct work minus reduced work, expressed as a fraction of direct work. $$ \mathrm{saving\ percentage}=100\frac{\mathrm{direct}-\mathrm{reduced}}{\mathrm{direct}} $$ ### Solution — step by step ### Step 1: direct convolution Each output entry uses 3 × 3 × 64 = **576** weight–input pairs. There are 16 × 16 × 32 = **8,192** output entries. $$ \mathrm{MACs}_{\mathrm{direct}}=8192\times576=4{,}718{,}592 $$ ### Step 2: 1 × 1 bottleneck reduction Each reduction output uses 1 × 1 × 64 = **64** weight–input pairs. There are 16 × 16 × 8 = **2,048** reduction outputs. $$ \mathrm{MACs}_{1\times1}=16\times16\times8\times1\times1\times64=131{,}072 $$ The new map is **16 × 16 × 8**. This changes the next layer's input channels to 8. ### Step 3: following 3 × 3 convolution Each output now uses 3 × 3 × 8 = **72** weight–input pairs, instead of 576. The number of outputs remains 16 × 16 × 32 = **8,192**. $$ \mathrm{MACs}_{3\times3}=16\times16\times32\times3\times3\times8=589{,}824 $$ ### Step 4: total bottleneck cost Include both stages; do not compare the direct layer against only the second stage. $$ \mathrm{MACs}_{\mathrm{bottleneck}}=131{,}072+589{,}824=720{,}896 $$ ### Step 5: absolute saving $$ \mathrm{saving}=4{,}718{,}592-720{,}896=3{,}997{,}696 $$ ### Step 6: percentage saving Divide the saving by the original direct cost, then multiply by 100. $$ \mathrm{saving\ percent}=\frac{3{,}997{,}696}{4{,}718{,}592}\times100=84.722222\% $$ **Final answer:** direct **4,718,592 MACs**; bottleneck **720,896 MACs**; saving **3,997,696 MACs**, approximately **84.72%**. Both routes end in a 16 × 16 × 32 output, though they are different learned transformations. **Yaad rakho:** 1 × 1 se channels kam; expensive 3 × 3 ka kaam kam. Dono stages ka cost jodna hai. ## Q4 ## Question **Q4 optional (3 marks). Explain transfer learning for image classification with limited labelled data. Distinguish feature extraction from fine-tuning and describe a suitable training procedure.** ## Answer — step by step ### 1. Definition Transfer learning reuses knowledge learned on a source task to help solve a target task. For image classification, a pretrained CNN can provide visual features that a new classifier uses for the target classes. This can reduce the amount of target data and training needed, provided the source features are useful for the target task. ### 2. Feature extraction and fine-tuning | Method | Which parameters learn? | Main effect | | --- | --- | --- | | Feature extraction | Keep pretrained backbone weights fixed; train a new classification head | Learn to classify using fixed pretrained features | | Fine-tuning | Update selected pretrained layers as well as the new head | Adapt representations to the target task | Freezing a layer stops its parameter updates; it does not remove its stored weights. Fine-tuning typically exposes more trainable parameters and therefore needs particular care with small datasets. ![Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update.](https://deep-learning-study-room.vercel.app/assets/figures/v_transfer.svg) Diagram: Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update.. Labels: Reuse learned features, then adapt the output | Reuse learned features, then adapt the output | Example: start with an image model trained on a large dataset. | New images | Your smaller | labeled dataset | Feature extractor | Pretrained | convolutional layers | New head | Your classes | Feature extraction: freeze the pretrained layers. | Fine-tuning: unfreeze selected layers and train carefully. | Use validation data to choose settings. A small learning rate often helps | preserve useful features. Feature extraction freezes the pretrained backbone; fine-tuning permits selected pretrained layers to update. Tap the diagram to enlarge. ### 3. Training procedure 1. Create separate training, validation and test splits for the target problem. 2. Choose a suitable pretrained CNN and apply its required input size and preprocessing. 3. Replace the original classifier with a new head matching the target classes. 4. Freeze the backbone and train the new head on the training set. 5. If validation results justify it, unfreeze selected later layers and continue with an appropriately smaller learning rate. 6. Select the best checkpoint using validation performance; evaluate the final model on the held-out test set. ### 4. Concrete example For a small dataset containing cats and dogs, reuse a pretrained image backbone and replace its original classifier with a binary-classification head. First learn the new head with the backbone frozen. Later, selected high-level layers may be fine-tuned if this improves validation performance. **Final answer:** feature extraction learns a new classifier over fixed features; fine-tuning also adapts selected pretrained features. The correct choice depends on target data quantity, similarity to the source data and validation results. **Yaad rakho:** Freeze mein purane features fixed; fine-tune mein selected purane features bhi seekhte hain. --- ## TEE · TEE paper rehearsal Canonical: https://deep-learning-study-room.vercel.app/read/dl/tee/mocks/tee/ Reader: https://deep-learning-study-room.vercel.app/#tee/mocks/TEE 3 hours · 100 marks. Q1 compulsory + any four from Q2–Q7. Historical paper rehearsal. ## F25 Q1a ## Question **Question (5 marks).** Give four ML versus DL differences with examples. Choose and justify ML or DL for (1) 2,000 short product reviews, (2) labeled financial fraud records, (3) captions for 100,000 labeled natural-scene images. ## Answer — step by step Deep learning is a branch of machine learning that learns representations using neural networks with multiple processing layers. Machine learning is itself a branch of artificial intelligence. In image recognition, early CNN layers can detect local edges, later layers combine patterns into textures and parts, and deeper layers support object recognition. | Comparison | Traditional ML approaches | Deep learning approaches | | --- | --- | --- | | Features | Often depend on manually designed features | Often learn features jointly with the prediction task | | Data | Often strong baselines for smaller structured datasets | Training large models from scratch often benefits from abundant data | | Computation | Many methods can train economically on CPUs | Large networks often need substantial compute and memory | | Interpretability | Some models such as small decision trees are easier to inspect | Large multilayer representations can be harder to explain | These are tendencies, not rules. A pretrained network can be useful with a small target dataset, and boosted trees can outperform deep networks on some tabular problems. Choose using data type, size, available pretrained models, resources and held-out performance. Drivers of deep learning include improved hardware, large datasets, automatic differentiation, improved activations/optimization and useful architectures. Advantages include automatic feature learning and strong modeling of complex image, audio and sequence patterns. Limitations include data and compute requirements, overfitting, sensitivity to training choices, and difficulty explaining decisions. Applications include image classification, speech recognition, translation and medical-image analysis. **Choices:** (1) Start with traditional ML using text features because the labeled dataset is small; pretrained text models are a viable extension. (2) Start with a tabular ML classifier such as a tree ensemble; labeled structured features do not automatically require DL. (3) DL is appropriate for learning visual features and generating a word sequence; a CNN encoder with a sequence decoder is a syllabus-aligned design. Compare held-out metrics rather than claiming one method must always win. ## F25 Q1b ## Question **Question (5 marks).** For automated leaf-disease image classification, discuss factors affecting loss selection. For classes Normal, Mild, Severe with target [0,1,0] and predicted probabilities [0.1,0.8,0.1], compute and interpret categorical cross-entropy. ## Answer — step by step ### Given Classes = Normal,Mild,Severe; target = [0,1,0]. Predicted probabilities = [0.1,0.8,0.1]; use natural logarithms. ### To find Calculate categorical cross-entropy and interpret the result. ### Formula — which one and why **Categorical cross-entropy.** For mutually exclusive classes with a one-hot target, all zero-target terms disappear. The loss is the negative log of the probability assigned to the true class. yⱼ = target entry; pⱼ = predicted probability; c = index of the true class. Use natural logarithms unless another base is specified. $$ L=-\sum_j y_j\ln p_j $$ $$ L=-\ln p_c $$ **Yaad rakhne ka cue:** One-hot mein sirf correct class ki probability ka minus ln. ### Solution — step by step Select a loss according to the task and output meaning: real-valued regression often uses MSE/MAE, binary classification uses binary cross-entropy, and one-of-K classification uses categorical cross-entropy. Consider class imbalance, noisy labels/outliers, whether outputs are calibrated probabilities, and differentiability for optimization. **Step 1: expand the categorical cross-entropy sum.** $$ L=-[0\ln(0.1)+1\ln(0.8)+0\ln(0.1)] $$ **Step 2: keep the term for the true class.** $$ L=-\ln(0.8) $$ **Step 3: calculate its natural logarithm.** $$ \ln(0.8)\approx-0.223143551 $$ **Step 4: negate the logarithm.** $$ L\approx0.223143551 $$ **Final answer:** loss ≈ 0.223144 nats. The true class received probability 0.8, so the loss is relatively small; probability 1 would give loss zero. ## F25 Q1c ## Question **Question (5 marks).** Which regularizer leads to sparsity and why? For an MNIST CNN with high training accuracy and lower validation accuracy, explain how dropout works during training and how it can improve generalization. ## Answer — step by step Overfitting means learning training-specific patterns that do not generalize well. Very good training results with substantially worse validation results suggest overfitting, provided both sets are comparable and correctly evaluated. Regularization limits this tendency; its strength should be chosen using validation data. $$ J_{L1}=L+\lambda\sum_j|w_j|,\qquad J_{L2}=L+\frac{\lambda}{2}\sum_jw_j^2 $$ L1 adds a magnitude penalty and can make weights exactly zero at the optimum, encouraging sparsity. Away from zero its added gradient is λ sign(w). L2 adds gradient λw under the displayed convention and smoothly discourages large weights. L2 generally shrinks many weights without making them exactly zero. Some notes omit the half in L2; their added gradient is 2 λw. Dropout randomly removes selected activations during training, reducing dependence on a particular combination of units. With inverted dropout, retained units are divided by the keep probability q; at inference dropout is disabled and the ordinary network is used. Early stopping monitors validation loss and restores the best validation checkpoint before prolonged fitting to training noise. Data augmentation creates label-preserving input variations. Too much regularization may underfit: both training and validation performance can then worsen. ## F25 Q1d ## Question **Question (5 marks).** Describe sequence processing, context and variable-length sentences in RNNs. Explain their use in language translation, as illustrated by the historical reference to Google Translate. ## Answer — step by step A recurrent neural network processes a sequence in order and maintains a hidden state that summarizes earlier information. Its weights are shared across time steps. $$ h_t=\tanh(W_{xh}x_t+W_{hh}h_{t-1}+b_h),\quad p_t=\operatorname{softmax}(W_{hy}h_t+b_y) $$ ![Current input and previous state jointly determine the new state](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: Current input and previous state jointly determine the new state. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← Current input and previous state jointly determine the new state Tap the diagram to enlarge. At each step, transform the current token representation, transform the previous hidden state, add the bias and apply the activation. An output layer may produce a prediction at each step or only at the end. This allows a sequence's length to vary while the parameter set stays fixed. For next-word prediction, tokenize text, represent each word numerically, update the state for each prefix token, and apply an output softmax over the vocabulary. Train against the actual next token using cross-entropy. For sentiment classification, use the sequence representation to predict positive, neutral or negative. Word order matters: “not good” should not be interpreted like “good”. A basic feedforward model has no built-in recurrent state, though a fixed window can be supplied explicitly. RNNs provide recurrent context but have sequential computation costs and difficulty learning long dependencies because of vanishing or exploding gradients. An encoder-decoder RNN can encode a source sentence and generate a target sentence of a different length; this is a historical RNN translation explanation, not a claim about any commercial service's current architecture. ## F25 Q2a ## Question **Question (10 marks).** A 2 → 2 → 1 sigmoid network has all four input-to-hidden weights 0.5, both hidden-to-output weights 0.25, hidden and output biases 0, and target 1. Perform forward and backward passes to update the x₁ → h₁ weight, h₁ → output weight and output bias at η = 0.5. The paper does not state the numerical inputs or loss. ![F25 Q2a connections with explicitly assumed inputs](https://deep-learning-study-room.vercel.app/assets/figures/net_f25.svg) Diagram: F25 Q2a connections with explicitly assumed inputs. Labels: Read each receiving neuron as one row of weights | INPUT | HIDDEN | OUTPUT | x1=1 | x2=1 | h1 | h2 | o1 | target 1 | h1 incoming: x1 × 0.5 + x2 × 0.5; bias 0 | h2 incoming: x1 × 0.5 + x2 × 0.5; bias 0 | o1 incoming: h1 × 0.25 + h2 × 0.25; bias 0 | Use these OLD connections for all gradients in the same pass. F25 Q2a connections with explicitly assumed inputs Tap the diagram to enlarge. ## Answer — step by step ### Given Architecture 2 → 2 → 1; every neuron uses sigmoid. All four input-to-hidden weights = 0.5; both hidden-to-output weights = 0.25. All biases = 0; target = 1; η = 0.5. The paper omits inputs and loss: this worked solution explicitly assumes x = [1,1] and half squared loss. ### To find Calculate the forward pass, then gradients and updates for x₁→h₁, h₁→output and the output bias. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **Half summed squared loss.** This loss convention takes one-half of the sum of squared prediction errors. Use it when the question specifies it, or explicitly state it as an assumption when the paper omits the loss. Differentiating cancels the factor one-half against the square derivative. oⱼ = output j; yⱼ = target j; L = half summed squared loss under the stated or explicitly assumed convention. $$ L=\frac12\sum_j(o_j-y_j)^2 $$ $$ \frac{\partial L}{\partial o_j}=o_j-y_j $$ **Yaad rakhne ka cue:** Half aur square ka 2 cancel: output derivative = prediction − target. **Sigmoid and its slope.** Use sigmoid when the neuron is specified to squash its preactivation into (0,1). Once its output is known, its derivative needs only output × (1−output). z = preactivation; p = sigmoid output; e is the exponential base, not a prediction error in this formula. $$ p=\frac{1}{1+e^{-z}} $$ $$ \frac{dp}{dz}=p(1-p) $$ **Yaad rakhne ka cue:** Sigmoid output p; uska slope p × (1−p). **Backpropagation.** The loss tells us how each output must change. Multiply by that neuron’s activation slope to obtain its delta. A weight gradient is receiving delta × sending value. Hidden deltas must include every outgoing path. δ (delta) = derivative of loss with respect to a neuron’s preactivation; f′ = activation derivative; aᵢ = sending activation; Wⱼᵢ = connection weight. Use the exact loss scaling in the question. $$ \delta_{o_j}=\frac{\partial L}{\partial o_j}f_o\prime(z_{o_j}) $$ $$ \delta_{h_i}=f_h\prime(z_{h_i})\sum_jW_{ji}^{(2)}\delta_{o_j} $$ $$ \frac{\partial L}{\partial W_{ji}}=\delta_ja_i $$ $$ \frac{\partial L}{\partial b_j}=\delta_j $$ $$ W_{ji,\mathrm{new}}=W_{ji}-\eta\frac{\partial L}{\partial W_{ji}} $$ **Yaad rakhne ka cue:** Receiving delta × sending value = us connection ka gradient. ### Solution — step by step **Assumptions:** use x = [1,1] and half squared loss. The paper permits suitable assumptions and its key also introduces assumed inputs. There is no unique numerical result without them. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. All gradients below use the OLD weights from this same forward pass. **Hidden neuron 1:** $$ z_{h_1} = (0.5 \times 1) + (0.5 \times 1) + (0) = 1 $$ $$ h_{1}=\frac{1}{1+e^{-(1)}}=0.731058579 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.5 \times 1) + (0.5 \times 1) + (0) = 1 $$ $$ h_{2}=\frac{1}{1+e^{-(1)}}=0.731058579 $$ **Output neuron 1:** $$ z_{o_1} = (0.25 \times 0.731058579) + (0.25 \times 0.731058579) + (0) = 0.365529289 $$ $$ o_{1}=\frac{1}{1+e^{-(0.365529289)}}=0.590378257 $$ **Step 2 loss convention.** This worked solution uses half summed squared loss, L = ½Σ(o − y)². Follow the loss specified in the question; where none is supplied, this is an explicit assumption. Prediction minus target = [-0.409621743]. Squared errors = [0.167789972]. Half-sum squared loss = 0.083894986; mean squared error = 0.167789972. These are different unless the number of output components makes the scale coincide. **Output delta 1:** $$ \delta_{o_1} = 1 \times (0.590378257-1) \times 0.241831771 = -0.099059551 $$ The derivative is o(1−o). **Step 3 output weights.** Gradient = receiving delta × sending activation. An output bias gradient equals its delta. **h1 → o1:** The calculation on h1 → o1 Sending activation**0.731058579**h1 × Receiving delta**−0.099059551**δ at o1 = Connection gradient**−0.072418335**h1 → o1 Now update this weight Old weight**0.25** → New weight**0.286209167** Subtract η × gradient = **−0.036209168**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,1}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,1}}=\delta_{o_1}h_1 $$ $$ g_{1,1}^{(2)}=(-0.099059551)\times(0.731058579)\approx -0.072418335 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,1,\mathrm{new}}=0.25-0.5(-0.072418335)\approx 0.286209167 $$ **h2 → o1:** The calculation on h2 → o1 Sending activation**0.731058579**h2 × Receiving delta**−0.099059551**δ at o1 = Connection gradient**−0.072418335**h2 → o1 Now update this weight Old weight**0.25** → New weight**0.286209167** Subtract η × gradient = **−0.036209168**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies its activation. $$ g_{1,2}^{(2)}=\frac{\partial L}{\partial W^{(2)}_{1,2}}=\delta_{o_1}h_2 $$ $$ g_{1,2}^{(2)}=(-0.099059551)\times(0.731058579)\approx -0.072418335 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(2)}_{1,2,\mathrm{new}}=0.25-0.5(-0.072418335)\approx 0.286209167 $$ **Step 4 hidden deltas.** A hidden unit can influence EVERY connected output, so add those paths BEFORE multiplying by the hidden activation derivative. **Hidden 1 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.099059551** × Old outgoing weight**0.25** = Path contribution**−0.024764888** ↓ Add every path contribution**−0.024764888**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.25)\times(-0.099059551)\approx -0.024764888 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.024764888 $$ $$ \delta_{h_1} = -0.024764888 \times 0.196611933 = -0.004869072 $$ **Hidden 2 downstream contribution:** Calculate each outgoing path using the **old** outgoing weight and that output's delta. Several output paths meet at this hidden neuron Output 1 Output delta**−0.099059551** × Old outgoing weight**0.25** = Path contribution**−0.024764888** ↓ Add every path contribution**−0.024764888**Then multiply by the hidden activation derivative. Multiply within each path. Add the resulting contributions at the shared hidden neuron. **Contribution from output 1.** $$ C_1=(0.25)\times(-0.099059551)\approx -0.024764888 $$ **Accumulate the contributions, then apply the hidden activation derivative below.** $$ \text{Sum}=C_1\approx -0.024764888 $$ $$ \delta_{h_2} = -0.024764888 \times 0.196611933 = -0.004869072 $$ **Step 5 input weights.** Multiply the hidden delta by the corresponding raw input, not by the hidden output. **x1 → h1:** The calculation on x1 → h1 Sending input**1**x1 × Receiving delta**−0.004869072**δ at h1 = Connection gradient**−0.004869072**x1 → h1 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,1}}=\delta_{h_1}x_1 $$ $$ g_{1,1}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,1,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x2 → h1:** The calculation on x2 → h1 Sending input**1**x2 × Receiving delta**−0.004869072**δ at h1 = Connection gradient**−0.004869072**x2 → h1 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{1,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{1,2}}=\delta_{h_1}x_2 $$ $$ g_{1,2}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{1,2,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x1 → h2:** The calculation on x1 → h2 Sending input**1**x1 × Receiving delta**−0.004869072**δ at h2 = Connection gradient**−0.004869072**x1 → h2 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,1}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,1}}=\delta_{h_2}x_1 $$ $$ g_{2,1}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,1,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **x2 → h2:** The calculation on x2 → h2 Sending input**1**x2 × Receiving delta**−0.004869072**δ at h2 = Connection gradient**−0.004869072**x2 → h2 Now update this weight Old weight**0.5** → New weight**0.502434536** Subtract η × gradient = **−0.002434536**. This weight **increases**. The sending value and receiving delta have different roles. All deltas still use the original forward-pass weights. The receiving neuron supplies the delta; the sending neuron supplies the input value. $$ g_{2,2}^{(1)}=\frac{\partial L}{\partial W^{(1)}_{2,2}}=\delta_{h_2}x_2 $$ $$ g_{2,2}^{(1)}=(-0.004869072)\times(1)\approx -0.004869072 $$ Apply the learning rate to this gradient and subtract the result from the old weight. $$ W^{(1)}_{2,2,\mathrm{new}}=0.5-0.5(-0.004869072)\approx 0.502434536 $$ **Step 6 bias updates.** A bias receives a constant input 1, so its gradient equals the corresponding delta. Use the same old forward pass and learning rate as the weight updates. $$ b_{h_1,\mathrm{new}}=0-0.5(-0.004869072)=0.002434536 $$ $$ b_{h_2,\mathrm{new}}=0-0.5(-0.004869072)=0.002434536 $$ $$ b_{o_1,\mathrm{new}}=0-0.5(-0.099059551)=0.049529775 $$ These are independent per-neuron biases. If the question fixes biases, keep them fixed. If it specifies a single tied bias, add all deltas sharing that bias before updating it. **Requested values:** input weight **0.502434536**, output weight **0.286209167**, output bias **0.049529775**. Other supplied inputs produce different correct conditional results. ## F25 Q2b ## Question **Question (10 marks).** Explain how ResNet skip connections influence gradient flow, feature reuse and vanishing gradients. Compare InceptionNet and AlexNet learning and feature extraction. ## Answer — step by step A residual block learns a residual mapping F(x) and adds the input through a shortcut: y = F(x)+x, followed by an activation when the block design specifies one. A plain block without the shortcut outputs F(x). If F(x) becomes zero, the residual block can still pass x forward. ![The shortcut adds the input to the learned residual before the next stage](https://deep-learning-study-room.vercel.app/assets/figures/residual.svg) Diagram: The shortcut adds the input to the learned residual before the next stage. Labels: x | Learned layers F(x) | + | y | Shortcut S(x): identity or projection | y = F(x) + S(x) | S(x) = x for identity; S(x) = W\_s x for projection. | Add matching shapes; do not concatenate. The shortcut adds the input to the learned residual before the next stage Tap the diagram to enlarge. $$ \frac{\partial L}{\partial x}=\frac{\partial L}{\partial y}\left(\frac{\partial F}{\partial x}+I\right) $$ The identity term gives a direct gradient path. This helps optimize very deep networks and reduces the burden of learning an identity mapping through many nonlinear layers. Features can be reused or refined instead of being completely reconstructed by every block. Residual connections address optimization degradation and can help gradient flow, although they do not guarantee that vanishing gradients can never occur anywhere in the network. Elementwise addition requires compatible shapes. If channel counts or spatial sizes differ, use a suitable projection shortcut, often a 1 × 1 convolution with the required stride. A shortcut is an addition, not channel concatenation. Example: x = [2,−1] and F(x) = [0.5,0.2] give y = [2.5,−0.8]; a subsequent ReLU gives [2.5,0]. Without the shortcut the preactivation output would be [0.5,0.2]. An Inception module captures features at multiple scales in parallel. A typical module has a 1 × 1 branch, a 1 × 1 reduction followed by 3 × 3 convolution, a 1 × 1 reduction followed by 5 × 5 convolution, and a pooling branch followed by 1 × 1 projection. Padding and strides preserve compatible spatial sizes so branch outputs can be concatenated along the channel axis. ![Parallel branches preserve height and width then concatenate channels](https://deep-learning-study-room.vercel.app/assets/figures/inception.svg) Diagram: Parallel branches preserve height and width then concatenate channels. Labels: Input | 1 × 1 | 1 × 1 reduce | 3 × 3 | 1 × 1 reduce | 5 × 5 | 3 × 3 pool | 1 × 1 | Concatenate channels | All branch height/width values must match. Parallel branches preserve height and width then concatenate channels Tap the diagram to enlarge. A 1 × 1 convolution mixes input channels at each position. With fewer output channels, it forms a bottleneck before an expensive larger kernel. ReLU following it adds nonlinearity. This reduces the number of input channels processed by 3 × 3 or 5 × 5 kernels while preserving useful learned combinations. For H × W spatial size, Cin input channels and Cout output filters, a K × K convolution uses H × W × Cin × Cout × K² multiply-accumulates when the output spatial size stays H × W. A bottleneck changes the calculation to H × W × Cin × r + H × W × r × Cout × K². A multiply-accumulate is counted as one MAC here; some FLOP conventions count it as two operations. The benefit depends on the selected bottleneck width r. Compared with AlexNet's sequential kernels, Inception explicitly combines local and larger-scale features within one module. AlexNet learns features along a mainly sequential stack of convolutional layers. Inception learns several spatial scales in parallel within a module and combines their channels. Both learn filter weights through backpropagation; their structural organization differs. ## F25 Q3a ## Question **Question (10 marks).** Train one epoch of a single perceptron with η = 1, initial weights and bias 0, threshold 0, on [−1,1,−1] → 0, [−1,1,1] → 1, [1,−1,1] → 1, [−1,1,1] → 1. Then map biological-neuron components to artificial-neuron components. ## Answer — step by step ### Given Initial weights and bias = 0; η = 1; threshold = 0. Samples in order: [−1,1,−1]→0; [−1,1,1]→1; [1,−1,1]→1; [−1,1,1]→1. Use output 1 when net > 0 and 0 otherwise, matching the supplied key; one epoch. ### To find Process each sample in order, carrying updated weights forward, and report final weights/bias. ### Formula — which one and why **Perceptron learning rule.** Evaluate the stated threshold, compare the predicted class with its target, then correct weights and bias. The next sample uses the newly updated parameters. t = target; o = thresholded output; η = learning rate; xᵢ = input i. Binary and bipolar output conventions must follow the question. $$ z=\sum_iw_ix_i+b $$ $$ w_i^{\mathrm{new}}=w_i+\eta(t-o)x_i $$ $$ b^{\mathrm{new}}=b+\eta(t-o) $$ **Yaad rakhne ka cue:** Prediction nikalo; target−prediction se correction nikalo. ### Solution — step by step **State the threshold rule.** Use binary output 1 if z>0 and 0 otherwise, matching the associated key. Update Δw = η(t−o)x and Δb = η(t−o), with η = 1. A different rule at zero can change the learning trajectory. Write your convention explicitly. ### Epoch 1, sample 1 **Step 1: copy this sample and target.** x = [-1, 1, -1], target = 0. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (0 × -1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, -1] = [0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [0, 0, 0], new b = 0. ### Epoch 1, sample 2 **Step 1: copy this sample and target.** x = [-1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (0 × 1) + (0) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [-1, 1, 1] = [-1, 1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [-1, 1, 1], new b = 1. ### Epoch 1, sample 3 **Step 1: copy this sample and target.** x = [1, -1, 1], target = 1. **Step 2: calculate the current net.** z = (-1 × 1) + (1 × -1) + (1 × 1) + (1) = 0 **Step 3: apply the threshold.** output = 0. **Step 4: calculate the update multiplier.** Update multiplier = 1 **Step 5: calculate all weight changes.** Δw = 1 × [1, -1, 1] = [1, -1, 1] **Step 6: calculate the bias change.** Δb = 1. **Step 7: carry the updated parameters forward.** New w = [0, 0, 2], new b = 2. ### Epoch 1, sample 4 **Step 1: copy this sample and target.** x = [-1, 1, 1], target = 1. **Step 2: calculate the current net.** z = (0 × -1) + (0 × 1) + (2 × 1) + (2) = 4 **Step 3: apply the threshold.** output = 1. **Step 4: calculate the update multiplier.** Update multiplier = 0 **Step 5: calculate all weight changes.** Δw = 0 × [-1, 1, 1] = [0, 0, 0] **Step 6: calculate the bias change.** Δb = 0. **Step 7: carry the updated parameters forward.** New w = [0, 0, 2], new b = 2. **Final after exactly the requested epochs:** w = **[0, 0, 2]**, b = **2**. A finite number of requested epochs does not itself prove convergence or correct classification of every sample. Biological dendrites receive signals, synapses influence connection strength, the cell body integrates information, and an axon carries an output signal. In the artificial analogy, inputs represent received signals, weights represent connection strengths, the weighted sum plus bias performs aggregation, and an activation function produces an output. The bias shifts the firing threshold. This is a simplified computational analogy, not a complete biological model. ![The artificial neuron multiplies each input by a weight then adds bias and activates](https://deep-learning-study-room.vercel.app/assets/figures/neuron.svg) Diagram: The artificial neuron multiplies each input by a weight then adds bias and activates. Labels: INPUTS | WEIGHTED SUM | ACTIVATION | x1 | w1 | x2 | w2 | x3 | w3 | z = Σwx + b | a = f(z) | Add one bias b | Multiply → add → activate The artificial neuron multiplies each input by a weight then adds bias and activates Tap the diagram to enlarge. ## F25 Q3b ## Question **Question (10 marks).** Design face-recognition attendance using a CNN when the supplied dataset is very small. Discuss ways to address the issue and explain transfer learning. ## Answer — step by step Transfer learning reuses features learned on a source task to solve a related target task. For a small face-attendance dataset, a pretrained CNN provides a stronger starting point than estimating a very large model entirely from that small dataset. 1. Collect labeled examples with varied lighting, pose and background. Split data so duplicate or near-duplicate photos do not leak across sets. For evaluation on enrolled identities, use different capture sessions in each split. 2. Resize images and apply the preprocessing expected by the pretrained model. Use label-preserving augmentation such as small crops, shifts or illumination changes. 3. Load a suitable pretrained CNN, remove its source classification head and attach a new head with one output per required class. For mutually exclusive classes use softmax and categorical cross-entropy. 4. Freeze the convolutional base and train the new head. Frozen parameters still participate in prediction but are not updated in this stage. 5. If validation results justify it, unfreeze selected later layers and fine-tune with a smaller learning rate. Use early stopping and appropriate regularization. 6. Evaluate on held-out sessions, inspect per-class errors, and reject low-confidence/unregistered cases if the attendance design requires that behavior. Feature extraction keeps the base frozen; fine-tuning updates some pretrained weights. Benefits include reduced target-data requirements and faster useful training. Risks include source-target mismatch and overfitting the small target set. Transfer learning improves the starting point, not a guarantee of perfect recognition. ## F25 Q4a ## Question **Question (10 marks).** For N = 1,000 samples and batch size 100, count weight updates per epoch for batch GD, SGD and mini-batch GD. For J(θ) = 0.5 θ², η = 0.1, momentum 0.9, v₀ = 0, θ₀ = 1, compute two momentum iterations. List the required momentum-training hyperparameters. ## Answer — step by step ### Given N = 1,000; mini-batch size = 100. J(θ) = 0.5θ²; θ₀ = 1; v₀ = 0; η = 0.1; β = 0.9. Use signed velocity. ### To find Count updates per epoch for batch, stochastic and mini-batch GD; then perform two momentum iterations. ### Formula — which one and why **Number of updates.** An update occurs after one processed batch. Full-batch GD uses one batch per epoch; SGD uses one sample per batch. Mini-batches need a stated remainder policy. N = dataset size; B = batch size; E = number of epochs. Ceiling rounds upward to include a smaller final batch; floor drops it. $$ U_{\mathrm{batch}}=E $$ $$ U_{\mathrm{SGD}}=EN $$ $$ U_{\mathrm{keep}}=E\left\lceil\frac NB\right\rceil $$ $$ U_{\mathrm{drop}}=E\left\lfloor\frac NB\right\rfloor $$ **Yaad rakhne ka cue:** Kitne batches process hue, utne updates hue. **Momentum with signed velocity.** Combine the retained old displacement with the current downhill correction, then add that signed displacement to the old parameter. v = signed velocity/displacement; β = momentum coefficient; g = current gradient; η = learning rate. This convention uses +v in the parameter update. $$ v_t=\beta v_{t-1}-\eta g_t $$ $$ \theta_t=\theta_{t-1}+v_t $$ **Yaad rakhne ka cue:** Pehle velocity banao; phir old parameter mein velocity jodo. ### Solution — step by step **Step 1: count full-batch updates.** All 1,000 samples contribute to one update: 1 per epoch. **Step 2: count stochastic updates.** One update per sample: 1,000 per epoch. **Step 3: count mini-batches.** $$ 1000/100=10 $$ Use signed velocity vₜ = βvₜ₋₁−ηgₜ and θₜ = θₜ₋₁+vₜ; g(θ) = θ. ### Momentum iteration 1 **Step 1: evaluate the gradient at the current parameter.** $$ g_1=1 $$ **Step 2: retain the previous displacement.** $$ 0.9(0)=0 $$ **Step 3: scale the current gradient.** $$ 0.1(1)=0.1 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_1=0-0.1=-0.1 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_1=1+(-0.1)=0.9 $$ ### Momentum iteration 2 **Step 1: evaluate the gradient at the current parameter.** $$ g_2=0.9 $$ **Step 2: retain the previous displacement.** $$ 0.9(-0.1)=-0.09 $$ **Step 3: scale the current gradient.** $$ 0.1(0.9)=0.09 $$ **Step 4: combine to obtain the new signed velocity.** $$ v_2=-0.09-0.09=-0.18 $$ **Step 5: add new velocity to the actual old parameter.** $$ \theta_2=0.9+(-0.18)=0.72 $$ **Hyperparameters:** the basic momentum update has two main tunable coefficients, learning rate η and momentum β. A full training run also requires choices such as epochs and batch size (one for literal SGD), and may include a learning-rate schedule. There is no universal single count for all training configurations; state what is counted. Initial θ and v are initialization values, not learned outputs of the first update. ## F25 Q4b ## Question **Question (10 marks).** Define vanishing gradients in RNNs and explain why they occur. Use “The dogs barked at the thief and … were appreciated” to discuss long context. Explain LSTM forgetting/retention, then compute the retained contribution for f = [0.1,0.9], Cprevious = [2,1]. ## Answer — step by step ### Given Forget-gate outputs f = [0.1,0.9]; old cell = [2,1]. No input gate or candidate is supplied. ### To find Explain forgetting/retention and calculate the retained old-cell part only. ### Formula — which one and why **LSTM retained old memory.** Multiply each old cell value by its corresponding forget-gate value. This gives only the old contribution, not the complete new cell. f = activated forget gate; C\_old = previous cell state; ⊙ means multiply matching entries. $$ C_{\mathrm{retained}}=f\odot C_{\mathrm{old}} $$ **Yaad rakhne ka cue:** Forget gate × old cell = bacha hua purana hissa. ### Solution — step by step Backpropagation through time trains an RNN by treating its time-unrolled computations as a deep computational graph. 1. Initialize the hidden state and run the forward recurrence from the first input to the last. 2. Compute the specified output loss, at the final step or across several steps. 3. Start from the last loss and propagate derivatives backwards through output and hidden operations. 4. At each earlier state, add the direct output-loss contribution and the contribution arriving from future states. 5. Sum all uses of each shared weight before applying the optimizer update. $$ \delta_t=(W_{hy}^{T}\delta_t^{out}+W_{hh}^{T}\delta_{t+1})\odot(1-h_t^2),\quad \frac{\partial L}{\partial W_{hh}}=\sum_t\delta_t h_{t-1}^{T} $$ The tanh derivative is 1−h². Repeated multiplication by recurrent weights and activation derivatives may shrink gradients toward zero or amplify them greatly. Early words then receive a weak learning signal or updates become unstable. Gated architectures help long-context learning; suitable initialization and learning rates help training; clipping limits exploding gradients. Truncated BPTT restricts the backward time window to reduce computation and memory. It does not mean creating independent weights for each time step. ![BPTT follows this unrolled network backwards and sums gradients of shared weights](https://deep-learning-study-room.vercel.app/assets/figures/rnn.svg) Diagram: BPTT follows this unrolled network backwards and sums gradients of shared weights. Labels: RNN | Output p1 | Input x1 | t = 1 | h1 | RNN | Output p2 | Input x2 | t = 2 | h2 | RNN | Output p3 | Input x3 | t = 3 | h0 | Shared weights at all steps; forward → and backward through time ← BPTT follows this unrolled network backwards and sums gradients of shared weights Tap the diagram to enlarge. The plural subject “dogs” must remain represented across intervening words. A vanilla RNN may fail to learn that dependency when early gradients are weak; it is not guaranteed to fail on every such sentence. In an LSTM, forget components close to 1 retain relevant subject information along the cell path. **Step 1: retain the first old-cell component.** $$ 0.1(2)=0.2 $$ **Step 2: retain the second old-cell component.** $$ 0.9(1)=0.9 $$ **Step 3: assemble the retained old-cell contribution.** $$ f\odot C_{\mathrm{old}}=[0.2,0.9] $$ **Final answer:** [0.2,0.9] is only retained old content. The input gate and candidate are missing, so the complete updated cell cannot be calculated. ## F25 Q5a ## Question **Question (10 marks).** State the purpose of each LSTM gate. For “Artificial Intelligence has revolutionized every domain. The work force will have to update themselves to remain relevant,” explain a context change. Calculate C and h given i = 0.7, f = 0.4, o = 0.8, candidate = 0.9, Cprevious = 0.5. Why is LSTM more effective than a simple RNN for language modeling? ## Answer — step by step ### Given Activated gate values: i = 0.7,f = 0.4,o = 0.8. Candidate = 0.9; old cell = 0.5. ### To find Explain the context change in the question; calculate the new cell and hidden state. ### Formula — which one and why **LSTM cell and hidden state.** Multiply old memory by the forget gate; multiply candidate content by the input gate; add those two contributions. Apply tanh to the new cell and multiply by the output gate. f,i,o = activated forget/input/output gates; C\_old = previous cell; C\_tilde = candidate content; C = new cell; h = visible hidden state. ⊙ means elementwise multiplication. $$ C=f\odot C_{\mathrm{old}}+i\odot\widetilde C $$ $$ h=o\odot\tanh C $$ **Yaad rakhne ka cue:** Purana rakho + naya lo; phir tanh aur output gate. ### Solution — step by step An LSTM contains a cell state C and hidden state h. Sigmoid gates control the fractions of information retained, admitted and exposed; a tanh transform proposes new content. $$ f_t=\sigma(W_f[h_{t-1},x_t]+b_f),\quad i_t=\sigma(W_i[h_{t-1},x_t]+b_i) $$ $$ \widetilde C_t=\tanh(W_c[h_{t-1},x_t]+b_c),\quad C_t=f_t\odot C_{t-1}+i_t\odot\widetilde C_t $$ $$ o_t=\sigma(W_o[h_{t-1},x_t]+b_o),\quad h_t=o_t\odot\tanh(C_t) $$ ![The cell path retains selected old content and adds selected new content](https://deep-learning-study-room.vercel.app/assets/figures/lstm.svg) Diagram: The cell path retains selected old content and adds selected new content. Labels: C previous | × | + | C new | tanh | × | h | new | σ: f | σ: i | tanh: C~ | σ: o | × | [h previous, x current] | Each gate has its own learned weights and bias. The cell path retains selected old content and adds selected new content Tap the diagram to enlarge. The forget gate f near 1 retains old content; near 0 removes it. The input gate i controls candidate admission. The output gate o selects the visible state. The additive cell path helps useful information and training signals travel over longer intervals, especially when forget gates retain relevant components. In language modeling, the cell can retain an earlier subject while intervening words modify other features. When the topic changes, a learned forget gate can reduce obsolete information while the input gate admits new context. These are intuitive interpretations, not numeric gate values derivable from words alone. Exact values require weights, inputs and previous states. LSTMs often learn longer dependencies more effectively than vanilla RNNs, but do not guarantee perfect indefinite memory. The second sentence shifts attention toward the workforce while retaining useful prior AI context. Input gates can admit the new subject; forget gates can reduce obsolete details. **Step 1: retain old memory.** $$ fC_{old}=0.4(0.5)=0.2 $$ **Step 2: admit candidate content.** $$ i\widetilde C=0.7(0.9)=0.63 $$ **Step 3: add the two contributions.** $$ C=0.2+0.63=0.83 $$ **Step 4: apply tanh to the updated cell.** $$ \tanh(0.83)\approx0.680476006 $$ **Step 5: multiply by the output gate.** $$ h=0.8(0.680476006)\approx0.544380805 $$ **Final answer:** C = 0.83 and h ≈ 0.544380805. The supplied gates are already activated; do not apply sigmoid again. ## F25 Q5b ## Question **Question (10 marks).** Explain AdaGrad, give its update rule and discuss advantages and disadvantages compared with standard SGD. ## Answer — step by step AdaGrad gives each parameter its own effective learning rate based on the sum of its past squared gradients. Start G₀ = 0, choose η and a small positive ε, then apply elementwise: $$ G_t=G_{t-1}+g_t^2,\quad\theta_t=\theta_{t-1}-\frac{\eta g_t}{\sqrt{G_t}+\epsilon} $$ Frequently updated coordinates accumulate a larger G and take smaller steps. Rarely active coordinates keep a relatively larger effective learning rate, which can be useful for sparse features. The square removes the sign in the accumulator; the unsquared current gradient still determines the direction of the parameter change. Example with θ₀ = 1, η = 0.1, gradients 2 then 1 and negligible ε: first G = 4, denominator = 2, change = 0.1 × 2/2 = 0.1, θ₁ = 0.9. Next G = 5, denominator = √5 = 2.236068, change = 0.1/2.236068 = 0.04472136, θ₂ = 0.85527864. Advantages over ordinary SGD include coordinatewise adaptation and usefulness for sparse gradients. Disadvantages include extra accumulator memory and continuously shrinking effective learning rates: G never decreases, so learning may become too slow. AdaGrad still requires a base learning rate and does not guarantee a global minimum. RMSProp addresses indefinite accumulation by replacing the sum with an exponentially weighted average. ## F25 Q6a ## Question **Question (10 marks).** Calculate every activation shape and trainable-parameter count for: Input 128 × 128 × 3. → Conv K = 3, filters = 16, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Conv K = 3, filters = 32, stride = 1, padding = 0. → Pool K = 2, stride = 2, padding = 0. → Flatten. → Dense 240 units. → Dense 120 units. → Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. Given matrices Input A Input A | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Kernel K Kernel K | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Stride 2; P = 0 (no padding). ## Answer — step by step ### Given Input 128 × 128 × 3. Conv K = 3, filters = 16, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Conv K = 3, filters = 32, stride = 1, padding = 0. Pool K = 2, stride = 2, padding = 0. Dense 240 units. Dense 120 units. Dense 5 units. Also convolve the input A with the kernel K shown below, valid padding and stride 2. ### To find Find every layer shape, its parameter count and the network total. Find every convolution output cell. ### Formula — which one and why **Complete CNN calculation.** Carry each output shape into the next layer. Convolution changes spatial size and channel count; pooling changes spatial size; flattening converts all activations into dense-layer inputs. H,W,C = feature-map height, width and channels; K = kernel size; S = stride; P\_pad = padding. One bias is assumed for each learned filter or dense neuron unless specified otherwise. $$ N_{\mathrm{out}}=\left\lfloor\frac{N+2P_{\mathrm{pad}}-K}{S}\right\rfloor+1 $$ $$ P_{\mathrm{conv}}=(K^2C_{\mathrm{in}}+1)C_{\mathrm{out}} $$ $$ n_{\mathrm{flat}}=HWC $$ $$ P_{\mathrm{dense}}=(n_{\mathrm{in}}+1)n_{\mathrm{out}} $$ $$ P_{\mathrm{pool}}=P_{\mathrm{flatten}}=0 $$ **Yaad rakhne ka cue:** Ek layer ka output, next layer ka input; shape aur parameters alag count. **Convolution values.** At each valid location, multiply corresponding window and kernel entries, add all products and add the filter bias once. Move by the stated stride and repeat. X = input; K = kernel; S = stride; b = filter bias; u,v locate a kernel entry. Use the printed kernel without flipping under the stated deep-learning convention. $$ Y_{ij}=\sum_{u,v}X_{iS+u,jS+v}K_{uv}+b $$ **Yaad rakhne ka cue:** Matching cells multiply; sab products add; bias sirf ek baar. ### Solution — step by step Follow each selected input window below. Matching row colors connect its input cells, kernel coefficients and products; the outlined output cell shows where their sum belongs. Use output size $\left\lfloor\frac{N+2P-K}{S}\right\rfloor+1$ , Conv weights K² × Cin × Cout and biases Cout, Dense weights Nin × Nout and biases Nout. Assume a bias per learned filter/unit, full input-channel connectivity and no trainable pooling parameters. ReLU and softmax activations themselves add no trainable parameters. ### Layer-by-layer calculation Carry the complete output shape of one layer into the next. Spatial size determines the number of activations; it does not multiply the number of shared convolution parameters. ### Layer 1 — convolution Layer 1: what changes and what is learned Input shape**128 × 128 × 3**height × width × channels → Convolution**16 filters · 3 × 3 · S = 1 · P = 0** → Output shape**126 × 126 × 16**height × width × channels Weights**432** + Biases**16** = Trainable parameters**448** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **128 × 128 × 3**. Use 16 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{128+2(0)-3}{1}\right\rfloor+1=126 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **126 × 126 × 16**. **3. Count trainable values.** Each filter receives all 3 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times3\times16=432 $$ $$ N_{\mathrm{biases}}=16 $$ $$ P_1=432+16=448 $$ ### Layer 2 — pooling Layer 2: what changes and what is learned Input shape**126 × 126 × 16**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**63 × 63 × 16**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **126 × 126 × 16**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{126-2}{2}\right\rfloor+1=63 $$ Pooling acts independently on each channel. The output is **63 × 63 × 16**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. ### Layer 3 — convolution Layer 3: what changes and what is learned Input shape**63 × 63 × 16**height × width × channels → Convolution**32 filters · 3 × 3 · S = 1 · P = 0** → Output shape**61 × 61 × 32**height × width × channels Weights**4,608** + Biases**32** = Trainable parameters**4,640** Spatial positions share the filter weights. Output depth equals the number of filters. Input: **63 × 63 × 16**. Use 32 filters of spatial size 3 × 3, stride 1, and padding 0 on each side. **1. Calculate height and width.** $$ N_{\mathrm{out}}=\left\lfloor\frac{63+2(0)-3}{1}\right\rfloor+1=61 $$ **2. Determine output channels.** Each filter produces one feature map. Therefore the output is **61 × 61 × 32**. **3. Count trainable values.** Each filter receives all 16 input channels. Its weights are reused at every spatial position. $$ N_{\mathrm{weights}}=3\times3\times16\times32=4{,}608 $$ $$ N_{\mathrm{biases}}=32 $$ $$ P_3=4{,}608+32=4{,}640 $$ ### Layer 4 — pooling Layer 4: what changes and what is learned Input shape**61 × 61 × 32**height × width × channels → Pooling**2 × 2 window · S = 2** → Output shape**30 × 30 × 32**height × width × channels Weights**0** + Biases**0** = Trainable parameters**0** Spatial dimensions change; the channel count stays the same. Pooling learns no weights. Input: **61 × 61 × 32**. Use a 2 × 2 window, stride 2, and zero padding. $$ N_{\mathrm{out}}=\left\lfloor\frac{61-2}{2}\right\rfloor+1=30 $$ Pooling acts independently on each channel. The output is **30 × 30 × 32**. **Trainable parameters: 0.** Max/average pooling has no learned filter weights or biases. The division gives 29.5; take its floor before adding 1. A partial window outside the unpadded input is not included. ### Layer 5 — flatten Layer 5: what changes and what is learned Input shape**30 × 30 × 32**height × width × channels → Flatten**Rearrange activations** → Output shape**28,800**output features Weights**0** + Biases**0** = Trainable parameters**0** The same activation values are rearranged into one vector. No parameter is learned. Input: **30 × 30 × 32**. Rearrange all activation values into one vector. $$ n_{\mathrm{flat}}=30\times30\times32=28{,}800 $$ Output: a vector with **28,800 entries**. Flattening changes the arrangement, so it adds **0 trainable parameters**. ### Layer 6 — fully connected layer Layer 6: what changes and what is learned Input shape**28,800**input features → Dense layer**240 neurons** → Output shape**240**output features Weights**6,912,000** + Biases**240** = Trainable parameters**6,912,240** Every output neuron receives every input feature, with one bias per output neuron. Input: **28,800 features**. Output: **240 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=28{,}800\times240=6{,}912{,}000 $$ $$ N_{\mathrm{biases}}=240 $$ $$ P_6=6{,}912{,}000+240=6{,}912{,}240 $$ ### Layer 7 — fully connected layer Layer 7: what changes and what is learned Input shape**240**input features → Dense layer**120 neurons** → Output shape**120**output features Weights**28,800** + Biases**120** = Trainable parameters**28,920** Every output neuron receives every input feature, with one bias per output neuron. Input: **240 features**. Output: **120 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=240\times120=28{,}800 $$ $$ N_{\mathrm{biases}}=120 $$ $$ P_7=28{,}800+120=28{,}920 $$ ### Layer 8 — fully connected layer Layer 8: what changes and what is learned Input shape**120**input features → Dense layer**5 neurons** → Output shape**5**output features Weights**600** + Biases**5** = Trainable parameters**605** Every output neuron receives every input feature, with one bias per output neuron. Input: **120 features**. Output: **5 neurons**. Every output neuron connects to every input feature and has one bias. $$ N_{\mathrm{weights}}=120\times5=600 $$ $$ N_{\mathrm{biases}}=5 $$ $$ P_8=600+5=605 $$ ### Summary of the complete network | Layer | Spatial or flatten calculation | Output | Weights | Biases | Total | | --- | --- | --- | --- | --- | --- | | 1 conv | $\left\lfloor\frac{128+2 \times 0-3}{1}\right\rfloor+1$ | 126 x 126 x 16 | 432 | 16 | 448 | | 2 pool | $\left\lfloor\frac{126-2}{2}\right\rfloor+1$ | 63 x 63 x 16 | 0 | 0 | 0 | | 3 conv | $\left\lfloor\frac{63+2 \times 0-3}{1}\right\rfloor+1$ | 61 x 61 x 32 | 4,608 | 32 | 4,640 | | 4 pool | $\left\lfloor\frac{61-2}{2}\right\rfloor+1$ | 30 x 30 x 32 | 0 | 0 | 0 | | 5 flat | 30 × 30 × 32 | 28800 | 0 | 0 | 0 | | 6 dense | 28800 → 240 | 240 | 6,912,000 | 240 | 6,912,240 | | 7 dense | 240 → 120 | 120 | 28,800 | 120 | 28,920 | | 8 dense | 120 → 5 | 5 | 600 | 5 | 605 | **Parameter substitutions, in layer order:** Layer 1: (3 x 3 x 3 + 1) x 16 = 448. Layer 3: (3 x 3 x 16 + 1) x 32 = 4,640. Layer 6: (28800 + 1) x 240 = 6,912,240. Layer 7: (240 + 1) x 120 = 28,920. Layer 8: (120 + 1) x 5 = 605. **Total:** 448 + 4,640 + 6,912,240 + 28,920 + 605 = **6,946,853 trainable parameters**. Pooling changes activation size; it does not itself add learned weights. Always floor noninteger spatial dimensions. **Separate convolution calculation:** **Step 1 shape:** $\left\lfloor\frac{5+2 \times 0-3}{2}\right\rfloor+1$ = 2 along each axis. Use the deep-learning cross-correlation convention: do not flip the printed kernel. ### Output position (1, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 1) into one output cell Input rows **1–3**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −2 | | 4 | 0 | −4 | | 3 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−2) = 1** Row 2**4 + 0 + (−4) = 0** Row 3**3 + 0 + (−1) = 2** **1 + 0 + 2 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(1)\times(0)+(2)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(2)\times(-2)=0 $$ **Window row 3.** $$ R_3=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Add the rows to obtain one output cell.** $$ Y_{1,1}=(1)+(0)+(2)=3 $$ ### Output position (1, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (1, 2) into one output cell Input rows **1–3**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 2 | 0 | −1 | | 4 | 0 | 0 | | 1 | 0 | −1 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**2 + 0 + (−1) = 1** Row 2**4 + 0 + 0 = 4** Row 3**1 + 0 + (−1) = 0** **1 + 4 + 0 = 5** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 1, column 2. **Window row 1.** $$ R_1=(2)\times(1)+(2)\times(0)+(1)\times(-1)=1 $$ **Window row 2.** $$ R_2=(2)\times(2)+(1)\times(0)+(0)\times(-2)=4 $$ **Window row 3.** $$ R_3=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Add the rows to obtain one output cell.** $$ Y_{1,2}=(1)+(4)+(0)=5 $$ ### Output position (2, 1) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 1) into one output cell Input rows **3–5**, columns **1–3**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 3 | 0 | −1 | | 0 | 0 | 0 | | 1 | 0 | 0 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**3 + 0 + (−1) = 2** Row 2**0 + 0 + 0 = 0** Row 3**1 + 0 + 0 = 1** **2 + 0 + 1 = 3** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 1. **Window row 1.** $$ R_1=(3)\times(1)+(2)\times(0)+(1)\times(-1)=2 $$ **Window row 2.** $$ R_2=(0)\times(2)+(3)\times(0)+(0)\times(-2)=0 $$ **Window row 3.** $$ R_3=(1)\times(1)+(1)\times(0)+(0)\times(-1)=1 $$ **Add the rows to obtain one output cell.** $$ Y_{2,1}=(2)+(0)+(1)=3 $$ ### Output position (2, 2) Use this position's input window. Multiply corresponding entries row by row; then add all row totals. Follow window (2, 2) into one output cell Input rows **3–5**, columns **3–5**. The window moves **2** cells per output step. 1. Select the input window 1. Select the input window | | | | | | | --- | --- | --- | --- | --- | | 3 | 1 | 2 | 2 | 1 | | 2 | 1 | 2 | 1 | 0 | | 3 | 2 | 1 | 0 | 1 | | 0 | 3 | 0 | 1 | 2 | | 1 | 1 | 0 | 2 | 3 | Only the outlined cells belong to this output. 2. Match the kernel 2. Match the kernel | | | | | --- | --- | --- | | 1 | 0 | −1 | | 2 | 0 | −2 | | 1 | 0 | −1 | Same row and column positions; no matrix multiplication. 3. Multiply cell by cell 3. Multiply cell by cell | | | | | --- | --- | --- | | 1 | 0 | −1 | | 0 | 0 | −4 | | 0 | 0 | −3 | Each cell is input × matching kernel coefficient. 4. Add the products row by row Row 1**1 + 0 + (−1) = 0** Row 2**0 + 0 + (−4) = −4** Row 3**0 + 0 + (−3) = −3** **0 + (−4) + (−3) = −7** 5. Place it in the output 5. Place it in the output | | | | --- | --- | | 3 | 5 | | 3 | −7 | Output row 2, column 2. **Window row 1.** $$ R_1=(1)\times(1)+(0)\times(0)+(1)\times(-1)=0 $$ **Window row 2.** $$ R_2=(0)\times(2)+(1)\times(0)+(2)\times(-2)=-4 $$ **Window row 3.** $$ R_3=(0)\times(1)+(2)\times(0)+(3)\times(-1)=-3 $$ **Add the rows to obtain one output cell.** $$ Y_{2,2}=(0)+(-4)+(-3)=-7 $$ **Final output:** $\begin{bmatrix}3 & 5\\3 & -7\end{bmatrix}$ . Each output came from its own complete window; stride changes the next window position, not the multiplication rule. ## F25 Q6b ## Question **Question (10 marks).** Explain activations and the role of nonlinearity. Compare sigmoid, tanh, ReLU and Leaky ReLU including advantages and disadvantages. ## Answer — step by step An activation function transforms a neuron's weighted sum. Hidden-layer nonlinearity allows a network to represent nonlinear decision boundaries. Without it, composing affine layers gives another affine map: W₂(W₁x+b₁)+b₂ = (W₂W₁)x+(W₂b₁+b₂). More linear layers alone therefore do not solve XOR. **Sigmoid** $$ \sigma(z)=\frac{1}{1+e^{-z}},\qquad 0<\sigma(z)<1 $$ Here z is the weighted sum and e is the exponential base. The whole expression 1 + e raised to minus z belongs below the fraction bar. Advantage: smooth probability-like output for binary classification. Limitation: saturation makes gradients small; the derivative never exceeds 0.25. **Hyperbolic tangent** $$ \tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},\qquad -1<\tanh(z)<1 $$ The numerator subtracts the exponentials; the denominator adds them. Advantage: zero-centered signed outputs, useful in recurrent states and candidates. Limitation: large positive or negative inputs still saturate. **ReLU** $$ \operatorname{ReLU}(z)=\max(0,z)=\begin{cases}z&z\geq0\\0&z<0\end{cases} $$ The output range is zero to positive infinity. Advantage: inexpensive and derivative 1 on the positive side. Limitation: the negative side has zero gradient and a unit can become inactive. **Leaky ReLU** $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\geq0\\\alpha z&z<0\end{cases},\qquad 0<\alpha<1 $$ Alpha is a small chosen slope on the negative side. Advantage: negative inputs retain a small gradient. Limitation: alpha must be chosen; a fixed slope is not automatically optimal. At zero the two branches agree on output zero, but the derivative is convention-dependent. Sigmoid derivative is a(1−a), tanh derivative is 1−a², and ReLU derivative is 1 for positive preactivation and 0 for negative preactivation. At zero ReLU is nondifferentiable; a common implementation selects zero. Strong saturation can slow learning by shrinking gradients. ReLU avoids positive-side saturation but cannot guarantee convergence. For mutually exclusive classes, use softmax at the output; for real-valued regression, a linear output is often suitable. ## F25 Q7a ## Question **Question (10 marks).** Explain feedforward information flow, weights, biases and activations, then calculate the output. Input x = [1, 0]. Each row below lists the incoming weights of one receiving neuron. W₁ is shown below; b₁ = [0.1, 0.2]; W₂ is shown below; b₂ = [0.1]. Hidden and output activations are ReLU. ![Given matrices for this question](https://deep-learning-study-room.vercel.app/assets/figures/given_412d332265.svg) Diagram: Given matrices for this question. Labels: GIVEN VALUES — read each matrix row from left to right | W₁ | 0.2 | 0.4 | 0.3 | 0.7 | W₂ | 0.5 | 0.6 Given matrices for this question Tap the diagram to enlarge. ## Answer — step by step ### Given Input x = [1, 0]. W₁ is printed with the question. b₁ = [0.1, 0.2]. W₂ is printed with the question. b₂ = [0.1]. Hidden and output activations are ReLU. ### To find Find all preactivations, hidden activations and the final output. Explain the role of weights, biases and activations in this forward pass. ### Formula — which one and why **Forward propagation.** Evaluate one layer completely before supplying its activated outputs to the next layer. Each layer has its own weights and biases. x = input vector; h = hidden outputs; W₁,b₁ = hidden parameters; W₂,b₂ = output parameters; f₁,f₂ = the stated activations. $$ z_h=W_1x+b_1 $$ $$ h=f_1(z_h) $$ $$ z_o=W_2h+b_2 $$ $$ o=f_2(z_o) $$ **Yaad rakhne ka cue:** Pehli layer ka activated output hi agli layer ka input hai. **ReLU and Leaky ReLU.** ReLU returns zero for negative input and retains positive input. Leaky ReLU instead multiplies a negative input by its supplied slope. z = preactivation; α (alpha) = supplied negative-side slope for Leaky ReLU. The derivative at zero needs an explicit convention. $$ \operatorname{ReLU}(z)=\max(0,z) $$ $$ \operatorname{LeakyReLU}(z)=\begin{cases}z&z\ge0\\\alpha z&z<0\end{cases} $$ **Yaad rakhne ka cue:** Positive ko rehne do; negative ko zero. Leaky mein chhota negative rehne do. ### Solution — step by step A feedforward network transforms inputs layer by layer with no recurrent feedback. Weights scale contributions; bias shifts the preactivation; activation supplies the nonlinear transformation. **Step 1 map the arrows.** Each row of W₁ receives all inputs; each row of W₂ receives all hidden outputs. Bias is added once per receiving neuron. **Hidden neuron 1:** $$ z_{h_1} = (0.2 \times 1) + (0.4 \times 0) + (0.1) = 0.3 $$ $$ h_{1} = \max(0,0.3) = 0.3 $$ **Hidden neuron 2:** $$ z_{h_2} = (0.3 \times 1) + (0.7 \times 0) + (0.2) = 0.5 $$ $$ h_{2} = \max(0,0.5) = 0.5 $$ **Output neuron 1:** $$ z_{o_1} = (0.5 \times 0.3) + (0.6 \times 0.5) + (0.1) = 0.55 $$ $$ o_{1} = \max(0,0.55) = 0.55 $$ **Matrix-orientation note:** this solution uses column input and rows as receiving neurons, giving **0.55**. The historical key uses the transpose interpretation for W₁ and obtains 0.61. If the problem does not define its orientation, state your convention; do not silently swap it during calculation. Under the alternative: hidden values = [0.3,0.6], output = 0.5 × 0.3+0.6 × 0.6+0.1 = 0.61. ## F25 Q7b ## Question **Question (10 marks).** Explain hierarchical CNN feature extraction and the roles of convolution, pooling and fully connected layers. Why prefer CNNs over fully connected ANNs for images? Give medical-image and autonomous-vehicle applications and advantages. ## Answer — step by step A convolutional neural network processes local spatial neighborhoods using learnable filters. A filter slides across the image; the same weights are reused at different locations. Each filter produces one output channel. Early features often respond to edges, subsequent layers combine local patterns into textures or parts, and deeper features support object-level classification. ![One filter is reused at successive spatial positions](https://deep-learning-study-room.vercel.app/assets/figures/convolution.svg) Diagram: One filter is reused at successive spatial positions. Labels: Step 1: start (1,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 1×1 + 2×0 + 4×0 | + 5×(−1) = −4 | Step 2: start (1,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 2×1 + 3×0 + 5×0 | + 6×(−1) = −4 | Step 3: start (2,1) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 4×1 + 5×0 + 7×0 | + 8×(−1) = −4 | Step 4: start (2,2) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | K = [1, 0] | [0, −1] | 5×1 + 6×0 + 8×0 | + 9×(−1) = −4 | Output: [−4, −4; −4, −4]. Kernel is not flipped. One filter is reused at successive spatial positions Tap the diagram to enlarge. Convolution extracts local patterns, a nonlinear activation lets the network combine them nonlinearly, and pooling reduces spatial resolution and sensitivity to small local shifts. Flattening or global pooling converts feature maps into a classification representation; a dense output layer produces class scores or probabilities. Compared with a fully connected image network, CNNs exploit locality and weight sharing, often requiring far fewer parameters for early image processing. An RGB filter extends through all input channels. Pooling does not change the number of channels and normally has no trainable parameters. A CNN is not automatically invariant to every transformation. For medical-image analysis, a CNN can classify a chest X-ray or identify suspicious regions from labeled scans, learning useful spatial patterns. For autonomous vehicles, CNN-based perception can detect road signs, pedestrians or lanes from camera images. Benefits include learned hierarchical visual features and reuse across image positions; both applications require held-out evaluation under relevant acquisition conditions. # Natural Language Processing — all public teaching content Exact source JSON: https://deep-learning-study-room.vercel.app/exports/nlp.json --- ## M2 · What NLP is doing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/foundations--language/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/foundations/language Prerequisites for M2 · no earlier NLP knowledge assumed **Natural Language Processing (NLP)** is the study of computational methods for analysing, interpreting and producing human language. An NLP system receives text or speech and produces an output such as a tag, a meaning, a translation or a probability. For M2, the central question is often: **how do counts from a training corpus become probabilities, and how do those probabilities help choose a sentence, a tag or a word sense?** You will learn the vocabulary first, then do the arithmetic with complete working. Training textexamples already observed → Countshow often things occur → Probabilitiescounts divided by totals → Decisioncompare possible answers | Term | Meaning | Concrete example | | --- | --- | --- | | Corpus | A collection of language data. The plural is corpora. | Three sentences saved together form a tiny corpus. | | Document | One text item within a collection. | One review or one article. | | Token | One occurrence produced by tokenization. | “cats like cats” has 3 word tokens. | | Type | A distinct token value. | The same text has 2 types: cats and like. | | Vocabulary | The set of token types recognised by the model. | V = {cats, like}; vocabulary size is 2. | | Training | Estimating rules or numbers using examples. | Count how often like follows cats. | | Test | Applying the learned model to a new example. | Calculate a probability for a sentence not used to build the counts. | ### Three levels you will repeatedly see | Level | Question it answers | Example | | --- | --- | --- | | Word sequence | Which word is likely next? | After “drink”, is “water” likely? | | Syntax | What role or structure does the word have? | “book” is a noun in “a book” and a verb in “book tickets”. | | Semantics | Which meaning is intended? | “bank” near “deposit” means a financial institution. | **Yaad rakho:** Word ka order = language model; word ka role = POS; word ka matlab = sense. ### Your M2 route Read the probability refresher below, then **N-grams and smoothing → POS/HMM → parsing/chunking/NER → word relations and WSD**. Answers sit directly below their questions. The separate Formulas tab is for revision after learning the methods. Both 2-mark questions are compulsory; attempt any two of the three 3-mark questions. Total attempted = 10 marks in one hour. M2 scope: Unit 3 slides 62–160; Unit 4 slides 1–128; Unit 5 slides 1–57. Earlier-unit material in this short opening chapter is included only to supply prerequisites. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Tokens, counts and symbols Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/foundations--notation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/foundations/notation Prerequisites for M2 · no earlier NLP knowledge assumed A formula is a short way of stating a calculation. First name the quantities; only then insert the numbers. In this guide, C means a count and P means a probability. | Notation | Read it as | What to do | | --- | --- | --- | | $$ C(\text{cat})=3 $$ | Count of cat is three | Count every occurrence, including repeats. | | $$ P(\text{cat}) $$ | Probability of cat | A number from 0 to 1. | | $$ P(\text{cat}\mid\text{the}) $$ | Probability of cat given the | Restrict attention to occasions where the history is “the”. | | $$ \sum_i x_i $$ | Sum of the x values | Add the terms: x₁ + x₂ + … | | $$ \prod_i x_i $$ | Product of the x values | Multiply the terms: x₁ × x₂ × … | | $$ \arg\max_c s(c) $$ | The choice c with the largest score | Return a label, not the numeric score. | | $$ \max_c s(c) $$ | Largest score | Return a number. | ### Example 1 · tokens and types For the corpus “we study nlp” and “we like nlp”, find the token count, vocabulary and count of “we”. **Given** Two lowercase sentences. Ignore punctuation; do not add boundary markers. **To find** Token count N, vocabulary V, vocabulary size |V| and C(we). ### Formula and why it applies A token is an occurrence; a type is a distinct word. Add sentence lengths to count tokens; remove duplicates only when finding the vocabulary. ### Solution #### Step 1. List every occurrence we, study, nlp, we, like, nlp. #### Step 2. Count tokens $$ N=3+3=6 $$ #### Step 3. Keep one copy of each type V = {we, study, nlp, like}; |V| = 4. #### Step 4. Count the requested word “we” appears once in each sentence, hence C(we) = 2. N = 6; |V| = 4; C(we) = 2. **Yaad rakho:** Token mein repeat gin; type mein duplicate hata. ### Example 2 · sum and product Given probabilities 0.5, 0.2 and 0.1, compute their sum and their product. **Given** Three numbers: 0.5, 0.2, 0.1. **To find** Sum and product; recognise that these are different operations. ### Formula and why it applies $$ \sum_{i=1}^{3}p_i=p_1+p_2+p_3 $$ $$ \prod_{i=1}^{3}p_i=p_1p_2p_3 $$ Add for a sum. Multiply for a product. A sentence-chain calculation normally uses the product of its conditional probabilities. ### Solution #### Step 1. Add the first two numbers $$ 0.5+0.2=0.7 $$ #### Step 2. Complete the sum $$ 0.7+0.1=0.8 $$ #### Step 3. Multiply the first two numbers $$ 0.5\times0.2=0.1 $$ #### Step 4. Complete the product $$ 0.1\times0.1=0.01 $$ Sum = 0.8; product = 0.01. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Probability, without skipped steps Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/foundations--probability/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/foundations/probability Prerequisites for M2 · no earlier NLP knowledge assumed **Empirical probability** estimates how likely an event is from its observed frequency. Divide the number of relevant observations by the total number of observations in the selected group. For a conditional probability, the denominator is the count of the condition, not the size of the entire corpus. $$ P(A)=\frac{C(A)}{N} $$ $$ P(B\mid A)=\frac{C(A,B)}{C(A)} $$ Here A is the conditioning event, B is the event being predicted, C(A,B) counts observations containing both in the required relationship, and N is the total number of observations. “Given A” tells you which denominator to use. Select the condition“given A” → Count matching outcomesC(A,B) → Divide by condition countC(A) ### Example 1 · word probability A corpus contains 20 tokens; “rain” occurs 5 times. Find the unigram probability of “rain”. **Given** N = 20 and C(rain) = 5. **To find** P(rain). ### Formula and why it applies $$ P(w)=\frac{C(w)}{N} $$ Unigram probability ignores neighbouring words. Every token in the corpus is one observation. ### Solution #### Step 1. Insert the count and total $$ P(\text{rain})=\frac{5}{20} $$ #### Step 2. Divide $$ \frac{5}{20}=0.25 $$ P(rain) = 0.25, or 25%. ### Example 2 · conditional probability In a corpus, “drink” is followed by another word 8 times. It is followed by “water” 6 times. Find P(water | drink). **Given** C(drink as a history) = 8; C(drink, water) = 6. **To find** Probability that the next word is water, given drink. ### Formula and why it applies $$ P(\text{water}\mid\text{drink})=\frac{C(\text{drink water})}{C(\text{drink})} $$ The condition is drink, so use the 8 drink-histories as the denominator. ### Solution #### Step 1. Identify the favourable continuations Six of the eight continuations are water. #### Step 2. Divide $$ P(\text{water}\mid\text{drink})=\frac68=0.75 $$ P(water | drink) = 0.75. **Yaad rakho:** Bar ke right mein jo hai, denominator uska count. ### Example 3 · multiply a sequence A model gives P(a | start) = 0.4, P(cat | a) = 0.5 and P(end | cat) = 0.2. Find the probability of the complete sentence “a cat”. **Given** Start → a = 0.4; a → cat = 0.5; cat → end = 0.2. **To find** Probability of the three-transition path. ### Formula and why it applies $$ P(\text{a cat, end})=P(\text{a}\mid\text{start})P(\text{cat}\mid\text{a})P(\text{end}\mid\text{cat}) $$ Every transition must happen in sequence, so multiply the model’s conditional probabilities. ### Solution #### Step 1. Write the three factors in order $$ P=0.4\times0.5\times0.2 $$ #### Step 2. Multiply the first pair $$ 0.4\times0.5=0.2 $$ #### Step 3. Multiply by the last factor $$ 0.2\times0.2=0.04 $$ Complete-sentence probability = 0.04. ### Example 4 · compare labels Two tag candidates have scores noun = 0.036 and verb = 0.048. Which tag does argmax return? If these are the only candidates, find the normalised noun probability. **Given** s(noun) = 0.036 and s(verb) = 0.048. **To find** Winning label and normalised noun probability. ### Formula and why it applies $$ \hat c=\arg\max_c s(c) $$ $$ P(c)=\frac{s(c)}{\sum_j s(j)} $$ A score need not sum to 1. Dividing each score by the sum gives a distribution over the candidates. ### Solution #### Step 1. Compare scores 0.048 > 0.036, so verb wins. #### Step 2. Find their total $$ 0.036+0.048=0.084 $$ #### Step 3. Divide the noun score by the total $$ P(\text{noun})=\frac{0.036}{0.084}=\frac37\approx0.428571 $$ argmax returns “verb”. The normalised noun probability is approximately 0.428571. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Bayes’ rule: prior, evidence and answer Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/foundations--bayes/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/foundations/bayes Prerequisites for M2 · no earlier NLP knowledge assumed **Bayes’ rule** reverses a conditional probability. A word-sense classifier asks “which sense is most likely given these context words?” Training data more easily supplies “how likely are these words when this sense is used?” Bayes connects the two. $$ P(c\mid x)=\frac{P(x\mid c)P(c)}{P(x)} $$ | Symbol | Meaning | | --- | --- | | c | A candidate class, tag or word sense. | | x | The observed evidence, such as a context word. | | P(c) | Prior: how common the class is before seeing the evidence. | | P(x | c) | Likelihood: how compatible the evidence is with that class. | | P(c | x) | Posterior: probability of the class after observing the evidence. | | P(x) | Evidence normaliser, shared by all classes for the same observation. | For ranking classes, calculate **prior × likelihood**. The common denominator does not affect which class is largest. For an actual posterior probability, divide by the sum of all class scores. ### A complete Bayes calculation A word has two senses: financial bank and river bank. Priors are 0.6 and 0.4. The probability of observing “loan” is 0.5 under the financial sense and 0.1 under the river sense. Find the most likely sense and its posterior. **Given** P(F) = 0.6; P(R) = 0.4; P(loan | F) = 0.5; P(loan | R) = 0.1. These two senses are exhaustive. **To find** Scores and P(F | loan). ### Formula and why it applies $$ s(c)=P(c)P(\text{loan}\mid c) $$ $$ P(F\mid\text{loan})=\frac{s(F)}{s(F)+s(R)} $$ Multiply prior and likelihood to reverse the conditioning. Then normalise because a posterior was requested. ### Solution #### Step 1. Score the financial sense $$ s(F)=0.6\times0.5=0.30 $$ #### Step 2. Score the river sense $$ s(R)=0.4\times0.1=0.04 $$ #### Step 3. Add the scores $$ P(\text{loan})=0.30+0.04=0.34 $$ #### Step 4. Normalise the financial score $$ P(F\mid\text{loan})=\frac{0.30}{0.34}=\frac{15}{17}\approx0.882353 $$ Financial bank wins; its posterior probability is about 0.882353. **Yaad rakho:** Pehle se kitna common × evidence kitna suit karta = score. ### Bayes Rule & Models id : nlp-m2-02-bayes-generative-discriminative title : Bayes Rule & Models title\_confirmed : true topic\_title : Bayes rule and generative versus discriminative models mode : m2 kind : focused status : available duration : 10:51 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Small probabilities, logs and calculator use Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/foundations--calculator/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/foundations/calculator Prerequisites for M2 · no earlier NLP knowledge assumed Repeated multiplication of probabilities produces small numbers. A calculator may show 0.0000125 as **1.25 × 10⁻⁵**; they are the same value. A logarithm turns a product into a sum. Log probabilities are often negative because probabilities between 0 and 1 have negative logs. $$ \log(ab)=\log a+\log b $$ $$ \log_b x=y\quad\Longleftrightarrow\quad b^y=x $$ Use the log base specified in the question. Natural log is ln (base e); ordinary calculator log is usually base 10. Perplexity is simplest to evaluate using the direct power formula. Do not mix bases in the same calculation. ### Calculator example · roots in perplexity A model assigns probability 1/16 to a test sequence with 4 predicted tokens. Calculate perplexity. **Given** P = 1/16; N = 4 predicted tokens. **To find** Perplexity PP. ### Formula and why it applies $$ PP=P^{-1/N}=\left(\frac1P\right)^{1/N} $$ Perplexity takes the reciprocal probability and then its Nth root. N must match the number of predicted events in the product. ### Solution #### Step 1. Take the reciprocal $$ \frac{1}{1/16}=16 $$ #### Step 2. Take the fourth root $$ PP=16^{1/4}=2 $$ #### Step 3. Interpret A perplexity of 2 means an effective average uncertainty comparable to two equally likely alternatives. It is not 2% error. PP = 2. ### Calculator entry order Enter the full product with parentheses. For the Nth root enter **(1 ÷ P)^(1 ÷ N)**. Keep at least 6 significant digits while working; round the final answer. If your calculator offers Ans, reuse the stored value instead of typing a heavily rounded result. **Yaad rakho:** Probability ke factors multiply; log mein add; perplexity mein ulta karke root. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Words, counts and probabilities: the notation you need Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-language-models--words-counts-probability/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-language-models/words-counts-probability Unit 3, pages 62–108; essential probability revision **A token** is one occurrence of a word. A **type** is one distinct word. A **corpus** is a collection of texts used for analysis or training. In “cats chase cats”, there are three word tokens but only two word types: cats and chase. A language model learns regularities from such a corpus to assign probabilities to word sequences. It does not simply count whether a sentence is grammatically correct. | Symbol | Read it as | Concrete meaning | | --- | --- | --- | | C(cat) | count of cat | Number of times cat occurs. | | C(the, cat) | count of the followed by cat | Count adjacent ordered pairs; cat the is different. | | P(cat) | probability of cat | A number from 0 to 1. | | P(cat | the) | probability of cat given the | Among contexts ending in the, how often cat follows. | | V | vocabulary size | Number of distinct outcomes the model may predict. | | N | number of tokens or predictions | Its exact meaning must be stated for the calculation. | The vertical bar **| means “given”**. The condition is on its right. In P(cat | the), the model has already seen “the” and is predicting “cat”. This direction matters: P(cat | the) and P(the | cat) usually have different denominators. **Question. A corpus contains 20 word tokens. The word cat occurs 5 times. Find the maximum-likelihood unigram probability of cat.** #### Given Total tokens N = 20; count C(cat) = 5. #### To find P(cat). #### Formula and why it applies $$ P(w)=\frac{C(w)}{N} $$ Relative frequency estimates the probability of drawing this word from the corpus. The denominator includes every token, not just distinct words. #### Solution ##### Step 1. Select the matching count The required outcome is cat, which occurs 5 times. ##### Step 2. Divide by the total number of trials $$ P(\text{cat})=\frac{5}{20}=0.25 $$ ##### Step 3. Interpret the result Under this unigram model, cat receives probability 0.25, or 25%. **Final answer.** P(cat) = 0.25. **Yaad rakho:** Jo chahiye uska count upar; total possible trials neeche. **Question. The word the has 10 outgoing occurrences in a corpus. It is followed by cat on 4 occasions and dog on 6. Find P(cat | the) and P(dog | the).** #### Given C(the, cat) = 4; C(the, dog) = 6; C(the) = 10 outgoing contexts. #### To find Both next-word probabilities. #### Formula and why it applies $$ P(w\mid h)=\frac{C(h,w)}{C(h)} $$ The condition h = the restricts attention to its 10 continuations. We do not divide by the total corpus size. #### Solution ##### Step 1. Use the cat continuation $$ P(\text{cat}\mid\text{the})=\frac{4}{10}=0.4 $$ ##### Step 2. Use the dog continuation $$ P(\text{dog}\mid\text{the})=\frac{6}{10}=0.6 $$ ##### Step 3. Check the complete row $$ 0.4+0.6=1 $$ **Final answer.** P(cat | the) = 0.4 and P(dog | the) = 0.6. **Yaad rakho:** Given waala word denominator choose karata hai. **Question. A collection has 8 sentences; 6 start with I and 2 start with we. Estimate each starting-word probability.** #### Given Each sentence receives a start marker . C() = 8; C(, I) = 6; C(, we) = 2. #### To find P(I | ) and P(we | ). #### Formula and why it applies $$ P(w_1\mid\langle s\rangle)=\frac{C(\langle s\rangle,w_1)}{C(\langle s\rangle)} $$ The start marker supplies the context for the first word; one start context is available per sentence. #### Solution ##### Step 1. Calculate the first option $$ P(I\mid\langle s\rangle)=6/8=0.75 $$ ##### Step 2. Calculate the second option $$ P(we\mid\langle s\rangle)=2/8=0.25 $$ **Final answer.** The estimated probabilities are 0.75 and 0.25. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Unigrams, bigrams and trigrams: construct them correctly Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-language-models--n-grams-construction/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-language-models/n-grams-construction Unit 3, pages 62–108; essential probability revision **Definition.** An n-gram is a contiguous sequence of n items. For word n-grams, the items are consecutive word tokens. A unigram contains one word; a bigram contains two; a trigram contains three. Character n-grams use characters instead, so always identify the unit being counted. Tokens: we | study | natural | language → Move a window one token at a time → Bigrams: we study / study natural / natural language For m tokens without padding, the number of n-gram *positions* is max(0, m−n+1). Repeated positions may contain the same n-gram type. A bag-of-n-grams representation counts each n-gram type and loses the overall sentence order beyond the local order inside each n-gram. $$ \text{Number of unpadded n-gram positions}=\max(0,m-n+1) $$ **Question. List all word unigrams, bigrams and trigrams for “the prime minister of our country”. Do not add boundary tokens.** #### Given Six tokens: the, prime, minister, of, our, country. n = 1, 2 and 3. #### To find All three lists and their counts. #### Formula and why it applies $$ \#\text{positions}=m-n+1 $$ Each window must contain consecutive words and move one position. No wraparound or invented final word is allowed. #### Solution ##### Step 1. Take one word at a time Unigrams: (the), (prime), (minister), (of), (our), (country). Count = 6−1+1 = 6. ##### Step 2. Take two consecutive words Bigrams: (the, prime), (prime, minister), (minister, of), (of, our), (our, country). Count = 6−2+1 = 5. ##### Step 3. Take three consecutive words Trigrams: (the, prime, minister), (prime, minister, of), (minister, of, our), (of, our, country). Count = 6−3+1 = 4. **Final answer.** 6 unigrams, 5 bigrams and 4 trigrams. **Question. Construct bigrams for “I like tea” using one start and one end marker.** #### Given Original tokens: I, like, tea. Add and once each. #### To find The boundary-aware bigram sequence. #### Formula and why it applies $$ \#\text{bigrams}=(m+2)-1=m+1 $$ The start context predicts the first word, and the last word predicts that the sentence ends. #### Solution ##### Step 1. Write the padded sequence I like tea ##### Step 2. Move a two-token window (, I), (I, like), (like, tea), (tea, ). ##### Step 3. Count the windows $$ 3+1=4 $$ **Final answer.** There are four boundary-aware bigrams. **Question. Construct trigrams for “I like tea” using two start markers and one end marker.** #### Given Original sentence has three words. For a trigram model each predicted token needs two prior context tokens. #### To find All trigram prediction events. #### Formula and why it applies $$ P(w_i\mid w_{i-2},w_{i-1}) $$ Two start markers provide a complete context even for the first word. #### Solution ##### Step 1. Pad with two starts I like tea ##### Step 2. Slide a three-token window (, , I), (, I, like), (I, like, tea), (like, tea, ). ##### Step 3. Count predicted events The model predicts I, like, tea and the end marker: four predictions. **Final answer.** Four trigrams. Start markers are context; they are not ordinary predicted words. **Question. Give the bag-of-bigrams counts for “a b a b” without padding.** #### Given Four tokens in their given order. #### To find Bigram positions, distinct bigram types and their counts. #### Formula and why it applies $$ \#\text{positions}=4-2+1=3 $$ A bag stores frequencies; repeated bigrams must not be discarded. #### Solution ##### Step 1. Enumerate all positions (a,b), (b,a), (a,b). ##### Step 2. Group identical pairs | Bigram type | Count | | --- | --- | | a b | 2 | | b a | 1 | **Final answer.** Three bigram positions but two distinct bigram types. **Yaad rakho:** n-gram mein n words; context mein n−1 words. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Collocations and the purpose of a language model Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-language-models--collocations-and-lm/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-language-models/collocations-and-lm Unit 3, pages 62–108; essential probability revision **A collocation** is a conventional word combination whose words occur together more strongly than ordinary chance or independent frequencies would suggest. Examples include “strong tea”, “heavy rain” and “red wine”. A bigram is merely two consecutive items; it need not be a collocation. “of the” can be frequent because both words are frequent, while a collocation expresses a particularly strong association. | Term | What it represents | Example | | --- | --- | --- | | Bigram | Any adjacent two-token sequence | the book | | Collocation | A characteristic strongly associated expression | heavy rain | | Language model | A distribution used to score or predict sequences | P(rain | heavy) | **Definition.** A statistical language model assigns a probability to a sequence of words, or equivalently predicts the next word from preceding context. It is useful for autocomplete, speech recognition, machine translation, spelling correction, assistive communication and ranking candidate text. High probability reflects patterns learned from its training distribution; it is not a guarantee of truth or grammatical correctness. Previous words: please turn your homework → Candidate: in / over / elephant → Language model gives each candidate a probability → Application selects or ranks candidates **Noisy-channel connection.** Suppose x is the observed noisy text or speech and y is a candidate intended message. The noisy-channel approach chooses the y that maximizes P(x | y)P(y). The channel model P(x | y) asks how y could produce x; the language model P(y) asks whether y is a plausible message. P(x) is the same for all candidate y and therefore can be omitted when ranking. $$ \hat y=\arg\max_y P(y\mid x)=\arg\max_y P(x\mid y)P(y) $$ **Question. A spelling-correction system considers “flowers” and “flours”. For the observed input, the channel likelihoods are 0.2 and 0.4, and the respective language-model probabilities in context are 0.08 and 0.01. Which candidate does the noisy-channel rule choose?** #### Given Candidate flowers: P(x|flowers)=0.2, P(flowers)=0.08. Candidate flours: P(x|flours)=0.4, P(flours)=0.01. #### To find The larger unnormalized candidate score. #### Formula and why it applies $$ \operatorname{score}(y)=P(x\mid y)P(y) $$ Both the error mechanism and the language context matter. The same denominator would normalize both candidates. #### Solution ##### Step 1. Score flowers $$ 0.2\times0.08=0.016 $$ ##### Step 2. Score flours $$ 0.4\times0.01=0.004 $$ ##### Step 3. Compare $$ 0.016>0.004 $$ **Final answer.** Choose flowers. The scores are not yet posterior probabilities. **Yaad rakho:** Har word-pair bigram hai; har frequent pair meaningful collocation nahi. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Probability chain rule and the Markov assumption Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-language-models--chain-rule-markov/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-language-models/chain-rule-markov Unit 3, pages 62–108; essential probability revision **The probability chain rule** decomposes the probability of a complete sequence into successive conditional probabilities. It is different from the derivative chain rule used in deep learning. Here we multiply probabilities, not derivatives. $$ P(w_1,w_2,w_3)=P(w_1)P(w_2\mid w_1)P(w_3\mid w_1,w_2) $$ Read left to right: probability of the first word; then probability of the second after the first; then probability of the third after both previous words. For a longer sentence, continue the same pattern. The product symbol ∏ simply means “multiply all of these terms”. $$ P(w_1,\ldots,w_m)=\prod_{i=1}^{m}P(w_i\mid w_1,\ldots,w_{i-1}) $$ **The Markov assumption** approximates long history by a limited recent context. A unigram model ignores preceding words. A bigram model keeps the previous word. A trigram model keeps the previous two words. This reduces the number of counts needed, but loses longer-range dependencies. | Model | Prediction rule | For the next word country after the prime minister of our | | --- | --- | --- | | Unigram | P(wᵢ) | P(country) | | Bigram | P(wᵢ | wᵢ₋₁) | P(country | our) | | Trigram | P(wᵢ | wᵢ₋₂,wᵢ₋₁) | P(country | of, our) | | Full chain rule | P(wᵢ | all previous words) | P(country | the, prime, minister, of, our) | **Question. For the three-word sentence “I like tea”, P(I)=0.2, P(like|I)=0.5 and P(tea|I,like)=0.4. Find its probability using the full chain rule.** #### Given Three conditional factors are given, including the full two-word history for tea. No end probability is requested. #### To find P(I, like, tea). #### Formula and why it applies $$ P(I,like,tea)=P(I)P(like\mid I)P(tea\mid I,like) $$ The chain rule supplies one factor per word in the specified sequence. #### Solution ##### Step 1. Insert the three supplied factors $$ P=0.2\times0.5\times0.4 $$ ##### Step 2. Multiply the first two $$ 0.2\times0.5=0.1 $$ ##### Step 3. Multiply by the third $$ 0.1\times0.4=0.04 $$ **Final answer.** The sentence probability is 0.04. **Question. A bigram model gives P(I|)=0.5, P(like|I)=0.6, P(tea|like)=0.25 and P(|tea)=0.8. Find the full sentence probability.** #### Given All four conditional probabilities, including the end event, are supplied. #### To find P(I like tea | ). #### Formula and why it applies $$ P=P(I\mid\langle s\rangle)P(like\mid I)P(tea\mid like)P(\langle/s\rangle\mid tea) $$ Under the bigram assumption, every factor uses one preceding token. #### Solution ##### Step 1. Insert the factors in sentence order $$ P=0.5\times0.6\times0.25\times0.8 $$ ##### Step 2. Start and second word $$ 0.5\times0.6=0.3 $$ ##### Step 3. Third word $$ 0.3\times0.25=0.075 $$ ##### Step 4. End the sentence $$ 0.075\times0.8=0.06 $$ **Final answer.** The complete sentence probability is 0.06. **Yaad rakho:** Sequence probability = saare required transitions ka product. **Typical 2-mark answer.** An n-gram language model predicts a word from its previous n−1 words using a Markov approximation. For example, a bigram model uses P(wᵢ | wᵢ₋₁). It estimates these probabilities from corpus counts and multiplies the appropriate factors to score a sentence. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Unigram estimation and complete worked examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-count-to-probability--unigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-count-to-probability/unigram-estimation Unit 3, pages 97–120 and appendix **Maximum likelihood estimation (MLE)** chooses probabilities equal to observed relative frequencies. In a unigram model, every word is predicted independently of its surrounding words. If a word occurs c times in N training tokens, its estimate is c/N. To score m specified words, multiply their m unigram probabilities. $$ P_{\mathrm{MLE}}(w)=\frac{C(w)}{N}\qquad P(w_1,\ldots,w_m)\approx\prod_{i=1}^{m}P(w_i) $$ Choose a tokenization convention before counting. Lowercasing, punctuation handling and boundary markers can change the vocabulary and counts. Never count words with one convention and form the denominator with another. A unigram model gives the same product to rearrangements of the same word multiset; that is one of its limitations. **Question. Train an unpadded, case-insensitive unigram model on “I like tea” and “I like coffee”. Find the probability of “I like tea”.** #### Given Training sentences: I like tea; I like coffee. Use lowercase words; exclude punctuation and sentence boundaries. #### To find The vocabulary, word probabilities and test-sentence probability. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P(I\ like\ tea)=P(I)P(like)P(tea) $$ The unigram approximation ignores order and uses total token count for every denominator. #### Solution ##### Step 1. Count every training token Each sentence contains three words, so N = 3+3 = 6. ##### Step 2. Count types and frequencies | Word | Count | MLE probability | | --- | --- | --- | | i | 2 | 2/6 = 1/3 | | like | 2 | 2/6 = 1/3 | | tea | 1 | 1/6 | | coffee | 1 | 1/6 | Vocabulary size V = 4. ##### Step 3. Write one factor per test word $$ P=\frac{2}{6}\times\frac{2}{6}\times\frac{1}{6} $$ ##### Step 4. Multiply the fractions $$ P=\frac{4}{216}=\frac{1}{54}\approx0.018519 $$ **Final answer.** The unigram sentence probability is approximately 0.018519. **Question. A corpus contains counts a: 4, b: 3 and c: 1. Find the unigram probability of “a b a”.** #### Given C(a)=4, C(b)=3, C(c)=1. No boundary markers. #### To find P(a b a). #### Formula and why it applies $$ P(a\,b\,a)=P(a)P(b)P(a) $$ A repeated test word contributes a probability factor at every occurrence; do not include it just once. #### Solution ##### Step 1. Find the total $$ N=4+3+1=8 $$ ##### Step 2. Estimate the required probabilities $$ P(a)=4/8=1/2,\qquad P(b)=3/8 $$ ##### Step 3. Use both occurrences of a $$ P=\frac12\times\frac38\times\frac12 $$ ##### Step 4. Simplify $$ P=\frac3{32}=0.09375 $$ **Final answer.** P(a b a) = 0.09375. **Yaad rakho:** Repeated word ka probability bhi repeat multiply hoga. **Question. An unsmoothed unigram model is trained on “red blue red”. Find the probability of “red green”.** #### Given N=3. Counts: red=2, blue=1, green=0. The test contains red and green. #### To find The unsmoothed test probability and the reason for its value. #### Formula and why it applies $$ P(red\ green)=P(red)P(green) $$ MLE assigns zero to an unseen outcome. Smoothing is needed if an unseen in-vocabulary item must receive probability. #### Solution ##### Step 1. Estimate red $$ P(red)=2/3 $$ ##### Step 2. Estimate the unseen word $$ P(green)=0/3=0 $$ ##### Step 3. Multiply $$ (2/3)\times0=0 $$ **Final answer.** Unsmoothed probability = 0. This reflects missing training evidence, not proof that the phrase is impossible. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Bigram counts: build the table before multiplying Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-count-to-probability--bigram-count-table/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-count-to-probability/bigram-count-table Unit 3, pages 97–120 and appendix A **bigram probability table** uses the previous word as its row and the next word as its column. Divide each cell count by its row's total outgoing count. With one end marker on every sentence, each ordinary training word has a continuation, so its outgoing count equals its token frequency. Do not create a bigram that crosses from one sentence into the next. $$ P(w\mid h)=\frac{C(h,w)}{\sum_{v\in V}C(h,v)}=\frac{C(h,w)}{C(h)} $$ Add boundaries to each sentence separately → List adjacent pairs → Count repeated pairs → Divide each row by its total → Follow the test sentence through the table **Question. Train a bigram model on “I like tea”, “I like coffee” and “you like tea”. Use one start and one end marker. Find the probability of “I like tea”.** #### Given Corpus: I like tea ; I like coffee ; you like tea . Preserve these tokens. #### To find All nonzero bigram estimates and the test probability. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Every sentence starts and ends independently. Four probability factors are required for the three test words plus end. #### Solution ##### Step 1. List the training pairs Sentence 1: (,I), (I,like), (like,tea), (tea,). Sentence 2: (,I), (I,like), (like,coffee), (coffee,). Sentence 3: (,you), (you,like), (like,tea), (tea,). ##### Step 2. Create the nonzero count table | History → next | Pair count | History count | Probability | | --- | --- | --- | --- | | → I | 2 | 3 | 2/3 | | → you | 1 | 3 | 1/3 | | I → like | 2 | 2 | 1 | | you → like | 1 | 1 | 1 | | like → tea | 2 | 3 | 2/3 | | like → coffee | 1 | 3 | 1/3 | | tea → | 2 | 2 | 1 | | coffee → | 1 | 1 | 1 | ##### Step 3. Write the test path $$ P=P(I\mid\langle s\rangle)P(like\mid I)P(tea\mid like)P(\langle/s\rangle\mid tea) $$ ##### Step 4. Substitute each matching row $$ P=\frac23\times1\times\frac23\times1 $$ ##### Step 5. Multiply $$ P=\frac49\approx0.444444 $$ **Final answer.** P(I like tea) = 4/9 ≈ 0.444444. **Question. Using the same three-sentence corpus given here—“I like tea”, “I like coffee”, “you like tea”—find the bigram probabilities of “you like coffee” and “tea like I”. Include boundaries.** #### Given Corpus with one start/end marker per sentence. Counts: start→you=1/3; you→like=1; like→coffee=1/3; coffee→end=1. No training sentence starts with tea. #### To find The two test probabilities. #### Formula and why it applies $$ P(\text{sentence})=\prod P(\text{next token}\mid\text{previous token}) $$ A model can assign a nonzero probability to a sentence it never saw if all of its local bigrams were seen. #### Solution ##### Step 1. Score you like coffee $$ P=\frac13\times1\times\frac13\times1=\frac19\approx0.111111 $$ ##### Step 2. Inspect tea like I $$ P(tea\mid\langle s\rangle)=0/3=0 $$ ##### Step 3. Apply the zero factor The full product is zero, regardless of the remaining factors. **Final answer.** P(you like coffee) = 1/9; P(tea like I) = 0. A new sentence is not necessarily a zero-probability sentence. **Question. Obtain the unsmoothed bigram probability of “students are from Vellore” from this corpus: “I am from Vellore”; “I am a teacher”; “students are good and are from various cities”; “students from Vellore do engineering”. Use sentence boundaries.** #### Given Four training sentences, each padded with one and . Test sequence: students are from Vellore . #### To find The five required conditional probabilities and their product. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Count all occurrences in the entire corpus. In particular, are occurs twice inside the third sentence, and Vellore is followed by end once and do once. #### Solution ##### Step 1. Count the histories | History | Occurrences with continuation | | --- | --- | | | 4 | | students | 2 | | are | 2 | | from | 3 | | Vellore | 2 | ##### Step 2. Count the required pairs | Required pair | Count | Probability | | --- | --- | --- | | students | 2 | 2/4 | | students are | 1 | 1/2 | | are from | 1 | 1/2 | | from Vellore | 2 | 2/3 | | Vellore | 1 | 1/2 | ##### Step 3. Write the complete product $$ P=\frac24\times\frac12\times\frac12\times\frac23\times\frac12 $$ ##### Step 4. Multiply progressively $$ \frac24\times\frac12=\frac14,\quad \frac14\times\frac12=\frac18 $$ $$ \frac18\times\frac23=\frac1{12},\quad \frac1{12}\times\frac12=\frac1{24} $$ **Final answer.** P = 1/24 ≈ 0.0416667. **Yaad rakho:** Same sentence mein word do baar aaye toh count bhi do. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Trigram estimates: two-word history, not two denominators Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-count-to-probability--trigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-count-to-probability/trigram-estimation Unit 3, pages 97–120 and appendix A trigram model predicts the next word from the two preceding words. The numerator is the count of a triple. The denominator is the count of its two-word history with an available next token. Do not multiply two independent bigram probabilities and call that a trigram probability. $$ P(w_i\mid w_{i-2},w_{i-1})=\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})} $$ **Question. A corpus contains 12 occurrences of “prime minister”. Eight are followed by “of”, three by “said”, and one by “visited”. Estimate the three trigram probabilities.** #### Given History h = (prime, minister). Count of this history = 12. Triple counts for of, said, visited are 8, 3, 1. #### To find The conditional distribution of the next word. #### Formula and why it applies $$ P(w\mid prime,minister)=C(prime,minister,w)/12 $$ All three candidates share one two-word history, so they share denominator 12. #### Solution ##### Step 1. Calculate of $$ P(of\mid prime,minister)=8/12=2/3 $$ ##### Step 2. Calculate said $$ P(said\mid prime,minister)=3/12=1/4 $$ ##### Step 3. Calculate visited $$ P(visited\mid prime,minister)=1/12 $$ ##### Step 4. Confirm normalization $$ 8/12+3/12+1/12=12/12=1 $$ **Final answer.** The probabilities are 2/3, 1/4 and 1/12. **Question. Train a trigram model on “I like tea”, “I like coffee” and “you like tea”. Use two start markers and one end marker. Score “I like tea”.** #### Given Three sentences. Padded first sentence is I like tea ; pad the other two in the same way. #### To find Four trigram prediction probabilities and the sentence probability. #### Formula and why it applies $$ P=\prod_i P(w_i\mid w_{i-2},w_{i-1}) $$ Each prediction uses a two-token context; the two start markers supply the first history. #### Solution ##### Step 1. Count the first-word event The history (,) occurs three times, and two are followed by I. $$ P(I\mid\langle s\rangle,\langle s\rangle)=2/3 $$ ##### Step 2. Count the second-word event (,I) occurs twice; both are followed by like. $$ P(like\mid\langle s\rangle,I)=2/2=1 $$ ##### Step 3. Count the third-word event (I,like) occurs twice; one is followed by tea. $$ P(tea\mid I,like)=1/2 $$ ##### Step 4. Count the end event (like,tea) occurs twice; both are followed by end. $$ P(\langle/s\rangle\mid like,tea)=2/2=1 $$ ##### Step 5. Multiply all four $$ P=\frac23\times1\times\frac12\times1=\frac13 $$ **Final answer.** Trigram P(I like tea) = 1/3. It differs from the bigram estimate because the context and denominator changed. **Question. A trigram model supplies P(we|,)=0.4, P(study|,we)=0.5, P(NLP|we,study)=0.8 and P(|study,NLP)=0.25. Score “we study NLP”.** #### Given Four trigram conditional factors. The end marker is included. #### To find The complete sentence probability. #### Formula and why it applies $$ P=P(we\mid s,s)P(study\mid s,we)P(NLP\mid we,study)P(end\mid study,NLP) $$ Use each supplied two-word history exactly; do not swap its word order. #### Solution ##### Step 1. Insert the values $$ P=0.4\times0.5\times0.8\times0.25 $$ ##### Step 2. Multiply the first pair $$ 0.4\times0.5=0.2 $$ ##### Step 3. Include NLP $$ 0.2\times0.8=0.16 $$ ##### Step 4. Include the end $$ 0.16\times0.25=0.04 $$ **Final answer.** P = 0.04. **Yaad rakho:** Trigram: teen-word count upar, pehle do-word count neeche. **Why larger n is not automatically better.** More context can capture local order more precisely, but each longer history occurs less often. Parameter count, storage and sparsity grow. A model trained on Shakespeare may assign poor probabilities to financial news because training and test distributions differ. Keep a held-out test set; do not evaluate only on memorized training sentences. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Class-style count-table questions and partial-table traps Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-count-to-probability--class-count-table-questions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-count-to-probability/class-count-table-questions Unit 3, pages 97–120 and appendix **Question. Train a bigram model on “I am Sam”, “Sam I am”, and “I do not like green eggs and ham”. Include sentence boundaries. Find P(I|), P(Sam|), P(am|I), P(do|I), P(Sam|am) and P(|Sam).** #### Given Three sentences; one and per sentence; preserve capital I and Sam. #### To find Six bigram estimates. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Count all matching histories, including repeated words in different sentence positions. #### Solution ##### Step 1. Count the histories START occurs 3 times; I occurs 3 times; am occurs 2 times; Sam occurs 2 times. ##### Step 2. Count the requested pairs | Pair | Pair count | History count | | --- | --- | --- | | START I | 2 | 3 | | START Sam | 1 | 3 | | I am | 2 | 3 | | I do | 1 | 3 | | am Sam | 1 | 2 | | Sam END | 1 | 2 | ##### Step 3. Divide pair by history | Requested probability | Result | | --- | --- | | P(I|START) | 2/3 | | P(Sam|START) | 1/3 | | P(am|I) | 2/3 | | P(do|I) | 1/3 | | P(Sam|am) | 1/2 | | P(END|Sam) | 1/2 | **Final answer.** The six estimates are 2/3,1/3,2/3,1/3,1/2,1/2 in the requested order. **Question. A bigram table gives P(i|)=0.25, P(want|i)=0.33, P(english|want)=0.0011, P(food|english)=0.5 and P(|food)=0.68. Score “i want english food” using these supplied rounded values.** #### Given All five transition probabilities are explicitly given. Include the end marker. #### To find The sentence probability from the supplied table. #### Formula and why it applies $$ P=P(i\mid START)P(want\mid i)P(english\mid want)P(food\mid english)P(END\mid food) $$ A table-based question uses the provided values; slight differences can arise if exact corpus fractions are used instead. #### Solution ##### Step 1. First two events $$ 0.25(0.33)=0.0825 $$ ##### Step 2. Add english $$ 0.0825(0.0011)=0.00009075 $$ ##### Step 3. Add food $$ 0.00009075(0.5)=0.000045375 $$ ##### Step 4. End the sentence $$ 0.000045375(0.68)=0.000030855 $$ **Final answer.** P≈0.000030855 using the supplied rounded probabilities. **Question. A corpus has vocabulary size 1446, C(i)=2533 and C(i,want)=827. Compute the unsmoothed and add-one-smoothed P(want|i). The displayed bigram table contains only eight vocabulary columns.** #### Given Full history count C(i)=2533; pair count 827; complete vocabulary size V=1446. The displayed table is only a subset. #### To find Both conditional probabilities. #### Formula and why it applies $$ P(want\mid i)=827/2533,\qquad P_L(want\mid i)=\frac{827+1}{2533+1446} $$ Use the supplied full history total. Summing only the eight displayed columns would wrongly ignore all other possible continuations. #### Solution ##### Step 1. Calculate the unsmoothed ratio $$ P=827/2533\approx0.326490 $$ ##### Step 2. Add one to the pair count $$ 827+1=828 $$ ##### Step 3. Add V to the history count $$ 2533+1446=3979 $$ ##### Step 4. Divide $$ P_L=828/3979\approx0.208092 $$ **Final answer.** Unsmoothed≈0.326490; smoothed≈0.208092. **Yaad rakho:** Partial table ka visible row-sum, full history total nahi hota. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Perplexity: meaning, formula and four complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-evaluation-smoothing--evaluation-perplexity/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-evaluation-smoothing/evaluation-perplexity Unit 3, pages 121–156 **Intrinsic evaluation** evaluates a model directly on a held-out test corpus, for example using perplexity. **Extrinsic evaluation** places it inside a downstream application, such as a speech recognizer or translation system, and measures the application's performance. Extrinsic testing is closer to actual use but often slower and more expensive. A lower perplexity is not a guarantee of improvement on every downstream task. **Perplexity** is the inverse probability of a test sequence, normalized by the number of prediction events. Intuitively, it is the model's effective average number of equally plausible choices. A model that always gives the correct next word probability 1/4 has perplexity 4. Lower is better when the models are tested on the same data, vocabulary, tokenization and boundary convention. $$ \operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}} $$ | Symbol | Meaning | | --- | --- | | W | The complete test sequence. | | P(W) | Product of the probabilities assigned to the actual next tokens. | | N | Number of scored prediction events under the stated convention. | | Exponent −1/N | Take the reciprocal, then the Nth root. | For calculator entry, use parentheses: P raised to (−1 ÷ N), or (1 ÷ P) raised to (1 ÷ N). Retain several digits in intermediate products. The ordinary start marker is usually given context, not a predicted event; the end marker counts if its probability is included. **Question. A model assigns probability 0.0625 to a test sequence containing four scored tokens. Compute its perplexity.** #### Given P(W)=0.0625=1/16; N=4. #### To find PP(W). #### Formula and why it applies $$ \operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N} $$ The root removes the effect of sequence length; the reciprocal makes lower surprise correspond to lower perplexity. #### Solution ##### Step 1. Take the reciprocal $$ 1/P(W)=1/0.0625=16 $$ ##### Step 2. Use the fourth root $$ \operatorname{PP}=16^{1/4} $$ ##### Step 3. Evaluate $$ 2^4=16\quad\Rightarrow\quad\operatorname{PP}=2 $$ **Final answer.** Perplexity = 2. **Question. Two language models score the same three-token test sequence. Model A gives next-token probabilities 0.5, 0.5, 0.5. Model B gives 0.5, 0.25, 0.25. Compute their perplexities and select the better intrinsic result.** #### Given Same sequence and N=3 for both models. A factors: 0.5,0.5,0.5. B factors: 0.5,0.25,0.25. #### To find Both sequence probabilities and perplexities. #### Formula and why it applies $$ P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3} $$ A fair comparison uses identical test tokens and scoring conventions. #### Solution ##### Step 1. Calculate A's probability $$ P_A=0.5\times0.5\times0.5=0.125=1/8 $$ ##### Step 2. Calculate A's perplexity $$ \operatorname{PP}_A=8^{1/3}=2 $$ ##### Step 3. Calculate B's probability $$ P_B=0.5\times0.25\times0.25=0.03125=1/32 $$ ##### Step 4. Calculate B's perplexity $$ \operatorname{PP}_B=32^{1/3}\approx3.174802 $$ ##### Step 5. Compare 2 is lower than 3.174802; A places more probability on the observed sequence. **Final answer.** Model A is better by perplexity on this test: 2 versus about 3.174802. **Question. The bigram training corpus is “This is a dog”, “This is a cat”, “I love my cat”, “This is my name”. Score “This is my cat”, including an end marker, and compute perplexity using five predicted events.** #### Given Each training sentence has one start and end marker. Test predictions: This, is, my, cat, . Thus N=5. #### To find Bigram sentence probability and perplexity. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5} $$ Start is given context. Four words plus the end marker produce five probability factors. #### Solution ##### Step 1. Count the required transitions | Transition | Pair count / history count | | --- | --- | | →This | 3/4 | | This→is | 3/3 | | is→my | 1/3 | | my→cat | 1/2 | | cat→ | 2/2 | ##### Step 2. Form the product $$ P=\frac34\times1\times\frac13\times\frac12\times1=\frac18 $$ ##### Step 3. Take the reciprocal $$ 1/P=8 $$ ##### Step 4. Take the fifth root $$ \operatorname{PP}=8^{1/5}\approx1.515717 $$ **Final answer.** P(W)=1/8 and perplexity≈1.515717 using five predicted events. **If the question explicitly states N = 6:** use that supplied convention and report PP = 81/6 ≈ 1.414214. The class slide counts both boundary markers in N. Write the chosen N beside the formula; do not silently switch between 5 and 6. Also, This occurs three times and is follows it three times, so P(is|This)=3/3, not 3/4. **Question. A uniform model assigns probability 1/10 to each actual next word in a five-token test. Find its perplexity.** #### Given Five scored tokens, each with assigned probability 0.1. N=5. #### To find The sentence probability and perplexity. #### Formula and why it applies $$ P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5} $$ Uniform equal probabilities make the effective number of choices especially clear. #### Solution ##### Step 1. Multiply the five factors $$ P(W)=0.1^5=0.00001=10^{-5} $$ ##### Step 2. Invert $$ 1/P(W)=10^5 $$ ##### Step 3. Take the fifth root $$ \operatorname{PP}=(10^5)^{1/5}=10 $$ **Final answer.** Perplexity = 10. **Yaad rakho:** Probability high → surprise low → perplexity low. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Log probabilities: replace long products with sums Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-evaluation-smoothing--log-probabilities/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-evaluation-smoothing/log-probabilities Unit 3, pages 121–156 Probabilities are at most 1, so multiplying many of them produces very small numbers. Computers can round extremely small positive products to zero: **numerical underflow**. Log probabilities convert multiplication into addition and are more stable for long sequences. Natural logs use base e; log₂ uses base 2. Choose one base consistently. $$ \ln P(W)=\sum_{i=1}^{N}\ln p_i $$ $$ P(W)=e^{\ln P(W)},\qquad \operatorname{PP}(W)=\exp\left(-\frac1N\sum_i\ln p_i\right) $$ The log of a probability below 1 is negative. A value closer to zero means a larger probability: ln(0.5) ≈ −0.693 is larger than ln(0.1) ≈ −2.303. A zero probability has log −∞; taking logs does not solve the unseen-event problem. Smoothing handles that separate problem. **Question. The next-word probabilities are 0.5, 0.25 and 0.5. Calculate the natural log probability, ordinary probability and perplexity.** #### Given p₁=0.5; p₂=0.25; p₃=0.5; N=3. #### To find ln P(W), P(W), and PP. #### Formula and why it applies $$ \ln P=\sum_i\ln p_i,\quad P=e^{\ln P},\quad\operatorname{PP}=e^{-\ln P/N} $$ Logarithms preserve products as sums and the exponent converts the result back. #### Solution ##### Step 1. Compute individual logs $$ \ln0.5\approx-0.693147,\quad\ln0.25\approx-1.386294 $$ ##### Step 2. Add all three terms $$ \ln P=-0.693147-1.386294-0.693147\approx-2.772589 $$ ##### Step 3. Recover the probability $$ P=e^{-2.772589}\approx0.0625 $$ ##### Step 4. Find average negative log probability $$ -\ln P/3\approx2.772589/3=0.924196 $$ ##### Step 5. Exponentiate $$ \operatorname{PP}=e^{0.924196}\approx2.519842 $$ **Final answer.** ln P≈−2.772589, P=0.0625, PP≈2.519842. **Question. A model has a total natural log probability −12 over six scored tokens. Calculate its perplexity.** #### Given ln P(W)=−12; N=6. #### To find PP without forming the tiny ordinary probability. #### Formula and why it applies $$ \operatorname{PP}=\exp(-\ln P(W)/N) $$ The logarithmic formula directly uses the supplied log score. #### Solution ##### Step 1. Negate the log score $$ -\ln P=12 $$ ##### Step 2. Divide by token count $$ 12/6=2 $$ ##### Step 3. Apply exponential $$ \operatorname{PP}=e^2\approx7.389056 $$ **Final answer.** Perplexity≈7.389056. **Question. A language model's average negative log₂ probability is 3 bits per token. Find its perplexity.** #### Given Cross-entropy H=3 bits/token. Log base is 2. #### To find PP. #### Formula and why it applies $$ \operatorname{PP}=2^H $$ The inverse operation of log₂ is raising 2 to that power; use e only for natural logs. #### Solution ##### Step 1. Identify the log base Bits mean base 2. ##### Step 2. Raise the base to H $$ \operatorname{PP}=2^3=8 $$ **Final answer.** Perplexity = 8. **Yaad rakho:** ln use kiya toh e se wapas; log₂ use kiya toh 2 se wapas. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Laplace smoothing: why both numerator and denominator change Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-evaluation-smoothing--laplace-basics/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-evaluation-smoothing/laplace-basics Unit 3, pages 121–156 **Smoothing** reallocates some probability mass to events that received zero or unreliable estimates from finite training data. **Laplace, or add-one, smoothing** adds one pseudo-count to every possible next outcome. If there are V possible outcomes, the total added count is V, so adding 1 only to the numerator would produce probabilities that do not sum to 1. $$ P_{\mathrm{Laplace}}(w)=\frac{C(w)+1}{N+V} $$ $$ P_{\mathrm{Laplace}}(w\mid h)=\frac{C(h,w)+1}{C(h)+V} $$ For unigram smoothing, N is the total training token count. For bigram smoothing, C(h) is the count of the particular history. In both cases V is the number of outcomes being predicted, not the number of training sentences and not the number of distinct bigrams. When end is a possible prediction, include it in V; the start marker is usually context only. Observed counts: 3, 1, 0 → Add one to every outcome: 4, 2, 1 → New total: 7 → Smoothed probabilities: 4/7, 2/7, 1/7 **Question. A fixed unigram vocabulary is {a,b,c}. The training counts are 3,1,0. Compute unsmoothed and Laplace-smoothed probabilities.** #### Given C(a)=3, C(b)=1, C(c)=0; N=4; V=3. #### To find The complete distributions. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P_L(w)=(C(w)+1)/(N+V) $$ The zero-count c is still in the fixed vocabulary, so it must receive one pseudo-count and contribute to V. #### Solution ##### Step 1. Compute unsmoothed values | Word | Unsmoothed | | --- | --- | | a | 3/4 = 0.75 | | b | 1/4 = 0.25 | | c | 0/4 = 0 | ##### Step 2. Calculate the new denominator $$ N+V=4+3=7 $$ ##### Step 3. Add one to each numerator $$ C(a)+1=4,\quad C(b)+1=2,\quad C(c)+1=1 $$ ##### Step 4. Divide | Word | Smoothed | | --- | --- | | a | 4/7 ≈ 0.571429 | | b | 2/7 ≈ 0.285714 | | c | 1/7 ≈ 0.142857 | ##### Step 5. Confirm the total $$ 4/7+2/7+1/7=1 $$ **Final answer.** Smoothed probabilities: a=4/7, b=2/7, c=1/7. Common a loses mass; unseen c gains mass. **Question. After history I, a corpus records like three times, study once and sleep zero times. The allowed next vocabulary is {like,study,sleep}. Compute all Laplace-smoothed bigram probabilities.** #### Given C(I,like)=3; C(I,study)=1; C(I,sleep)=0. History count C(I)=4. V=3. #### To find P\_L(like|I), P\_L(study|I), P\_L(sleep|I). #### Formula and why it applies $$ P_L(w\mid I)=\frac{C(I,w)+1}{C(I)+V} $$ This is smoothing within the I row; use its total 4, not the global corpus token count. #### Solution ##### Step 1. Form the common denominator $$ C(I)+V=4+3=7 $$ ##### Step 2. Calculate like $$ P_L(like\mid I)=(3+1)/7=4/7 $$ ##### Step 3. Calculate study $$ P_L(study\mid I)=(1+1)/7=2/7 $$ ##### Step 4. Calculate sleep $$ P_L(sleep\mid I)=(0+1)/7=1/7 $$ **Final answer.** The row is (4/7, 2/7, 1/7). An unseen bigram is no longer forced to zero. **Question. A fixed vocabulary has five possible next tokens. A history h is unseen in training. What is its add-one probability for each next token?** #### Given C(h)=0; C(h,w)=0 for each outcome; V=5. #### To find P\_L(w|h) for each token. #### Formula and why it applies $$ P_L(w\mid h)=\frac{0+1}{0+V} $$ All outcomes receive equal pseudo-counts because the training data gives no preference for this history. #### Solution ##### Step 1. Add the pseudo-count $$ 0+1=1 $$ ##### Step 2. Form the denominator $$ 0+5=5 $$ ##### Step 3. Divide and sum $$ P_L(w\mid h)=1/5=0.2,\qquad5(0.2)=1 $$ **Final answer.** Every outcome gets 0.2 under this fixed-vocabulary model. **Vocabulary warning.** Add-one does not automatically define probabilities for every word in the world. Unknown words require a declared vocabulary/UNK convention. If a question supplies V, use that stated V rather than inventing additional tokens. **Yaad rakho:** Add-one: upar +1, neeche +V; bigram mein neeche history ka count. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · A complete unseen-bigram question from corpus to final answer Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-evaluation-smoothing--smoothed-sentence/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-evaluation-smoothing/smoothed-sentence Unit 3, pages 121–156 **Question. Consider the corpus “I love to code in Python. I love to learn new things. I code every day.” Obtain the bigram probabilities needed for “I love Python”. Explain the zero-probability issue, then use Laplace smoothing to score it. Treat each sentence separately, preserve case, remove punctuation, and include an end token among predicted outcomes.** #### Given Three sentences padded with one start/end marker. Distinct words: I, love, to, code, in, Python, learn, new, things, every, day. With , V=12. Start is context only. #### To find Unsmoothed and smoothed P(I love Python | ). #### Formula and why it applies $$ P(w\mid h)=\frac{C(h,w)}{C(h)},\qquad P_L(w\mid h)=\frac{C(h,w)+1}{C(h)+12} $$ The test contains an unseen transition love→Python. Smoothing assigns probability to that transition and renormalizes every required history row. #### Solution ##### Step 1. Count the contexts | History | Count | | --- | --- | | | 3 | | I | 3 | | love | 2 | | Python | 1 | ##### Step 2. Count each required pair | Pair | Count | | --- | --- | | I | 3 | | I love | 2 | | love Python | 0 | | Python | 1 | ##### Step 3. Compute the unsmoothed product $$ P=\frac33\times\frac23\times\frac02\times\frac11=0 $$ Both words love and Python were seen, but their adjacent pair was not. This is an unseen bigram, not an unknown-word case. ##### Step 4. Smooth the first event $$ P_L(I\mid\langle s\rangle)=\frac{3+1}{3+12}=\frac4{15} $$ ##### Step 5. Smooth the second event $$ P_L(love\mid I)=\frac{2+1}{3+12}=\frac3{15} $$ ##### Step 6. Smooth the unseen event $$ P_L(Python\mid love)=\frac{0+1}{2+12}=\frac1{14} $$ ##### Step 7. Smooth the end event $$ P_L(\langle/s\rangle\mid Python)=\frac{1+1}{1+12}=\frac2{13} $$ ##### Step 8. Multiply the smoothed factors $$ P_L=\frac4{15}\times\frac3{15}\times\frac1{14}\times\frac2{13} $$ $$ P_L=\frac{24}{40950}=\frac4{6825}\approx0.000586081 $$ **Final answer.** Unsmoothed P=0. With the stated V=12 and boundaries, Laplace-smoothed P≈0.000586081. If an exam states a different convention—for example excluding boundary tokens from the entire calculation—recompute both V and the set of probability factors consistently. Do not combine a boundary-aware numerator with a boundary-free denominator. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Adjusted counts and relative discounting Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-evaluation-smoothing--adjusted-count-discount/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-evaluation-smoothing/adjusted-count-discount Unit 3, pages 121–156 The slides also express smoothing through an **adjusted count** c\*. Instead of writing the smoothed probability with denominator N+V, they rescale its numerator so the denominator can remain N. Algebraically these are equivalent: $$ c^*=(c+1)\frac{N}{N+V},\qquad P_L(w)=\frac{c^*}{N} $$ $$ d=\frac{c^*}{c}\quad(c>0) $$ The relative discount d compares adjusted and original counts. It is undefined when c=0 because division by zero is undefined. Add-one smoothing moves the distribution toward uniformity: frequent events are reduced, but some rare seen events may increase. It is not correct to claim that every nonzero count must decrease under Laplace smoothing. **Question. For N=20 and V=5, a word has original count c=10. Find its adjusted count, smoothed probability and relative discount.** #### Given N=20; V=5; c=10. #### To find c\*, P\_L and d. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\quad P_L=c^*/N,\quad d=c^*/c $$ These expressions rewrite add-one smoothing without changing its probability. #### Solution ##### Step 1. Form the new total $$ N+V=20+5=25 $$ ##### Step 2. Find the adjusted count $$ c^*=(10+1)\frac{20}{25}=11(0.8)=8.8 $$ ##### Step 3. Find the probability $$ P_L=8.8/20=0.44 $$ ##### Step 4. Find the relative discount $$ d=8.8/10=0.88 $$ **Final answer.** c\*=8.8, P\_L=0.44, d=0.88. The original probability 0.5 is reduced. **Question. Under N=20, V=5, a rare word occurs once. Compute its adjusted count and compare the original and smoothed probabilities.** #### Given N=20; V=5; c=1. #### To find c\*, original P and smoothed P\_L. #### Formula and why it applies $$ c^*=(c+1)N/(N+V) $$ The same smoothing rule may increase a rare event's probability. #### Solution ##### Step 1. Original probability $$ P=1/20=0.05 $$ ##### Step 2. Adjusted count $$ c^*=(1+1)(20/25)=1.6 $$ ##### Step 3. Smoothed probability $$ P_L=1.6/20=0.08 $$ **Final answer.** c\*=1.6 and P\_L=0.08, which exceeds the original 0.05. **Question. For N=20 and V=5, an in-vocabulary word has count zero. Find its adjusted count and smoothed probability. Can a relative discount be calculated?** #### Given c=0; N=20; V=5. #### To find c\*, P\_L and whether d exists. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\qquad P_L=c^*/N $$ The pseudo-count creates positive mass for the unseen vocabulary item. #### Solution ##### Step 1. Adjusted count $$ c^*=(0+1)(20/25)=0.8 $$ ##### Step 2. Smoothed probability $$ P_L=0.8/20=0.04 $$ ##### Step 3. Relative discount d would be 0.8/0. That is undefined, so do not report a finite discount. **Final answer.** c\*=0.8 and P\_L=0.04; relative discount is undefined for c=0. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Character-model perplexity: keep exact probabilities Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-evaluation-smoothing--class-perplexity-characters/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-evaluation-smoothing/class-perplexity-characters Unit 3, pages 121–156 **Question. A unigram character model gives P(A)=0.25, P(B)=0.50, P(C)=0.25. Find the perplexity of AAA and ABC. Then find both perplexities when the three characters are equally likely.** #### Given Each test string contains 3 scored characters, so N=3. Initial distribution 0.25,0.50,0.25; comparison distribution 1/3,1/3,1/3. #### To find Both sequence probabilities and perplexities under each distribution. #### Formula and why it applies $$ P(c_1c_2c_3)=P(c_1)P(c_2)P(c_3),\qquad PP=P(W)^{-1/3} $$ Characters, not words, are the model's prediction units in this question. Use exact 1/3 for uniform probabilities, rather than rounded 0.33. #### Solution ##### Step 1. Score AAA $$ P(AAA)=0.25^3=\frac1{64}=0.015625 $$ ##### Step 2. Find AAA perplexity $$ PP(AAA)=64^{1/3}=4 $$ ##### Step 3. Score ABC $$ P(ABC)=0.25(0.5)(0.25)=\frac1{32}=0.03125 $$ ##### Step 4. Find ABC perplexity $$ PP(ABC)=32^{1/3}\approx3.174802 $$ ##### Step 5. Change to an exactly uniform model $$ P(AAA)=P(ABC)=(1/3)^3=1/27 $$ ##### Step 6. Find both uniform-model perplexities $$ PP=27^{1/3}=3 $$ **Final answer.** Original model: AAA→4, ABC→3.174802. Uniform model: both→3. These values follow the stated formulas without intermediate rounding. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Parts of speech and contextual ambiguity Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-pos-tagging--word-classes-pos/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-pos-tagging/word-classes-pos Unit 4, pages 3–46 **Part-of-speech (POS) tagging** assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence. | Tag | Role | Example in context | | --- | --- | --- | | NN / NNS | Singular / plural common noun | a book / two books | | NNP | Proper noun | Mary, Mumbai | | VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads | | JJ | Adjective | the red book | | RB | Adverb | runs quickly | | DT | Determiner | the, a, this | | IN | Preposition or subordinating conjunction | on the table | | PRP | Personal pronoun | she, they | | MD | Modal auxiliary | can, will, should | **Open classes** readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. **Closed classes** contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes. Input: Book that flight → Context: an imperative request → Output: Book/VB that/DT flight/NN Compare “hand me that **book**”: book is a noun here. Similarly, “I **can** swim” has can/MD, while “the **can** is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is **POS ambiguity**. | Sentence | Tagged interpretation | Reason | | --- | --- | --- | | They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. | | The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. | | She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. | | Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. | POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning. **Yaad rakho:** Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai. ### NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule id : nlp-m2-07-pos-tags-tagger-methods title : NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule title\_confirmed : true topic\_title : POS tags, ambiguity and tagging methods mode : m2 kind : focused status : available duration : 9:06 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Tagger methods, Brill net gain and the full conditional rule; clarified tagging and imperative explanations. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Lexical, rule-based, stochastic, Brill and neural taggers Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-pos-tagging--tagger-methods/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-pos-tagging/tagger-methods Unit 4, pages 3–46 **A lexical baseline** gives each known word its most frequent training tag. It is simple and often useful, but ignores context. A **rule-based tagger** uses a lexicon of possible tags plus manually designed contextual or morphological rules to resolve ambiguity. For example, after a determiner, a noun reading of “can” may be preferred in “the can”. Rules are usually heuristics with exceptions. “A red car” shows that the word immediately after an article is not always a noun. The suffix -ed does not make “red” a past-tense verb. A robust rule system uses multiple linguistic constraints instead of treating such shortcuts as universal laws. | Method | How it chooses tags | Strength | Limitation | | --- | --- | --- | --- | | Most frequent / lexical | Most common training tag for the word | Simple baseline | Misses contextual changes and unknown words | | Rule-based | Lexicon plus handwritten rules | Readable, controllable linguistic decisions | Expert effort; rules may conflict or need updates | | Stochastic | Scores tag sequences using estimated probabilities | Uses corpus evidence and context | Sparse/unseen events; depends on training domain | | Transformation-based / Brill | Start with baseline, then apply learned correction rules | Readable learned corrections | Needs tagged training data and suitable rule templates | | Neural | Learns contextual representations and sequence predictions | Can exploit richer context | Needs training resources and data; less directly interpretable | **Transformation-based error-driven learning (Brill tagging)** combines a simple starting assignment with learned correction rules. First tag the training corpus with a baseline. Compare against the gold tags. Generate candidate correction rules from templates, such as “change MD to NN after DT”. Apply the rule that fixes the most errors net of newly introduced errors; repeat until further improvement is too small. At test time apply the learned ordered rule list. Baseline: The/DT can/MD was/VBD crushed/VBN → Context rule: change can from MD to NN after DT → Result: The/DT can/NN was/VBD crushed/VBN **Rule-based versus Brill:** a purely rule-based system depends on linguists specifying its rules; a Brill tagger learns which transformations improve a labeled corpus. A **stochastic tagger** such as an HMM finds a high-probability tag sequence using transition and emission probabilities. CRFs and neural sequence taggers are other learning approaches; they do not have to make the same HMM independence assumptions. **Exam-ready answer.** POS tagging assigns a grammatical tag to each word in context. A rule-based tagger uses lexicons and linguistic rules, whereas a stochastic tagger chooses tags through probabilities estimated from a corpus. For example, can is MD in “I can swim” but NN in “the can”; context resolves the ambiguity. ### NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule id : nlp-m2-07-pos-tags-tagger-methods title : NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule title\_confirmed : true topic\_title : POS tags, ambiguity and tagging methods mode : m2 kind : focused status : available duration : 9:06 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Tagger methods, Brill net gain and the full conditional rule; clarified tagging and imperative explanations. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Generative versus discriminative models and Bayes' rule Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-pos-tagging--generative-discriminative/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-pos-tagging/generative-discriminative Unit 4, pages 3–46 Let x be the observed words and y be the desired tag sequence. A **generative model** describes their joint distribution P(x,y), often as P(y)P(x|y). A **discriminative model** learns P(y|x) directly or learns a decision boundary that separates possible labels. Both kinds can be trained with labeled data; “generative” does not mean “always unsupervised”. $$ P(y\mid x)=\frac{P(x\mid y)P(y)}{P(x)} $$ | Quantity | Meaning | POS interpretation | | --- | --- | --- | | P(y) | Prior | How plausible is this tag or tag sequence before seeing the words? | | P(x|y) | Likelihood | How plausible are these words under that tag sequence? | | P(x) | Evidence | Total probability of the observed words across possible labels. | | P(y|x) | Posterior | Probability of the tags after seeing the words. | | Aspect | Generative | Discriminative | | --- | --- | --- | | Models | Joint P(x,y), or prior × likelihood | Conditional P(y|x) or decision boundary | | Classification examples | Naive Bayes | Logistic regression, SVM | | Sequence-labeling examples | HMM | Linear-chain CRF | | Intuition | Model how a label could generate the observation | Model which label fits the observation | **Question. A token has two possible tags, noun N and verb V. P(N)=0.6, P(V)=0.4, P(book|N)=0.2 and P(book|V)=0.5. Calculate P(N|book) and P(V|book).** #### Given Two tags are exhaustive. Priors: 0.6,0.4. Word likelihoods: 0.2,0.5. #### To find Normalized posterior probabilities. #### Formula and why it applies $$ P(t\mid w)=\frac{P(w\mid t)P(t)}{\sum_{u}P(w\mid u)P(u)} $$ Bayes' rule combines prior and likelihood. Normalization is needed to convert scores into posterior probabilities. #### Solution ##### Step 1. Calculate the noun score $$ 0.2\times0.6=0.12 $$ ##### Step 2. Calculate the verb score $$ 0.5\times0.4=0.20 $$ ##### Step 3. Find the evidence $$ P(book)=0.12+0.20=0.32 $$ ##### Step 4. Normalize the noun score $$ P(N\mid book)=0.12/0.32=0.375 $$ ##### Step 5. Normalize the verb score $$ P(V\mid book)=0.20/0.32=0.625 $$ **Final answer.** Verb is more probable in this simplified model: 0.625 versus 0.375. **Question. Two candidate tags have prior-likelihood products 0.06 and 0.14. Find their normalized probabilities.** #### Given Unnormalized scores s₁=0.06 and s₂=0.14; these are the only candidates. #### To find The two posterior probabilities. #### Formula and why it applies $$ P_i=s_i/(s_1+s_2) $$ A common denominator converts the candidate scores to a distribution summing to one. #### Solution ##### Step 1. Add scores $$ 0.06+0.14=0.20 $$ ##### Step 2. Normalize first $$ 0.06/0.20=0.3 $$ ##### Step 3. Normalize second $$ 0.14/0.20=0.7 $$ **Final answer.** Posterior probabilities are 0.3 and 0.7. **Yaad rakho:** Generative: tag se word ka model. Discriminative: word se tag ka decision. ### Bayes Rule & Models id : nlp-m2-02-bayes-generative-discriminative title : Bayes Rule & Models title\_confirmed : true topic\_title : Bayes rule and generative versus discriminative models mode : m2 kind : focused status : available duration : 10:51 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Markov chains: read a transition table and multiply a path Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm--markov-chain/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm/markov-chain Unit 4, pages 47–73 **A Markov chain** is a probabilistic model of a sequence of states. Under the first-order Markov assumption, the next state's probability depends on the current state rather than the complete previous history. The **initial probability** tells us how likely each first state is. A **transition probability** tells us how likely a move from one state to another is. $$ P(s_t\mid s_1,\ldots,s_{t-1})=P(s_t\mid s_{t-1}) $$ $$ P(s_1,\ldots,s_T)=P(s_1)\prod_{t=2}^{T}P(s_t\mid s_{t-1}) $$ For T states in a path, use one initial factor and T−1 transition factors. A row-oriented transition matrix must have each complete row sum to 1. Always read the row/column labels; some exam diagrams use a transposed arrangement. | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. **Question. For the Rain/Dry Markov chain with initial P(Rain)=0.4 and P(Dry)=0.6, and transitions Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8, find the probability of Dry, Dry, Rain, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Dry,Dry,Rain,Rain). #### Formula and why it applies $$ P(D,D,R,R)=P(D)P(D\mid D)P(R\mid D)P(R\mid R) $$ Each step conditions on the immediately preceding state. The sequence contains four states, so it needs one initial plus three transition factors. #### Solution ##### Step 1. Start in Dry $$ P(D)=0.6 $$ ##### Step 2. Move Dry to Dry $$ 0.6\times0.8=0.48 $$ ##### Step 3. Move Dry to Rain $$ 0.48\times0.2=0.096 $$ ##### Step 4. Move Rain to Rain $$ 0.096\times0.3=0.0288 $$ **Final answer.** Path probability = 0.0288. **Question. For initial probabilities Rain=0.4, Dry=0.6 and transition rows Rain=(0.3,0.7), Dry=(0.2,0.8) in column order Rain,Dry, find the probability of Rain, Dry, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Rain,Dry,Rain). #### Formula and why it applies $$ P(R,D,R)=P(R)P(D\mid R)P(R\mid D) $$ Do not replace Dry→Rain with Rain→Dry: the two directional probabilities differ. #### Solution ##### Step 1. Initial Rain $$ P(R)=0.4 $$ ##### Step 2. First transition $$ P(D\mid R)=0.7,\quad0.4(0.7)=0.28 $$ ##### Step 3. Second transition $$ P(R\mid D)=0.2,\quad0.28(0.2)=0.056 $$ **Final answer.** Path probability = 0.056. **Question. For the same fully specified chain—initial Rain=0.4, Dry=0.6; Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8—find P(Dry,Dry). Also find the conditional probability of next Dry given current Dry.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find The two-state joint probability and the one-step conditional probability. #### Formula and why it applies $$ P(D,D)=P(D)P(D\mid D) $$ The joint path includes an uncertain initial state. The conditional transition assumes the current state is already Dry. #### Solution ##### Step 1. Read the conditional transition $$ P(D\mid D)=0.8 $$ ##### Step 2. Include the initial state for the joint path $$ P(D,D)=0.6\times0.8=0.48 $$ **Final answer.** P(Dry,Dry)=0.48; P(next Dry|current Dry)=0.8. These answer different questions. **Yaad rakho:** Path: pehle start probability, phir har arrow ki probability. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Hidden states, observations, transitions and emissions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm--hmm-components/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm/hmm-components Unit 4, pages 47–73 **A Hidden Markov Model (HMM)** is a probabilistic sequence model in which the underlying state sequence is unobserved, while each state emits an observed item. For POS tagging, words are observed and their grammatical tags are hidden. The task is to infer a suitable tag sequence from the words. Hidden: Noun → Hidden: Verb → Hidden: Noun Observed: birds → Observed: eat → Observed: seeds Horizontal state-to-state links are **transitions**. The relation from each hidden state to its observed word is an **emission**. A transition P(V|N) asks how often a verb tag follows a noun tag. An emission P(eat|V) asks how often the observed word eat appears among verb-tagged tokens. | Component | Notation | Meaning | | --- | --- | --- | | State set | S={N,V,…} | Possible hidden tags. | | Observation vocabulary | O={birds,eat,seeds,…} | Visible word tokens. | | Initial distribution | πⱼ=P(t₁=j) | Probability of starting in tag j. | | Transition matrix | aᵢⱼ=P(tₖ=j|tₖ₋₁=i) | Probability of next tag j after i. | | Emission matrix | bⱼ(w)=P(w|tag j) | Probability that state j emits word w. | Two basic assumptions are used: the current hidden state depends only on the previous hidden state, and the current observation depends only on the current hidden state. A standard first-order HMM therefore does not explicitly condition an emitted word on the whole sentence. $$ P(t_{1:T},w_{1:T})=\pi_{t_1}b_{t_1}(w_1)\prod_{k=2}^{T}a_{t_{k-1},t_k}b_{t_k}(w_k) $$ Read the formula as **start × first emission × transition × emission × transition × emission…**. There are T emission factors and T−1 transitions, plus one initial factor. If the question supplies an explicit end-state probability, multiply that too. Do not invent an end factor when the model has none. **Three different questions.** (1) Scoring a specified tag path uses the joint product above. (2) Finding the probability of the observed words sums scores over all possible tag paths. (3) Finding the best tag path maximizes the path score, typically using Viterbi. A joint path score is not a normalized posterior P(tags|words). **Yaad rakho:** Transition = tag se agla tag. Emission = tag se visible word. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Estimate transition and emission probabilities from tagged data Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm--transition-emission-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm/transition-emission-estimation Unit 4, pages 47–73 $$ P(t'\mid t)=\frac{C(t,t')}{C(t)\text{ with a following tag/end event}} $$ $$ P(w\mid t)=\frac{C(w,t)}{C(t)} $$ In a transition denominator, count the previous tag's outgoing events. In an emission denominator, count all tokens bearing that tag. With explicit end transitions the totals align conveniently. Keep the two numerator types separate: tag→tag counts are not word→tag counts. **Question. A tagged corpus contains 116454 determiner tags DT, each counted with an outgoing event. DT is followed by NN 56509 times. Estimate P(NN|DT).** #### Given C(DT)=116454; C(DT,NN)=56509. #### To find The DT→NN transition probability. #### Formula and why it applies $$ P(NN\mid DT)=C(DT,NN)/C(DT) $$ The denominator is the number of previous DT contexts because DT is the condition. #### Solution ##### Step 1. Identify the transition count DT followed by NN occurs 56509 times. ##### Step 2. Identify the conditional total There are 116454 DT contexts. ##### Step 3. Divide $$ P(NN\mid DT)=\frac{56509}{116454}\approx0.485247 $$ **Final answer.** P(NN|DT)≈0.485247. **Question. The tag VBZ occurs 21627 times, and 10073 of those tokens are the word “is”. Estimate P(is|VBZ).** #### Given C(VBZ)=21627; C(is,VBZ)=10073. #### To find The emission probability of is from VBZ. #### Formula and why it applies $$ P(is\mid VBZ)=C(is,VBZ)/C(VBZ) $$ The condition is the tag VBZ; we ask which word it emits. #### Solution ##### Step 1. Choose the word-tag count The required word is is with tag VBZ: count 10073. ##### Step 2. Choose the tag total All VBZ-tagged word tokens: count 21627. ##### Step 3. Divide $$ P(is\mid VBZ)=\frac{10073}{21627}\approx0.465760 $$ **Final answer.** P(is|VBZ)≈0.465760. This is different from P(VBZ|is). **Question. Use the tagged sentences “dogs/N chase/V cats/N” and “cats/N sleep/V”. Estimate the initial tag probabilities, all nonzero transitions including END, and emissions.** #### Given Corpus: START dogs/N chase/V cats/N END; START cats/N sleep/V END. #### To find Initial distribution, transition rows and emission rows. #### Formula and why it applies $$ \pi_t=C(START,t)/2,\quad a_{tu}=C(t,u)/C(t),\quad b_t(w)=C(w,t)/C(t) $$ The corpus explicitly tells us the tags, so we can count training frequencies directly. #### Solution ##### Step 1. Count tag tokens N occurs 3 times: dogs, cats, cats. V occurs 2 times: chase, sleep. ##### Step 2. Count sentence starts Both sentences start with N. $$ \pi_N=2/2=1,\quad \pi_V=0/2=0 $$ ##### Step 3. Count transitions | Pair | Count | | --- | --- | | N→V | 2 | | N→END | 1 | | V→N | 1 | | V→END | 1 | ##### Step 4. Normalize transition rows | Current | To N | To V | To END | | --- | --- | --- | --- | | N | 0 | 2/3 | 1/3 | | V | 1/2 | 0 | 1/2 | ##### Step 5. Count and normalize emissions | Tag | dogs | cats | chase | sleep | | --- | --- | --- | --- | --- | | N | 1/3 | 2/3 | 0 | 0 | | V | 0 | 0 | 1/2 | 1/2 | **Final answer.** π=(1,0); transitions and emissions are given in the completed tables. Each complete probability row sums to 1. **Yaad rakho:** Tag→tag table aur tag→word table ko mix mat karna. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · HMM path probabilities: three complete numerical examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm--hot-cold-paths/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm/hot-cold-paths Unit 4, pages 47–73 The following model uses hidden weather states Hot (H) and Cold (C). Observations are numbers 1, 2 and 3. These numbers are observation labels, not quantities to multiply directly. Their emission probabilities come from the table. | Initial state | Probability | | --- | --- | | H | 0.8 | | C | 0.2 | | From → to | H | C | | --- | --- | --- | | H | 0.7 | 0.3 | | C | 0.4 | 0.6 | | Emission state | Observation 1 | Observation 2 | Observation 3 | | --- | --- | --- | --- | | H | 0.2 | 0.4 | 0.4 | | C | 0.5 | 0.4 | 0.1 | **Question. An HMM has π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for observations (1,2,3) are H:(0.2,0.4,0.4) and C:(0.5,0.4,0.1). Find the joint probability of hidden path H,C,H and observations 1,3,1.** #### Given Hidden path H→C→H. Observations 1→3→1. Model probabilities are specified in the question. There is no END factor. #### To find P(H,C,H; 1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HC}b_C(3)a_{CH}b_H(1) $$ There are three hidden states and observations: three emissions, two transitions, and one initial probability. #### Solution ##### Step 1. Start Hot and emit 1 $$ 0.8\times0.2=0.16 $$ ##### Step 2. Transition Hot to Cold $$ 0.16\times0.3=0.048 $$ ##### Step 3. Cold emits 3 $$ 0.048\times0.1=0.0048 $$ ##### Step 4. Transition Cold to Hot $$ 0.0048\times0.4=0.00192 $$ ##### Step 5. Hot emits the last 1 $$ 0.00192\times0.2=0.000384 $$ **Final answer.** Joint probability = 0.000384 = 3.84×10⁻⁴. **Question. For π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels (1,2,3), find the joint probability of path H,H,H and observations 1,3,1.** #### Given State path H→H→H, observations 1→3→1; no END factor. #### To find P(H,H,H;1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HH}b_H(3)a_{HH}b_H(1) $$ The state remains Hot; each transition still contributes its probability 0.7. #### Solution ##### Step 1. Initial state and first emission $$ 0.8(0.2)=0.16 $$ ##### Step 2. First self-transition $$ 0.16(0.7)=0.112 $$ ##### Step 3. Emit observation 3 $$ 0.112(0.4)=0.0448 $$ ##### Step 4. Second self-transition $$ 0.0448(0.7)=0.03136 $$ ##### Step 5. Emit final observation 1 $$ 0.03136(0.2)=0.006272 $$ **Final answer.** Joint probability = 0.006272. A self-transition is not automatically probability 1. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6) in next-state order H,C; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3, find the score of path C,C,H with observations 1,3,1.** #### Given Hidden path C→C→H. Observation sequence 1→3→1. No end-state factor. #### To find The joint path-and-observation probability. #### Formula and why it applies $$ P=\pi_C b_C(1)a_{CC}b_C(3)a_{CH}b_H(1) $$ The initial state is Cold, so use π(C), not π(H). Each emission is selected from the state at the same position. #### Solution ##### Step 1. Start Cold and emit 1 $$ 0.2(0.5)=0.1 $$ ##### Step 2. Stay Cold $$ 0.1(0.6)=0.06 $$ ##### Step 3. Emit 3 from Cold $$ 0.06(0.1)=0.006 $$ ##### Step 4. Move Cold to Hot $$ 0.006(0.4)=0.0024 $$ ##### Step 5. Emit 1 from Hot $$ 0.0024(0.2)=0.00048 $$ **Final answer.** Joint probability = 0.00048. **Yaad rakho:** Har column par: state ka emission. Columns ke beech: transition. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · POS HMM worked corpus: Mary, Jane, can, will and spot Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm--tagging-corpus-case/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm/tagging-corpus-case Unit 4, pages 47–73 The class example contains words that can be either names or ordinary verbs/modal auxiliaries. The term is **modal** (M), not “model”. To avoid guessing tags from spelling, the training corpus below supplies them explicitly and lowercases words for the emission vocabulary. | Training sentence | Gold tags | | --- | --- | | mary jane can see will | N N M V N | | spot will see mary | N M V N | | will jane spot mary | M N V N | | mary will pat spot | N M V N | There are 9 noun tokens, 4 modal tokens and 4 verb tokens. Each sentence includes a start and end transition. For nouns, the next event can be another noun, modal, verb or END. | Current → next | N | M | V | END | | --- | --- | --- | --- | --- | | START | 3/4 | 1/4 | 0 | 0 | | N | 1/9 | 3/9 | 1/9 | 4/9 | | M | 1/4 | 0 | 3/4 | 0 | | V | 4/4 | 0 | 0 | 0 | | Tag → word | mary | jane | can | see | will | spot | pat | | --- | --- | --- | --- | --- | --- | --- | --- | | N | 4/9 | 2/9 | 0 | 0 | 1/9 | 2/9 | 0 | | M | 0 | 0 | 1/4 | 0 | 3/4 | 0 | 0 | | V | 0 | 0 | 0 | 2/4 | 0 | 1/4 | 1/4 | **Question. Using the fully tagged four-sentence corpus above, calculate the joint score of “will jane spot mary” assigned tags M,N,V,N, including END.** #### Given Required factors from the tables: START→M=1/4; emit will|M=3/4; M→N=1/4; jane|N=2/9; N→V=1/9; spot|V=1/4; V→N=1; mary|N=4/9; N→END=4/9. #### To find The specified tag-path score, including words and sentence end. #### Formula and why it applies $$ P=\pi_M b_M(will)a_{MN}b_N(jane)a_{NV}b_V(spot)a_{VN}b_N(mary)a_{N,END} $$ Both transitions and emissions are needed. A tag transition alone cannot tell us whether a particular word is plausible. #### Solution ##### Step 1. Start and emit will $$ \frac14\times\frac34=\frac3{16} $$ ##### Step 2. Move to N and emit jane $$ \frac3{16}\times\frac14\times\frac29=\frac1{96} $$ ##### Step 3. Move to V and emit spot $$ \frac1{96}\times\frac19\times\frac14=\frac1{3456} $$ ##### Step 4. Move to N and emit mary $$ \frac1{3456}\times1\times\frac49=\frac1{7776} $$ ##### Step 5. End the sentence $$ \frac1{7776}\times\frac49=\frac1{17496}\approx0.0000571559 $$ **Final answer.** The joint score is 1/17496 ≈ 0.0000571559. **Question. For the same tagged corpus and tables, explain the unsmoothed score of “will can spot mary” assigned tags M,V,N,N.** #### Given The proposed second word is can with hidden tag V. In the training corpus, can occurs only with M, so b\_V(can)=0. #### To find Whether the complete path score is zero and what this means. #### Formula and why it applies $$ P(\text{path,words})=(\text{other required factors})\times b_V(can) $$ A product containing one zero factor is zero even if every other factor is positive. #### Solution ##### Step 1. Find the critical emission $$ b_V(can)=C(can,V)/C(V)=0/4=0 $$ ##### Step 2. Use the full-product consequence $$ P(\text{path,words})=0 $$ ##### Step 3. Interpret correctly This unsmoothed corpus model has no evidence for can as a verb. It does not prove that a verb usage of can is impossible in English. **Final answer.** The unsmoothed path score is zero. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Why exhaustive search grows exponentially Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm-decoding--sequence-count/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm-decoding/sequence-count Unit 4 page 73: Viterbi motivation, explained with small worked extensions If each of T observed words can receive any of N tags, there are N choices at each position. Thus the total number of complete tag sequences is N multiplied by itself T times. This counts possible assignments before grammar rules or zero probabilities eliminate any. $$ \text{Number of tag sequences}=N^T $$ **Question. A four-word sentence is tagged using three possible POS tags per word. How many complete tag sequences are possible?** #### Given N=3 possible tags per position; T=4 word positions. #### To find The number of tag sequences. #### Formula and why it applies $$ M=N^T $$ Every independent position contributes a factor of three choices. #### Solution ##### Step 1. Write one factor per position $$ M=3\times3\times3\times3 $$ ##### Step 2. Multiply $$ 3\times3=9,\quad9\times3=27,\quad27\times3=81 $$ **Final answer.** 81 possible tag sequences. **Question. A five-word sentence has two candidate tags per word. Count its tag sequences.** #### Given N=2; T=5. #### To find M. #### Formula and why it applies $$ M=N^T $$ The length is the exponent, not a multiplier. #### Solution ##### Step 1. Substitute $$ M=2^5 $$ ##### Step 2. Evaluate $$ 2\times2\times2\times2\times2=32 $$ **Final answer.** 32 possible sequences. **Question. Using 45 possible tags, compare the number of tag sequences for sentences of length 2 and 3.** #### Given N=45. First T=2, then T=3. #### To find Both counts and the growth factor. #### Formula and why it applies $$ M_2=45^2,\quad M_3=45^3 $$ Adding one position multiplies the possibilities by the number of tag choices. #### Solution ##### Step 1. Two words $$ 45^2=2025 $$ ##### Step 2. Three words $$ 45^3=2025\times45=91125 $$ ##### Step 3. Compare $$ 91125/2025=45 $$ **Final answer.** 2 words: 2025 paths. 3 words: 91125 paths. One additional word multiplies the count by 45. The slides introduce Viterbi to avoid enumerating every path. It reuses the best prefix score for each ending state. For a dense first-order HMM, it needs O(TN²) recurrence work instead of enumerating NT complete paths. **Yaad rakho:** Choices per word ki power = sentence length. ### M2 10 Viterbi Decoding id : nlp-m2-10-viterbi-decoding title : M2 10 Viterbi Decoding title\_confirmed : true topic\_title : Viterbi decoding with backpointers mode : m2 kind : focused status : available duration : 9:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Viterbi decoding: compare incoming paths, keep the best, trace back Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm-decoding--viterbi-explained/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm-decoding/viterbi-explained Unit 4 page 73: Viterbi motivation, explained with small worked extensions **The Viterbi algorithm** finds the most likely hidden-state sequence for a fixed observation sequence. At each position, it keeps the best path ending in each state. This is safe because, under the HMM assumptions, future extensions depend on that ending state, not the discarded details of the prefix. $$ \delta_1(j)=\pi_jb_j(o_1) $$ $$ \delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ $$ \psi_t(j)=\arg\max_i[\delta_{t-1}(i)a_{ij}] $$ | Symbol | Meaning in ordinary words | | --- | --- | | δₜ(j) | Score of the best prefix that ends in state j at position t. | | ψₜ(j) | Previous state that supplied that best prefix; keep it as a backpointer. | | max | Choose the larger candidate score; do not add. | | argmax | Remember which previous state won, rather than only its value. | Initialize: start × first emission → For each next state: compare incoming scores → Multiply winner by this state's emission → Keep score + winning predecessor → Choose best final state and trace arrows backward Do not choose the best state independently at each word. The best last state can come from a different best predecessor than a local guess. Backpointers recover one consistent complete path. If scores tie, either tied path is optimal under the model; state the tie. **Question. Use an HMM with π(H)=0.8, π(C)=0.2, transitions H:(0.7,0.3), C:(0.4,0.6) in H,C order, and emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3. Find the most likely hidden path for observations 1,3,1 using Viterbi.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find The best hidden-state sequence and its joint score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(o_1),\quad\delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ Viterbi keeps two best prefix scores per column, one ending H and one ending C. #### Solution ##### Step 1. Initialize at observation 1 $$ \delta_1(H)=0.8(0.2)=0.16 $$ $$ \delta_1(C)=0.2(0.5)=0.10 $$ ##### Step 2. At observation 3, calculate the H cell $$ H\to H:\ 0.16(0.7)=0.112 $$ $$ C\to H:\ 0.10(0.4)=0.040 $$ $$ \delta_2(H)=\max(0.112,0.040)(0.4)=0.0448 $$ Backpointer for this H cell: H. ##### Step 3. At observation 3, calculate the C cell $$ H\to C:\ 0.16(0.3)=0.048 $$ $$ C\to C:\ 0.10(0.6)=0.060 $$ $$ \delta_2(C)=\max(0.048,0.060)(0.1)=0.006 $$ Backpointer for this C cell: C. ##### Step 4. At final observation 1, calculate H $$ H\to H:\ 0.0448(0.7)=0.03136 $$ $$ C\to H:\ 0.006(0.4)=0.0024 $$ $$ \delta_3(H)=0.03136(0.2)=0.006272 $$ Backpointer: H. ##### Step 5. At final observation 1, calculate C $$ H\to C:\ 0.0448(0.3)=0.01344 $$ $$ C\to C:\ 0.006(0.6)=0.0036 $$ $$ \delta_3(C)=0.01344(0.5)=0.00672 $$ Backpointer: H. ##### Step 6. Select the final state 0.00672 > 0.006272, so the best final state is C. ##### Step 7. Trace back Final C came from H at position2. That H came from H at position1. Reading forward gives H,H,C. ##### Step 8. Verify the recovered complete path $$ 0.8(0.2)(0.7)(0.4)(0.3)(0.5)=0.00672 $$ **Final answer.** Best path H→H→C with joint score 0.00672. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 3,1.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best two-state hidden path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(3),\quad\delta_2(j)=b_j(1)\max_i[\delta_1(i)a_{ij}] $$ The same recurrence applies even though the observation order changed. #### Solution ##### Step 1. Initialize observation 3 $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.1)=0.02 $$ ##### Step 2. Calculate the final H $$ \max(0.32(0.7),0.02(0.4))(0.2) $$ $$ =\max(0.224,0.008)(0.2)=0.0448 $$ Winning predecessor H. ##### Step 3. Calculate the final C $$ \max(0.32(0.3),0.02(0.6))(0.5) $$ $$ =\max(0.096,0.012)(0.5)=0.048 $$ Winning predecessor H. ##### Step 4. Select and trace Final C wins with 0.048 and came from H. The path is H,C. **Final answer.** Best path H→C, score 0.048. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 2,2.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(2),\quad\delta_2(j)=b_j(2)\max_i[\delta_1(i)a_{ij}] $$ Both states emit2 with probability 0.4, but their initial and transition probabilities still differ. #### Solution ##### Step 1. Initialize $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.4)=0.08 $$ ##### Step 2. Final H $$ \max(0.32(0.7),0.08(0.4))(0.4) $$ $$ =\max(0.224,0.032)(0.4)=0.0896 $$ Predecessor H. ##### Step 3. Final C $$ \max(0.32(0.3),0.08(0.6))(0.4) $$ $$ =\max(0.096,0.048)(0.4)=0.0384 $$ Predecessor H. ##### Step 4. Select and trace The final H score 0.0896 is larger. Its predecessor is H. **Final answer.** Best path H→H with score 0.0896. **Yaad rakho:** Har destination ke liye incoming paths compare; winner ka arrow yaad. ### M2 10 Viterbi Decoding id : nlp-m2-10-viterbi-decoding title : M2 10 Viterbi Decoding title\_confirmed : true topic\_title : Viterbi decoding with backpointers mode : m2 kind : focused status : available duration : 9:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · A path score is not the probability of the observed sentence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-hmm-decoding--joint-vs-posterior/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-hmm-decoding/joint-vs-posterior Unit 4 page 73: Viterbi motivation, explained with small worked extensions **Question. An HMM has π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3. For observations 1,3, calculate the probability of the observations by summing all four two-state paths.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find P(observations 1,3), not just the best-path score. #### Formula and why it applies $$ P(o_1,o_2)=\sum_{s_1,s_2}\pi_{s_1}b_{s_1}(o_1)a_{s_1s_2}b_{s_2}(o_2) $$ The hidden path is unspecified, so its mutually exclusive alternatives must be summed. #### Solution ##### Step 1. Score H,H $$ 0.8(0.2)(0.7)(0.4)=0.0448 $$ ##### Step 2. Score H,C $$ 0.8(0.2)(0.3)(0.1)=0.0048 $$ ##### Step 3. Score C,H $$ 0.2(0.5)(0.4)(0.4)=0.016 $$ ##### Step 4. Score C,C $$ 0.2(0.5)(0.6)(0.1)=0.006 $$ ##### Step 5. Add all paths $$ P(1,3)=0.0448+0.0048+0.016+0.006=0.0716 $$ **Final answer.** Observation probability =0.0716. The best single-path score 0.0448 is a different quantity. If asked for the posterior of path H,H given observations 1,3, normalize: 0.0448/0.0716 ≈0.625698. Viterbi chooses the best path using joint scores because the observation denominator is common to every candidate. It does not compute that denominator by taking a maximum. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Chunking groups words; chinking removes words from a candidate chunk Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-chunks-grammar--chunking-chinking/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-chunks-grammar/chunking-chinking Unit 4, pages 14–19, 74–97 **Chunking**, also called shallow parsing, groups adjacent tagged words into meaningful non-overlapping units, commonly noun phrases. It usually identifies local phrase boundaries without producing the complete hierarchical structure of a sentence. **Chinking** removes specified tokens from an existing candidate chunk, splitting or trimming it. The/DT small/JJ cat/NN sat/VBD on/IN the/DT mat/NN → NP: [The small cat] → Outside: sat on → NP: [the mat] A POS tagger labels individual words. A chunker groups words into phrases. A full parser shows how phrases relate recursively to form the entire sentence. These are related tasks with different outputs. | Operation | Input | Output | | --- | --- | --- | | POS tagging | The small cat | The/DT small/JJ cat/NN | | NP chunking | Tagged tokens | [NP The small cat] | | Chinking | A broad candidate chunk | Remove verbs/prepositions, keep noun-phrase segments | | Full parsing | Sentence + grammar | A nested tree with sentence, noun phrase and verb phrase nodes | **Rule-based chunking** uses tag patterns. Statistical or learned chunkers infer boundaries from labeled examples. A useful NP rule is an optional determiner, followed by zero or more adjectives, followed by a noun. In NLTK-style notation: `NP: {
?*}`. Here ? means zero or one occurrence; \* means zero or more; + means one or more when used. The braces mean “form a chunk around the matched sequence”. **Question. Apply the NP rule optional DT, zero or more JJ, then one NN to “The/DT quick/JJ brown/JJ fox/NN jumps/VBZ over/IN the/DT lazy/JJ dog/NN”.** #### Given Tagged input is supplied. Rule: DT? JJ\* NN. #### To find All noun-phrase chunks. #### Formula and why it applies `NP → DT? JJ* NN` The rule groups a noun with its preceding determiner and adjectives; a verb or preposition does not match this NP pattern. #### Solution ##### Step 1. Start at The The matches optional DT; quick and brown both match JJ\*; fox matches NN. ##### Step 2. Form the first NP **[NP The quick brown fox]** ##### Step 3. Skip nonmatching tokens jumps/VBZ and over/IN do not satisfy the NP pattern. ##### Step 4. Match the second NP the is DT, lazy is JJ, dog is NN: **[NP the lazy dog]**. **Final answer.** [NP The quick brown fox] jumps over [NP the lazy dog]. **Question. Using NP → DT? JJ\* NN, identify the chunks in “bright/JJ stars/NN shine/VBP and/CC a/DT moon/NN glows/VBZ”. Use the supplied tags exactly.** #### Given Input tags label stars as NN for this exercise. Rule requires optional DT, zero or more JJ, then NN. #### To find The matched NP spans. #### Formula and why it applies `NP → DT? JJ* NN` The optional determiner may be absent and the adjective repetition may be empty. #### Solution ##### Step 1. Match bright stars No determiner is present, which DT? permits. bright matches JJ; stars matches NN. ##### Step 2. Inspect shine and and VBP and CC fall outside the NP rule. ##### Step 3. Match a moon a matches DT; no adjective is present; moon matches NN. **Final answer.** Chunks are [NP bright stars] and [NP a moon]. With ordinary Penn tagging, stars is usually NNS; broaden the noun pattern to include NNS when that tag is used. **Question. Begin with one candidate chunk containing “The/DT cat/NN sat/VBD on/IN the/DT mat/NN”. Chink out verbs and prepositions. Show the remaining chunks.** #### Given Initial candidate: [The cat sat on the mat]. Remove tags VBD and IN. #### To find The chunk segments after exclusion. #### Formula and why it applies Chinking removes matched tokens from the existing chunk. Removing the middle verb/preposition region divides the original candidate into two remaining contiguous spans. #### Solution ##### Step 1. Locate the tokens to remove sat is VBD and on is IN. ##### Step 2. Remove those tokens from chunk membership The remaining token groups are “The cat” and “the mat”. sat and on still exist in the sentence; they are simply outside these chunks. ##### Step 3. Write the final chunked sentence **[The cat]** sat on **[the mat]**. **Final answer.** Two chunks remain: The cat; the mat. **Yaad rakho:** Chunking = andar group karo; chinking = unwanted tokens ko group se bahar karo. ### NLP M2 — Chunking and Chinking | Reviewed rules and worked examples id : nlp-m2-11-chunking-chinking title : NLP M2 — Chunking and Chinking | Reviewed rules and worked examples title\_confirmed : true topic\_title : Chunking and chinking from POS-tagged words mode : m2 kind : focused status : available duration : 8:22 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Correct POS labels including JJ = adjective, full NP rule application and chinking boundaries. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · CFG: terminals, nonterminals, production rules and parse trees Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-chunks-grammar--cfg-foundation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-chunks-grammar/cfg-foundation Unit 4, pages 14–19, 74–97 **A context-free grammar (CFG)** specifies a language using production rules whose left side is a single nonterminal. Formally, G=(N,Σ,R,S): N is the set of nonterminals, Σ the terminal vocabulary, R the production rules, and S the start symbol. A rule can expand its nonterminal regardless of the neighboring symbols; that is the meaning of “context-free”. | Concept | Plain explanation | Example | | --- | --- | --- | | Terminal | A word at a leaf; it is not expanded further | the, girl, read | | Nonterminal | A grammatical category that can be expanded | S, NP, VP, Det, N, V | | Production | A permitted replacement | S → NP VP | | Start symbol | Category for a complete sentence | S | | Lexicon | Word-to-category rules | N → girl | paper | | Derivation | Sequence of rule applications | S ⇒ NP VP ⇒ Det N VP ⇒ … | | Parse tree | Hierarchical picture of those expansions | Root S, internal phrases, word leaves | $$ S\to NP\ VP,\quad NP\to Det\ N,\quad VP\to V\ NP $$ Read S→NP VP as “a sentence can contain a noun phrase followed by a verb phrase”. It does not mean subtraction or multiplication. A vertical bar in a grammar rule means alternatives: Det→the | a allows either the or a. This is different from the probability bar meaning “given”. S: the whole sentence → NP: the girl + VP: read the paper → NP→Det N; VP→V NP → Leaves in order: the / girl / read / the / paper **A constituent** is a group of words that acts as a grammatical unit, such as “the girl”. **Syntactic parsing** assigns a grammatical structure to an input sentence using a grammar. It can support grammar checking, information extraction, translation and later semantic analysis. Failure under one restricted grammar means that grammar could not derive the sentence; it does not necessarily prove the sentence is invalid English. **A treebank** is a corpus annotated with parse trees. It supplies training/evaluation structures for parsers and lets linguists study recurring syntactic patterns. A tagged corpus gives word tags; a treebank additionally records relationships among phrases. | Representation | Example information retained | | --- | --- | | Bag of words | Words occur, but word order/grammar is lost | | POS tags | Each word's local grammatical category | | Chunks | Short phrase boundaries | | Parse tree | Nested constituent structure of the whole sentence | **Yaad rakho:** Grammar rule ek expansion instruction hai: S ko NP aur VP mein kholo. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Ambiguity and grammar coverage Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-chunks-grammar--structural-ambiguity/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-chunks-grammar/structural-ambiguity Unit 4, pages 14–19, 74–97 **Structural ambiguity** occurs when a sentence has more than one valid parse tree. In “I saw the man with a telescope”, the prepositional phrase “with a telescope” can attach to saw (the instrument I used) or to the man (the man possessing the telescope). The words and their order stay the same, but the grouping changes. | Reading | Attachment | Meaning | | --- | --- | --- | | I [saw the man] [with a telescope] | PP modifies VP | I used a telescope to see the man. | | I saw [the man with a telescope] | PP modifies NP | The man had a telescope. | A parser may need context, probabilities or semantic constraints to choose between trees. A plain CFG describes permitted structures but does not itself assign a probability to each tree. Do not invent rule probabilities for an ordinary CFG question. Lexical ambiguity also matters: book can be a noun or verb. In “Book the flight”, the imperative analysis begins S→VP, followed by VP→V NP. A parser that initially chooses book as a noun may reach a dead end and must try another permitted analysis. **Yaad rakho:** Same words, alag grouping → alag meaning. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Recursive-descent top-down parsing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-parsing--top-down-theory/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-parsing/top-down-theory Unit 4, pages 98–116 **Top-down parsing** starts at the grammar's start symbol S and expands nonterminals until the leaves match the input words in order. **Recursive descent** commonly implements each nonterminal as a procedure that tries its productions. It builds from the root toward the leaves. Start with S → Choose an allowed S production → Expand its next nonterminal → Match the resulting terminal to the input → Continue, or backtrack if the choice fails **Algorithm.** Start with S as the current goal. Choose a production for the leftmost unexpanded nonterminal. Replace that nonterminal by the production's right side. If a terminal appears, compare it with the next input token. If it matches, consume the token; if it does not, reject that branch and try another production. Accept only when the complete input is consumed and all grammar goals are satisfied. A **leftmost derivation** always expands the leftmost remaining nonterminal first. Top-down parsing can make guesses that the actual words cannot support, so it may need backtracking. A naive recursive-descent parser can loop on left-recursive rules such as NP→NP PP unless the grammar or algorithm is changed. Predictive LL(1) parsing chooses productions using one lookahead token when the grammar permits; not every English CFG is LL(1). | Approach | Begins with | Direction | Common difficulty | | --- | --- | --- | --- | | Top-down | Start symbol S | Root to word leaves | Predictions that do not match input; left recursion | | Bottom-up | Actual input tokens | Leaves to root S | Reductions that cannot contribute to a complete sentence | **Yaad rakho:** Top-down: sentence ka structure pehle, words tak neeche aao. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Three full top-down derivations with parse trees Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-parsing--top-down-examples/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-parsing/top-down-examples Unit 4, pages 98–116 **Question. Using S→NP VP, NP→Det N, VP→V NP, Det→the, N→girl|paper and V→read, give a leftmost top-down derivation and parse tree for “the girl read the paper”.** #### Given | Rule type | Rules | | --- | --- | | Structure | S → NP VP; NP → Det N; VP → V NP | | Lexicon | Det → the; N → girl | paper; V → read | #### To find A successful derivation and tree whose leaves equal the input. #### Formula and why it applies At every stage expand the leftmost nonterminal using a supplied production. A top-down derivation begins at S; the input determines which terminal alternatives can succeed. #### Solution ##### Step 1. Begin at the sentence symbol S ##### Step 2. Expand the sentence S ⇒ NP VP ##### Step 3. Expand its first noun phrase ⇒ Det N VP ##### Step 4. Match the first determiner ⇒ the N VP ##### Step 5. Match the subject noun ⇒ the girl VP ##### Step 6. Expand the verb phrase ⇒ the girl V NP ##### Step 7. Match the verb ⇒ the girl read NP ##### Step 8. Expand the object noun phrase ⇒ the girl read Det N ##### Step 9. Match its determiner ⇒ the girl read the N ##### Step 10. Match the last noun ⇒ the girl read the paper ##### Step 11. Draw the tree Parse tree: the girl read the paperSNPDettheNgirlVPVreadNPDettheNpaper Parse tree: the girl read the paper **Final answer.** The grammar derives the complete sentence. The subject NP is “the girl”; the object NP is “the paper”. **Question. Use S→NP VP, NP→Det N|N, VP→V NP, Det→A, N→restaurant|Dosas and V→serves to parse “A restaurant serves Dosas” by a leftmost top-down derivation.** #### Given | Rule type | Rules | | --- | --- | | Structure | S → NP VP; NP → Det N | N; VP → V NP | | Lexicon | Det → A; N → restaurant | Dosas; V → serves | #### To find A derivation and parse tree. #### Formula and why it applies Expand the leftmost nonterminal; choose NP→Det N for the subject and NP→N for the object. The object has no determiner, so forcing NP→Det N there would fail. #### Solution ##### Step 1. Start and expand S S ⇒ NP VP ##### Step 2. Choose the subject NP ⇒ Det N VP ##### Step 3. Match A ⇒ A N VP ##### Step 4. Match restaurant ⇒ A restaurant VP ##### Step 5. Expand the VP ⇒ A restaurant V NP ##### Step 6. Match serves ⇒ A restaurant serves NP ##### Step 7. Choose the object NP rule ⇒ A restaurant serves N ##### Step 8. Match Dosas ⇒ A restaurant serves Dosas ##### Step 9. Draw the successful tree Parse tree: A restaurant serves DosasSNPDetANrestaurantVPVservesNPNDosas Parse tree: A restaurant serves Dosas **Final answer.** The parse succeeds with subject NP→Det N and object NP→N. **Question. Using S→NP VP|VP, NP→Det N, VP→V NP, Det→that, N→book|flight and V→book, derive the imperative “book that flight” and explain the role of book.** #### Given | Rule type | Rules | | --- | --- | | Structure | S → NP VP | VP; NP → Det N; VP → V NP | | Lexicon | Det → that; N → book | flight; V → book | #### To find The successful derivation, tree and POS of book. #### Formula and why it applies Choose a production for S that can derive the actual input; an imperative may consist of VP. The grammar allows book as both noun and verb, but this imperative needs the verb reading. #### Solution ##### Step 1. Choose the imperative sentence rule S ⇒ VP ##### Step 2. Expand the verb phrase ⇒ V NP ##### Step 3. Choose the verb book ⇒ book NP ##### Step 4. Expand the object ⇒ book Det N ##### Step 5. Match that ⇒ book that N ##### Step 6. Match flight ⇒ book that flight ##### Step 7. Draw the tree Imperative parse: book that flightSVPVbookNPDetthatNflight Imperative parse: book that flight **Final answer.** book is V; that flight is NP; S→VP derives the imperative. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Shift-reduce bottom-up parsing: stack, input and actions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-parsing--shift-reduce-theory/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-parsing/shift-reduce-theory Unit 4, pages 98–116 **Bottom-up parsing** starts from the input tokens and combines them into larger constituents until the start symbol is reached. A **shift-reduce parser** maintains a stack of recognized material and a list of remaining input tokens. | Action | Exact operation | Example | | --- | --- | --- | | Shift | Move the next input token onto the stack | Stack: the; remaining input: dog runs | | Reduce | Replace a suffix of the stack matching a rule's RHS with its LHS | Det N becomes NP using NP→Det N | | Accept | Stack is S and input is exhausted | Stack: S; input: empty | | Fail/backtrack | Chosen reductions cannot complete a parse | A noun reading of book blocks an imperative parse | Reduction uses a production in reverse: the grammar NP→Det N expands an NP top-down, but reduces Det N into NP bottom-up. The stack's **rightmost item is its top** in the traces below. Each row shows the state **after** the listed action, avoiding an off-by-one mismatch between actions and stack contents. A lexical reduction such as N→dog still counts as a reduction. Do not skip the sequence from word to category to phrase when the question asks for a complete trace. Backtracking means returning to an earlier choice and trying another applicable rule, not changing the original input. **Yaad rakho:** Shift = input se stack. Reduce = chhote pieces se bada grammatical unit. ### M2 13 — Shift-reduce parsing and backtracking id : nlp-m2-13-bottom-up-shift-reduce title : M2 13 — Shift-reduce parsing and backtracking title\_confirmed : false topic\_title : Shift-reduce parsing and backtracking mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Three full shift-reduce traces Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-parsing--shift-reduce-examples/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-parsing/shift-reduce-examples Unit 4, pages 98–116 **Question. Show a complete shift-reduce parse of “the big dog sees the cat” using S→NP VP, NP→Det N|Det Adj N, VP→V NP, Det→the, Adj→big, N→dog|cat and V→sees.** #### Given Grammar is as specified. Stack starts empty; input is the big dog sees the cat. Top of stack is on the right. #### To find The stack, remaining input and action at every step. #### Formula and why it applies Shift tokens; reduce matched right-hand sides to their left-hand nonterminals. The first noun phrase includes an adjective and must use NP→Det Adj N. #### Solution ##### Step 1. Trace each action | Step / action | Stack after action | Remaining input | | --- | --- | --- | | 0. Start | ∅ | the big dog sees the cat | | 1. Shift the | the | big dog sees the cat | | 2. Reduce Det→the | Det | big dog sees the cat | | 3. Shift big | Det big | dog sees the cat | | 4. Reduce Adj→big | Det Adj | dog sees the cat | | 5. Shift dog | Det Adj dog | sees the cat | | 6. Reduce N→dog | Det Adj N | sees the cat | | 7. Reduce NP→Det Adj N | NP | sees the cat | | 8. Shift sees | NP sees | the cat | | 9. Reduce V→sees | NP V | the cat | | 10. Shift the | NP V the | cat | | 11. Reduce Det→the | NP V Det | cat | | 12. Shift cat | NP V Det cat | ∅ | | 13. Reduce N→cat | NP V Det N | ∅ | | 14. Reduce NP→Det N | NP V NP | ∅ | | 15. Reduce VP→V NP | NP VP | ∅ | | 16. Reduce S→NP VP | S | ∅ | ##### Step 2. Check acceptance Input is empty and stack is exactly S; therefore accept. ##### Step 3. Draw the resulting structure Parse tree: the big dog sees the catSNPDettheAdjbigNdogVPVseesNPDettheNcat Parse tree: the big dog sees the cat **Final answer.** Accepted. The parser builds both noun phrases, then VP, then S. **Question. Use S→NP VP, NP→Det N, VP→V NP, Det→the|this, N→man|book and V→read to give a shift-reduce trace for “the man read this book”.** #### Given The input has five tokens. Stack is empty initially. The lexical verb read is supplied by V→read. #### To find A complete bottom-up trace. #### Formula and why it applies Reduce only when the top stack sequence matches a supplied production's right side. Both noun phrases follow Det N, and the verb phrase follows V NP. #### Solution ##### Step 1. Trace all operations | Step / action | Stack after action | Remaining input | | --- | --- | --- | | 0. Start | ∅ | the man read this book | | 1. Shift the | the | man read this book | | 2. Reduce Det→the | Det | man read this book | | 3. Shift man | Det man | read this book | | 4. Reduce N→man | Det N | read this book | | 5. Reduce NP→Det N | NP | read this book | | 6. Shift read | NP read | this book | | 7. Reduce V→read | NP V | this book | | 8. Shift this | NP V this | book | | 9. Reduce Det→this | NP V Det | book | | 10. Shift book | NP V Det book | ∅ | | 11. Reduce N→book | NP V Det N | ∅ | | 12. Reduce NP→Det N | NP V NP | ∅ | | 13. Reduce VP→V NP | NP VP | ∅ | | 14. Reduce S→NP VP | S | ∅ | ##### Step 2. Verify the final state The complete sentence has reduced to S with no input left. **Final answer.** Accepted. Subject: the man. Predicate: read this book. Object: this book. **Question. Parse “book the flight” bottom-up using S→NP VP|VP, NP→Det N, VP→V NP, Det→the, N→book|flight, V→book. Explain why choosing the noun reading first can fail and show the successful alternative.** #### Given book has two lexical categories N and V. The input is an imperative. Stack starts empty. #### To find The failed choice and a complete successful trace with backtracking. #### Formula and why it applies Try lexical reductions allowed by the grammar; on a dead end, return to the ambiguous choice and try another. Grammar ambiguity requires a search choice. The noun reading cannot supply the required imperative verb. #### Solution ##### Step 1. Show the noun-first dead end Shift book and reduce it to N. Continue shifting/reducing the flight to Det N. Reducing Det N→NP leaves stack N NP and empty input. No supplied rule reduces N NP into S, so this branch fails. ##### Step 2. Return to the lexical choice Restore the state just after shifting book, then choose V→book instead of N→book. ##### Step 3. Trace the successful branch | Step / action | Stack after action | Remaining input | | --- | --- | --- | | 0. Restored after shifting book | book | the flight | | 1. Reduce V→book | V | the flight | | 2. Shift the | V the | flight | | 3. Reduce Det→the | V Det | flight | | 4. Shift flight | V Det flight | ∅ | | 5. Reduce N→flight | V Det N | ∅ | | 6. Reduce NP→Det N | V NP | ∅ | | 7. Reduce VP→V NP | VP | ∅ | | 8. Reduce S→VP | S | ∅ | ##### Step 4. Accept the alternative The verb reading produces S and consumes all tokens. **Final answer.** The noun-first branch fails; backtracking to book/V yields the successful imperative parse. **Yaad rakho:** Wrong branch fail hua toh sentence wrong nahi; next legal rule try karo. ### M2 13 — Shift-reduce parsing and backtracking id : nlp-m2-13-bottom-up-shift-reduce title : M2 13 — Shift-reduce parsing and backtracking title\_confirmed : false topic\_title : Shift-reduce parsing and backtracking mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Entity spans, categories and context Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-ner--ner-definition/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-ner/ner-definition Unit 4, pages 117–128 **Named Entity Recognition (NER)** identifies text spans referring to entities and assigns their types, such as PERSON, ORGANIZATION and LOCATION. Many practical schemas also include DATE, TIME, MONEY or domain-specific types. The exact allowed labels depend on the application's annotation scheme. NER has two parts: **boundary detection** finds where the entity starts and ends; **classification** assigns the appropriate type. A multiword name should normally be one entity span: “New York” is a location span, not two unrelated location decisions. Text: Mary joined Google in New York on Monday. → Mary → PERSON → Google → ORGANIZATION → New York → LOCATION → Monday → DATE Not every noun is a named entity. In “a developer works at Google”, developer is an ordinary job description while Google is a named organization. A specialized HR system may define a JOB\_TITLE category, but it must state that schema explicitly. NER identifies mentions; entity linking is the separate task of connecting a mention to a unique database identity. | Task | Question it answers | Example | | --- | --- | --- | | POS | What grammatical role does this token have? | Google/NNP | | Chunking | Which adjacent words form a phrase? | [NP a skilled developer] | | NER | Which span names an entity, and of what type? | [Google] ORGANIZATION | | Entity linking | Which real entity does this mention denote? | Connect Paris to the intended city or person record | **Applications.** NER structures search results and documents, extracts organizations/people from news, routes support tickets by product or organization, supports recommendation, and extracts specialized entities from healthcare or legal text. Context is essential: “Apple released a laptop” names an organization, while “she ate an apple” describes fruit. **Yaad rakho:** NER mein do kaam: naam kahan se kahan tak, aur kis category ka. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Dictionary, rules, machine learning and deep learning Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-ner--ner-approaches/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-ner/ner-approaches Unit 4, pages 117–128 | Approach | How it works | Example | Main limitation | | --- | --- | --- | --- | | Dictionary / gazetteer | Match text against lists of known names | A city list includes Mumbai and Pune | New names, spelling variants and ambiguous names are missed | | Rule-based | Use word shapes, regular expressions and context | A title such as Dr. followed by a name-like span suggests PERSON | Manual maintenance; exceptions and domain/language dependence | | Machine learning | Learn a labeling rule from annotated examples using features | Capitalization, suffixes, POS and surrounding words help a classifier/CRF | Requires labeled data and useful features | | Deep learning | Learn contextual word/character representations and predict labels | A contextual sequence encoder labels multiword entity spans | Training/domain dependence and resource demands | A **pattern-based rule** examines the form of text, such as an email pattern or a date format. A **context-based rule** examines neighboring words: “Dr.” can support a following PERSON span, while “headquartered in” can support a following LOCATION span. Capitalization alone is insufficient because sentence-initial common nouns are also capitalized. A dictionary may find “May” as a person's name, but “in May” may refer to a month. A learned model uses surrounding evidence to disambiguate. Hybrid systems can combine dictionaries, rules and learned models; these categories are practical tools rather than mutually exclusive restrictions. **Preprocessing for NER.** Tokenize while preserving token offsets and useful capitalization. Blindly removing capital letters or stopwords before recognition can damage names such as “Bank of India”. NER output should preserve the original span so the reader can locate it in the input. **Exam-ready 3-mark answer.** NER locates named-entity spans and classifies them into categories such as person, organization and location. A dictionary-based approach matches known name lists; a rule-based approach uses patterns or context; machine-learning and deep-learning approaches learn entity decisions from labeled data and representations. For example, in “Mary works at Google in Mumbai”, Mary is PERSON, Google is ORGANIZATION and Mumbai is LOCATION. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Three complete entity-recognition examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/m2-ner--ner-worked-examples/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/m2-ner/ner-worked-examples Unit 4, pages 117–128 **Question. Identify PERSON, ORGANIZATION and LOCATION entities in “Asha Rao joined Infosys in Bengaluru.”** #### Given Sentence: Asha Rao joined Infosys in Bengaluru. Allowed entity types: PERSON, ORGANIZATION, LOCATION. #### To find Complete entity spans and labels. #### Formula and why it applies Find the span first, then assign its type from context. A multiword personal name is a single entity; the verb and preposition remain outside entities. #### Solution ##### Step 1. Identify the person span Asha Rao names the person who joined, so take both words together. ##### Step 2. Identify the organization Infosys is the organization joined. ##### Step 3. Identify the location Bengaluru names the city. ##### Step 4. Write the output | Span | Label | | --- | --- | | Asha Rao | PERSON | | Infosys | ORGANIZATION | | Bengaluru | LOCATION | **Final answer.** [Asha Rao]PERSON joined [Infosys]ORGANIZATION in [Bengaluru]LOCATION. **Question. Using PERSON, ORGANIZATION, LOCATION and DATE, identify entities in “Dr. Meera Shah visited New Delhi on 5 October 2026.” Treat professional titles as outside the PERSON span for this exercise.** #### Given Allowed types and title-boundary convention are specified. Tokens include a multiword name, city and date. #### To find Entity boundaries and types. #### Formula and why it applies Boundary rules come from the annotation schema; use the stated title convention. Dr. supplies context for a person but is excluded from that span here. #### Solution ##### Step 1. Locate the person Meera Shah is the PERSON span; Dr. is outside by instruction. ##### Step 2. Locate the location New Delhi is one LOCATION span. ##### Step 3. Locate the date 5 October 2026 is one DATE span. ##### Step 4. Check organization mentions No organization is named in this sentence. **Final answer.** Meera Shah→PERSON; New Delhi→LOCATION; 5 October 2026→DATE. **Question. Explain the entity label of Apple in “Apple opened an office in Mumbai” and apple in “Ravi ate an apple”. Use PERSON, ORGANIZATION and LOCATION only.** #### Given Two contexts contain the same surface word apart from capitalization. The schema contains no FOOD category. #### To find Contextual labels and the ordinary noun case. #### Formula and why it applies NER labels a mention in context, not every occurrence of a dictionary string identically. A company can open an office; the object eaten in the second sentence is a fruit. #### Solution ##### Step 1. Analyze the first sentence Apple refers to the company, so label it ORGANIZATION. Mumbai is LOCATION. ##### Step 2. Analyze the second sentence Ravi is PERSON. apple is an ordinary fruit noun and receives no entity label under this schema. ##### Step 3. State the dictionary limitation Blindly labeling every apple as an organization would produce a false positive. **Final answer.** First: Apple→ORGANIZATION, Mumbai→LOCATION. Second: Ravi→PERSON; apple is outside the entity types. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Lexeme, lemma, sense and context Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-meaning--u5-words/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-meaning/u5-words Unit 5 · PDF pages 1–22 **Semantics is the study of meaning in language.** Lexical semantics studies meanings of words and relationships between their meanings. Sentence semantics studies how word meanings and sentence structure combine. A **word form** is what appears in the sentence. A **lexeme** is the underlying vocabulary item whose grammatical forms belong together. A **lemma** is the dictionary form used to name it. A **sense** is one particular meaning. | Term | Example | What to understand | | --- | --- | --- | | Word forms | run, runs, ran, running | Different grammatical forms | | Lexeme | RUN | The shared underlying vocabulary item | | Lemma | run | Dictionary heading | | Sense | run a race / run a business | Different meanings need context | | Lexicon | A language's vocabulary resource | Stores lexical items and information about them | The four forms *run, ran, runs, running* belong to the same lexeme. They are not four unrelated lexemes. “I deposited cash in the bank” and “We sat by the bank of the river” contain the same written form but different senses. word: bank → context: cash, deposit → sense: financial institution **Syntax** asks how words are arranged. **Semantics** asks what the expression means. **Pragmatics** uses situation and speaker intention: “Can you open the window?” is grammatically a question about ability but is normally a request. Form = jo likha; lemma = dictionary naam; sense = iss sentence mein matlab. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Every lexical relation with examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-meaning--u5-relations/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-meaning/u5-relations Unit 5 · PDF pages 1–22 **A lexical relation connects words or senses by a meaning relationship.** State the relation and its direction before giving the example. | Relation | Exam definition | Examples and direction | | --- | --- | --- | | Homonymy | The same form has unrelated meanings. | bank: financial institution / river edge; bat: animal / sports implement | | Polysemy | One lexical item has multiple related senses. | head: body part / head of a department; paper: material / article | | Synonymy | Different words have identical or closely similar meanings in a specified sense. | sofa / couch; automobile / car | | Antonymy | Words have opposite meanings. | hot / cold; alive / dead; buy / sell | | Hyponymy | A more specific concept is a type of a more general one. | rose is a flower; car is a vehicle | | Hypernymy | A broader concept includes a more specific one. | flower is a hypernym of rose | | Meronymy | A concept is a part of another concept. | wheel is part of car; finger is part of hand | | Holonymy | A concept is the whole containing a part. | car is a holonym of wheel | ### How to distinguish the commonly confused pairs For homonymy versus polysemy, ask whether the senses share an identifiable meaning. A river bank and a financial bank are unrelated. A financial bank and a blood bank share a repository/storage idea, illustrating polysemy in the supplied notes. For hyponymy versus meronymy, use the sentence test: **“X is a kind of Y”** indicates hyponymy; **“X is part of Y”** indicates meronymy. A wheel is not a type of car. | Expression | Correct relation | Reason | | --- | --- | --- | | sparrow → bird | Hyponymy | A sparrow is a kind of bird | | bird → sparrow | Hypernymy | Bird is the broader category | | keyboard → computer | Meronymy | Keyboard is a component | | computer → keyboard | Holonymy | Computer is the containing whole | | mouth of a person / mouth of a river | Polysemy | Both senses share the idea of an opening; the river use is a related extension | Synonyms are not always interchangeable in every context or register. “Big mistake” and “large mistake” can sound different even though big and large overlap in meaning. Antonyms can be gradable (hot/cold), complementary (alive/dead) or relational converses (buy/sell). HYPO = specific type; HYPER = bigger category. MERO = part; HOLO = whole. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · WordNet: synsets, glosses and hierarchy Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-meaning--u5-wordnet/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-meaning/u5-wordnet Unit 5 · PDF pages 1–22 **WordNet is a lexical database that groups word senses into synonym sets called synsets and links them through semantic and lexical relations.** It is a network of meanings, not merely a list of similar spellings. A synset represents one concept. Its **lemmas** are the words that can express that concept. A **gloss** is a short definition, often accompanied by an example. An ambiguous word can occur in several synsets. vehicle (general) → car / automobile (same concept) → sports car (specific) This is a simplified teaching hierarchy, not a claim about every intermediate WordNet node. Going upward reaches hypernyms; going downward reaches hyponyms. Multiple parents can exist, so the structure need not be a single simple tree. | Operation | Meaning | Illustration | | --- | --- | --- | | Get senses of a word | Find candidate synsets | bank → financial sense, river-edge sense, … | | Get lemmas | Find synonyms within one synset | car, automobile, motorcar | | Get definition/examples | Read the gloss and uses | Choose the intended concept | | Get hypernyms | Move to broader categories | car → vehicle | | Get hyponyms | Move to narrower categories | vehicle → particular vehicle categories | | Get meronyms/holonyms | Find part/whole links | car ↔ wheel | ### WordNet versus a thesaurus A thesaurus mainly groups related expressions for vocabulary lookup. WordNet explicitly distinguishes senses and labels relations such as hypernymy, hyponymy and meronymy. This supports WSD, query expansion, lexical similarity and semantic search. ### Three uses explained 1. **Search expansion:** a query for “automobile” can retrieve relevant documents using “car”. Expansion should use the right sense. 2. **WSD:** definitions of the senses of “bank” provide words that can be compared with the sentence context. 3. **Similarity:** concepts near one another in an “is-a” hierarchy tend to be semantically related. Path-based measures depend on which senses were selected. WordNet mein spelling ka group nahi—meaning ka synset, aur meanings ke beech labelled links. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · What WSD solves and how approaches differ Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-wsd--u5-wsd-basics/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-wsd/u5-wsd-basics Unit 5 · PDF pages 23–48 **Word sense disambiguation (WSD) is the task of selecting the intended sense of an ambiguous word in its context from a set of possible senses.** The output is a sense label, not a different replacement word. Input: “He grilled the bass.” → Candidate senses: fish / low sound → Context: grilled → Output: fish sense The **target word** is the ambiguous word. The **context** is the surrounding evidence. The **sense inventory** lists candidate meanings, often using WordNet. A **feature** is a usable clue: nearby word, position, part-of-speech tag or a phrase. | Approach | What it needs | How it decides | Limitation | | --- | --- | --- | --- | | Knowledge based | Dictionary, thesaurus or semantic graph | Definition overlap, category evidence or graph scores | Short definitions and missing knowledge | | Supervised | Examples already labelled with senses | Learns context → sense mapping | Labelling cost, limited rare-sense examples | | Semi-supervised | Small labelled set plus unlabelled data | Bootstraps from reliable seed examples | Early errors can spread | | Unsupervised sense induction | Unlabelled contextual occurrences | Clusters similar contexts | Clusters are not automatically named dictionary senses | | Neural/contextual | Learned word/context representations | Predicts or compares a context representation | Needs suitable data/model and evaluation | **Bag-of-words features** record nearby words while ignoring exact positions. **Collocational features** preserve a positional phrase such as “river bank”, “bank account”, “chocolate bar” or “bar exam”. ### Why WSD matters Machine translation needs the correct target-language meaning; information retrieval should distinguish river-bank travel from banking services; text-to-speech may need different pronunciations for “bass”; information extraction should avoid connecting entities through the wrong relation. WSD: same spelling ke options mein context dekh ke sahi meaning select. ### M2 16 WSD and Lesk Scoring id : nlp-m2-16-lesk-wsd title : M2 16 WSD and Lesk Scoring title\_confirmed : true topic\_title : Word-sense disambiguation and Lesk overlap scoring mode : m2 kind : focused status : available duration : 10:17 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Lesk algorithm, from words to overlap scores Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-wsd--u5-lesk/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-wsd/u5-lesk Unit 5 · PDF pages 23–48 **Lesk is a knowledge-based WSD method that chooses senses whose dictionary descriptions have the greatest overlap with contextual evidence.** Original Lesk compares definitions of neighbouring words' senses. A common simplified version compares each target-sense gloss directly with the sentence context. State which version you use. ### Simplified Lesk procedure 1. Remove the target word from the context; tokenize and normalize the remaining content words. 2. Obtain the gloss for each candidate sense and apply the same normalization. 3. Count shared words between the context and each gloss. 4. Select the sense with maximum overlap. If scores tie, report the tie or use a specified tie-break rule. $$ \operatorname{score}(s)=|C\cap G_s|,\qquad s^*=\arg\max_s\operatorname{score}(s) $$ $$ C $$ is the set of context words, $$ G_s $$ the set of words in sense $$ s $$ 's gloss, $$ \cap $$ means common elements, $$ |\cdot| $$ means count, and $$ \arg\max $$ means “the sense giving the largest score”. This version counts unique matching words once. **Question. Use simplified Lesk to disambiguate “bank” in “The river water rose near the bank.” Use C = {river, water, rose, near}; financial gloss = {institution, money, deposit}; river gloss = {land, river, water, edge}.** #### Given The normalized context and two gloss sets are given. Count exact shared words. #### To find The score of each sense and the selected meaning. #### Formula and why it applies Use $$ \operatorname{score}(s)=|C\cap G_s| $$ because the task specifies context–gloss overlap. #### Step 1. Compare the financial sense No word occurs in both {river, water, rose, near} and {institution, money, deposit}. Overlap = {}. #### Step 2. Count the financial score $$ \operatorname{score}(F)=0 $$ . #### Step 3. Compare the river sense Shared words are {river, water}. “Rose” and “near” are not in the gloss. #### Step 4. Count and select $$ \operatorname{score}(R)=2 $$ . Since $$ 2>0 $$ , choose the river-edge sense. **Answer.** Bank means land beside a river. **Question. Use simplified Lesk for “The dog barked loudly.” Treat barked as bark. Context C = {dog, loudly}; sound gloss = {sharp, cry, dog}; tree gloss = {outer, covering, tree}; ship gloss = {sailing, ship}.** #### Given Three glosses and normalized context {dog, loudly}. #### To find Select the sense of bark. #### Formula and why it applies $$ \operatorname{score}(s)=|C\cap G_s| $$ . Lemmatization lets barked match the target lemma bark; it does not create additional overlap words. #### Step 1. Sound sense Intersection = {dog}; score = 1. #### Step 2. Tree sense Intersection = {}; score = 0. #### Step 3. Ship sense Intersection = {}; score = 0. #### Step 4. Compare The sound sense has the unique maximum: 1 > 0. **Answer.** Bark is the dog's cry. **Question. Apply definition–definition Lesk to “pine cone”. Pine senses: P1 = {evergreen, tree, needle, leaf}; P2 = {waste, sorrow, illness}. Cone senses: C1 = {solid, shape, circular}; C2 = {fruit, evergreen, tree}.** #### Given Two candidate senses for each word. Tokens are already normalized; tree and trees would be reduced to the same form if necessary. #### To find The most compatible pair of senses. #### Formula and why it applies Use $$ |G_{P_i}\cap G_{C_j}| $$ for every pair because this is original-style gloss–gloss comparison. #### Step 1. P1 with C1 No shared word → 0. #### Step 2. P1 with C2 Shared {evergreen, tree} → 2. #### Step 3. P2 with each cone sense P2–C1 has no overlap → 0; P2–C2 has no overlap → 0. #### Step 4. Choose the best pair The four scores are 0, 2, 0, 0. Maximum 2 belongs to P1–C2. **Answer.** Pine is the evergreen tree; cone is its fruit-like reproductive structure. **Limitations:** exact overlap misses synonyms, short glosses may produce zeros, and different preprocessing choices change the score. Extended Lesk may include related synset definitions or longer phrase overlaps; do not silently substitute it for a simple question. Lesk: context aur dictionary mein common words gino; sabse bada overlap jeeta. ### M2 16 WSD and Lesk Scoring id : nlp-m2-16-lesk-wsd title : M2 16 WSD and Lesk Scoring title\_confirmed : true topic\_title : Word-sense disambiguation and Lesk overlap scoring mode : m2 kind : focused status : available duration : 10:17 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Thesaurus / Walker approach, with three complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-wsd--u5-walker/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-wsd/u5-walker Unit 5 · PDF pages 23–48 **The Walker approach uses thesaurus categories as semantic evidence.** Associate each candidate sense with a category. Count or weight context words that support that category. Choose the largest total. $$ \operatorname{score}(s)=\sum_{w\in C} I(w,\operatorname{category}(s)) $$ $$ I=1 $$ if the supplied category mapping says the context word supports the sense, otherwise $$ I=0 $$ . The summation symbol means add all those contributions. Weighted questions may replace 1 with a supplied weight. Category membership must be supplied or justified; it is not guessed from word frequency. **Question. Disambiguate bank using Walker scoring. Context words are money, interest, annum, fetch. The finance category supports money, interest and annum; the location category supports none of these words.** #### Given Two categories: Finance and Location. Each supported word contributes 1. #### To find The two totals and selected bank sense. #### Formula and why it applies Sum the category indicators for each sense. #### Step 1. Make a contribution table | Word | Finance | Location | | --- | --- | --- | | money | 1 | 0 | | interest | 1 | 0 | | annum | 1 | 0 | | fetch | 0 | 0 | #### Step 2. Add Finance $$ 1+1+1+0=3 $$ . #### Step 3. Add Location $$ 0+0+0+0=0 $$ . #### Step 4. Select 3 > 0, so select the financial institution. **Answer.** Finance, with score 3 versus 0. **Question. Use Walker scoring for “The fisherman caught bass beside the lake.” Context {fisherman,caught,lake}. The Fish category contains all three clues; the Music category contains none.** #### Given Fish and Music senses, binary category membership given. #### To find The correct bass sense. #### Formula and why it applies Score = sum of context-category matches. #### Step 1. Record Fish matches fisherman → 1; caught → 1; lake → 1. #### Step 2. Add Fish score $$ 1+1+1=3 $$ . #### Step 3. Add Music score $$ 0+0+0=0 $$ . #### Step 4. Choose Fish has the larger score. **Answer.** Bass refers to a fish. **Question. Score “The player used a bat in the match.” Context {player,match}. Sport supports both words with weights 2 and 1; Animal supports neither. Use the supplied weighted Walker scheme.** #### Given Sport weights {player:2, match:1}; Animal weights {player:0, match:0}. #### To find Weighted sense scores. #### Formula and why it applies Use the sum of supplied weights, not an unweighted count, because the question explicitly gives weights. #### Step 1. Sport contribution of player Add 2. #### Step 2. Sport contribution of match Add 1. Sport total = 2 + 1 = 3. #### Step 3. Animal total 0 + 0 = 0. #### Step 4. Compare 3 > 0 selects the sporting implement. **Answer.** Bat is the sports implement. Walker: dictionary ke exact words nahi, context kis topic/category ko support karta hai woh count. ### M2 17 — Walker category scoring and graph-based WSD id : nlp-m2-17-walker-graph-wsd title : M2 17 — Walker category scoring and graph-based WSD title\_confirmed : false topic\_title : Walker category scoring and graph-based WSD mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Random-walk WSD and supervised alternatives Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-wsd--u5-graph/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-wsd/u5-graph Unit 5 · PDF pages 23–48 **Graph-based WSD represents candidate senses as vertices and semantic relationships as weighted edges.** A random walk repeatedly transfers score along these edges. Senses supported by other strongly connected context senses tend to obtain higher ranks. 1. Create a vertex for every candidate sense of every relevant word. 2. Connect semantically related senses; similarity or gloss overlap can set edge weights. 3. Normalize outgoing edge weights into transition probabilities. 4. Run a graph-ranking method such as PageRank. 5. For each target word, choose its highest-ranked candidate sense. church context → bell: instrument sense ↔ ring: sound sense → mutually supported senses receive rank One common rank update is $$ r_{t+1}(j)=\frac{1-d}{N}+d\sum_i r_t(i)P(i\rightarrow j). $$ $$ r_t(i) $$ is current score of vertex $$ i $$ , $$ P(i\to j) $$ is its outgoing probability to $$ j $$ , $$ N $$ is the number of vertices, and $$ d $$ is the probability of following a graph edge. The remaining probability $$ 1-d $$ gives a uniform restart here. A personalized variant uses a nonuniform restart. The course requires the graph method concept; numerical details depend on supplied ranking rules. **Question. A two-vertex toy sense graph has transition matrix rows A:[0.2,0.8], B:[0.4,0.6]. Starting scores are A=0.5, B=0.5. Find one random-walk update with no restart.** #### Given Rows denote source vertices, columns destination vertices. d = 1. #### To find New scores at A and B. #### Formula and why it applies Use $$ r_{t+1}(j)=\sum_i r_t(i)P(i\to j) $$ : incoming contributions are added. #### Step 1. Contribution to A from A $$ 0.5\times0.2=0.1 $$ . #### Step 2. Contribution to A from B $$ 0.5\times0.4=0.2 $$ , hence $$ r_1(A)=0.1+0.2=0.3 $$ . #### Step 3. Contribution to B from A and B $$ 0.5\times0.8=0.4 $$ ; $$ 0.5\times0.6=0.3 $$ . #### Step 4. Add B and verify total $$ r_1(B)=0.4+0.3=0.7 $$ ; $$ 0.3+0.7=1 $$ . **Answer.** A=0.3, B=0.7 after one update. This is one iteration, not a converged PageRank result. ### Other approaches explained without assuming ML knowledge **A supervised classifier** learns from examples such as “cash … bank” labelled FINANCE and “river … bank” labelled RIVER. A decision list applies ranked context rules; a decision tree branches on clues; an SVM separates feature representations; a neural model learns contextual patterns. Naive Bayes, developed next, multiplies class-specific context probabilities. **Semi-supervised bootstrapping** begins with a few labelled examples or high-confidence rules, labels suitable unlabelled cases, and retrains. **Unsupervised clustering** groups usages by context similarity without gold sense labels; an additional mapping is needed to name the discovered groups. Graph WSD: meanings ko jodo; jo meaning context ke strong meanings se supported hai uska rank badhega. ### M2 17 — Walker category scoring and graph-based WSD id : nlp-m2-17-walker-graph-wsd title : M2 17 — Walker category scoring and graph-based WSD title\_confirmed : false topic\_title : Walker category scoring and graph-based WSD mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Prior, likelihood and posterior—before the formula Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-naive-bayes--u5-bayes-meaning/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-naive-bayes/u5-bayes-meaning Unit 5 · PDF pages 49–57 **A probability is a number from 0 to 1 measuring how likely an event is under a stated model.** A class probability of $$ 3/6=0.5 $$ means 3 of 6 training examples belong to that class. **Conditional probability** means probability inside a restricted group. $$ P(\text{loan}\mid F) $$ reads “probability of the word loan given the finance sense”. The vertical bar means “given”; it is not division. | Symbol | Name | Plain meaning | | --- | --- | --- | | $$ P(F) $$ | Prior | How common finance is before reading the new context | | $$ P(C\mid F) $$ | Likelihood | How compatible the observed context is with finance | | $$ P(F\mid C) $$ | Posterior | How likely finance is after seeing the context | | $$ P(C) $$ | Evidence | Total probability of this context under all senses | $$ P(F\mid C)=\frac{P(C\mid F)P(F)}{P(C)}. $$ Bayes' theorem reverses a conditional direction using the prior. Do not swap $$ P(F\mid C) $$ and $$ P(C\mid F) $$ : one asks about the sense, the other about the evidence. The **naive conditional independence assumption** says that, once the sense is fixed, the context-word features can be multiplied as if they were independent. It is an approximation: natural-language words are not actually independent in general. Training examples → count classes → Prior per sense → Count words within each sense → Likelihood of query clues → Multiply → compare → normalize if asked Prior = pehle kitna common; likelihood = clue kitna fit; posterior = clue dekhne ke baad meaning ki chance. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Multinomial Naive Bayes: the exact solving method Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-naive-bayes--u5-bayes-steps/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-naive-bayes/u5-bayes-steps Unit 5 · PDF pages 49–57 Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. ### Always keep these counts separate Class priors use **number of examples**. Word likelihoods in this multinomial model use **number of word occurrences**. The vocabulary counts **unique words across all classes**. 1. List the class examples and calculate priors. 2. Write one shared vocabulary and count its size. 3. Count total word occurrences in each class. 4. Count each query word in each class. 5. Apply the given smoothing rule. 6. Multiply each prior by all its query likelihoods. 7. Compare scores; divide each score by their sum only if a normalized posterior is requested. $$ P(s\mid C)=\frac{\operatorname{score}(s)}{\sum_{s'}\operatorname{score}(s')}. $$ **Why smoothing is needed:** an unseen clue gives $$ 0/N_s=0 $$ ; one zero makes the whole product zero. Add-one smoothing redistributes probability across every vocabulary word and prevents these zero factors. Use it only when requested, taught for the task, or state the assumption. **Different model warning:** a Bernoulli Naive Bayes question counts presence/absence per document and may include absence factors. Do not use its denominator in a multinomial word-count question. Pehle sentences se prior; phir words se likelihood; dono counts mix mat kar. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · Mam's bank numerical—every count and every multiplication Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-naive-bayes--u5-bayes-mam/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-naive-bayes/u5-bayes-mam Unit 5 · PDF pages 49–57 **Question. Classify the sense of “bank” using the training table below. The sentence is: “As the stream flowed gently past the shore, we set up camp by the bank of the river, discussing the possibility of borrowing a loan to buy a boat for future trips.” Use only the highlighted feature words stream, shore and loan, and add-one multinomial smoothing.** #### Given | Training row | Context words | Sense | | --- | --- | --- | | 1 | money, deposit, loan | Finance | | 2 | river, water, shore | Riverbank | | 3 | account, savings, investment | Finance | | 4 | boat, river, stream | Riverbank | | 5 | branch, deposit, check | Finance | | 6 | fishing, shore, river | Riverbank | Query feature set: {stream, shore, loan}. Other sentence words are not included as features in this supplied example. #### To find The most probable bank sense and normalized posteriors. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Calculate priors Finance appears in 3 of the 6 rows; Riverbank appears in 3 of 6. $$ P(F)=\frac36=\frac12,\qquad P(R)=\frac36=\frac12. $$ #### Step 2. Construct the vocabulary {money, deposit, loan, river, water, shore, account, savings, investment, boat, stream, fishing, branch, check}. Thus V=14. Repeated deposit or river is counted once in vocabulary. #### Step 3. Count word occurrences per class There are 3 finance rows × 3 words = 9 word occurrences. There are 3 river rows × 3 words = 9. Thus $$ N_F=N_R=9 $$ , and each smoothed denominator is $$ 9+14=23 $$ . #### Step 4. Count the three query words | Word | Finance count | Riverbank count | | --- | --- | --- | | stream | 0 | 1 | | shore | 0 | 2 | | loan | 1 | 0 | #### Step 5. Calculate finance likelihoods $$ P(\text{stream}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{shore}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{loan}\mid F)=\frac{1+1}{23}=\frac2{23} $$ #### Step 6. Multiply the finance score $$ \operatorname{score}(F)=\frac12\times\frac1{23}\times\frac1{23}\times\frac2{23}=\frac1{23^3}=\frac1{12167}\approx0.00008219. $$ #### Step 7. Calculate river likelihoods $$ P(\text{stream}\mid R)=\frac{1+1}{23}=\frac2{23} $$ $$ P(\text{shore}\mid R)=\frac{2+1}{23}=\frac3{23} $$ $$ P(\text{loan}\mid R)=\frac{0+1}{23}=\frac1{23} $$ #### Step 8. Multiply the river score $$ \operatorname{score}(R)=\frac12\times\frac2{23}\times\frac3{23}\times\frac1{23}=\frac3{12167}\approx0.00024657. $$ #### Step 9. Compare scores 3/12167 is larger than 1/12167. Riverbank wins even though “loan” is financial evidence: the combined clues matter. #### Step 10. Normalize if a posterior is requested $$ P(F\mid C)=\frac{1/12167}{4/12167}=\frac14=0.25 $$ $$ P(R\mid C)=\frac{3/12167}{4/12167}=\frac34=0.75. $$ **Answer.** Bank is the riverbank sense. Scores are 1/12167 and 3/12167; normalized model posteriors are 0.25 and 0.75. **Without smoothing:** stream and shore are absent in Finance; loan is absent in Riverbank. Both unsmoothed scores become zero, so there is no valid winning class from those products. Also, unnormalized scores should not be labelled as final posterior probabilities. Zero word? Add-one: count+1 upar, total words+vocabulary neeche. Last mein score bada wala sense. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · More WSD numericals: given likelihoods, unequal priors and repeated clues Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/read/u5-naive-bayes--u5-bayes-practice/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/read/u5-naive-bayes/u5-bayes-practice Unit 5 · PDF pages 49–57 **Question. For bass, let P(Fish)=0.6 and P(Music)=0.4. Given Fish likelihoods P(caught|Fish)=0.5, P(lake|Fish)=0.4 and Music likelihoods P(caught|Music)=0.05, P(lake|Music)=0.1, classify context {caught,lake} and normalize.** #### Given Priors and likelihoods are provided directly; no smoothing or training-count estimation is requested. #### To find The winning sense and its posterior. #### Formula and why it applies $$ \operatorname{score}(s)=P(s)P(\text{caught}\mid s)P(\text{lake}\mid s) $$ , because the two features are treated as conditionally independent. #### Step 1. Fish multiplication $$ 0.6\times0.5=0.3 $$ , then $$ 0.3\times0.4=0.12 $$ . #### Step 2. Music multiplication $$ 0.4\times0.05=0.02 $$ , then $$ 0.02\times0.1=0.002 $$ . #### Step 3. Total evidence $$ 0.12+0.002=0.122 $$ . #### Step 4. Normalize $$ P(Fish\mid C)=0.12/0.122\approx0.983607 $$ $$ P(Music\mid C)=0.002/0.122\approx0.016393. $$ **Answer.** Fish, with model posterior approximately 0.983607. **Question. Disambiguate bank from the repeated context tokens {loan, loan, river}. Priors are Finance=2/3 and River=1/3. Vocabulary V=4. Finance has N=8 total tokens, loan count 3 and river count 0; River has N=4, loan count 0 and river count 2. Apply add-one smoothing.** #### Given Multinomial counts are supplied. The two appearances of loan must both contribute. #### To find Scores and normalized Finance posterior. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Finance denominator $$ 8+4=12 $$ . $$ P(loan\mid F)=(3+1)/12=1/3 $$ ; $$ P(river\mid F)=(0+1)/12=1/12 $$ . #### Step 2. River denominator $$ 4+4=8 $$ . $$ P(loan\mid R)=(0+1)/8=1/8 $$ ; $$ P(river\mid R)=(2+1)/8=3/8 $$ . #### Step 3. Finance product $$ \operatorname{score}(F)=\frac23\times\frac13\times\frac13\times\frac1{12}=\frac1{162}\approx0.00617284. $$ #### Step 4. River product $$ \operatorname{score}(R)=\frac13\times\frac18\times\frac18\times\frac38=\frac1{512}\approx0.001953125. $$ #### Step 5. Compare and normalize $$ P(F\mid C)=\frac{1/162}{1/162+1/512}=\frac{512}{674}=\frac{256}{337}\approx0.759644. $$ **Answer.** Finance wins. Repeated loan contributes a squared likelihood; do not reduce the query to a unique-word set. ### Answer-writing order in an exam Write the class priors, vocabulary size, class token totals, a likelihood table, both score products and the final class. Keep fractions until the comparison where possible. If the question only asks classification, normalization is optional; if it asks posterior probabilities, normalization is required. Repeated word aaye toh factor bhi repeat. Sirf word-set bana diya toh multinomial answer badal jayega. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## M2 · The M2 formula sheet Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/formula-index--all/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/formula-index/all M2 · formulas with meaning · full examples follow Read each formula with its meaning. The remaining lessons in this tab contain the complete worked examples for these methods. Use the same vocabulary, boundary, case and token-count conventions throughout one solution. ### N-gram count $$ \max(T-n+1,0) $$ Unpadded sequence of T tokens. Count only valid contiguous windows. ### Unigram MLE $$ P(w)=C(w)/N $$ N is the training-token total under the stated boundary convention. ### Bigram MLE $$ P(w_i\mid w_{i-1})=C(w_{i-1},w_i)/C(w_{i-1}) $$ Denominator = history count. ### Trigram MLE $$ P(w_i\mid w_{i-2},w_{i-1})=C(w_{i-2},w_{i-1},w_i)/C(w_{i-2},w_{i-1}) $$ Denominator = two-word history count. ### Add-one / Laplace $$ P(w\mid h)=\frac{C(hw)+1}{C(h)+|V|} $$ Add one to every possible vocabulary continuation. ### Sentence probability $$ P(W)\approx\prod_iP(w_i\mid w_{i-1}) $$ Multiply in order; include an end factor if specified. ### Perplexity $$ PP=P(W)^{-1/N} $$ N is the number of predicted tokens/events used by the question. ### Emission $$ P(w\mid t)=C(w,t)/C(t) $$ Word given tag; count tokens with that tag. ### Transition $$ P(t_j\mid t_i)=C(t_i\to t_j)/\sum_k C(t_i\to t_k) $$ Next tag given previous tag; count outgoing transitions. ### HMM joint path $$ P(W,T)=\pi_{t_1}b_{t_1}(w_1)\prod_{i=2}^{n}a_{t_{i-1},t_i}b_{t_i}(w_i) $$ Multiply initial, emission and transition factors; add end probability if used. ### Tag sequences $$ K^T $$ K tags, T word positions; count before constraints. ### Viterbi $$ \delta_i(t)=b_t(w_i)\max_s[\delta_{i-1}(s)a_{s,t}] $$ Choose the largest predecessor score; store a backpointer. ### Lesk $$ score(s)=|context\cap gloss(s)| $$ Use the specified normalisation and overlap convention. ### Naive Bayes class prior $$ P(c)=D_c/D $$ D counts labelled examples; this is not the word-token total. ### Smoothed NB likelihood $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ N\_c counts training tokens in class c. ### NB decision $$ \hat c=\arg\max_c P(c)\prod_iP(w_i\mid c) $$ Select label with largest score; normalise only for posterior values. **Yaad rakho:** Condition jis par hai, denominator usi ka; sequence mein multiply; options mein maximum. --- ## M2 · Unigrams, bigrams and trigrams: construct them correctly Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-counts--n-grams-construction/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-counts/n-grams-construction Formulas explained through full worked examples **Definition.** An n-gram is a contiguous sequence of n items. For word n-grams, the items are consecutive word tokens. A unigram contains one word; a bigram contains two; a trigram contains three. Character n-grams use characters instead, so always identify the unit being counted. Tokens: we | study | natural | language → Move a window one token at a time → Bigrams: we study / study natural / natural language For m tokens without padding, the number of n-gram *positions* is max(0, m−n+1). Repeated positions may contain the same n-gram type. A bag-of-n-grams representation counts each n-gram type and loses the overall sentence order beyond the local order inside each n-gram. $$ \text{Number of unpadded n-gram positions}=\max(0,m-n+1) $$ **Question. List all word unigrams, bigrams and trigrams for “the prime minister of our country”. Do not add boundary tokens.** #### Given Six tokens: the, prime, minister, of, our, country. n = 1, 2 and 3. #### To find All three lists and their counts. #### Formula and why it applies $$ \#\text{positions}=m-n+1 $$ Each window must contain consecutive words and move one position. No wraparound or invented final word is allowed. #### Solution ##### Step 1. Take one word at a time Unigrams: (the), (prime), (minister), (of), (our), (country). Count = 6−1+1 = 6. ##### Step 2. Take two consecutive words Bigrams: (the, prime), (prime, minister), (minister, of), (of, our), (our, country). Count = 6−2+1 = 5. ##### Step 3. Take three consecutive words Trigrams: (the, prime, minister), (prime, minister, of), (minister, of, our), (of, our, country). Count = 6−3+1 = 4. **Final answer.** 6 unigrams, 5 bigrams and 4 trigrams. **Question. Construct bigrams for “I like tea” using one start and one end marker.** #### Given Original tokens: I, like, tea. Add and once each. #### To find The boundary-aware bigram sequence. #### Formula and why it applies $$ \#\text{bigrams}=(m+2)-1=m+1 $$ The start context predicts the first word, and the last word predicts that the sentence ends. #### Solution ##### Step 1. Write the padded sequence I like tea ##### Step 2. Move a two-token window (, I), (I, like), (like, tea), (tea, ). ##### Step 3. Count the windows $$ 3+1=4 $$ **Final answer.** There are four boundary-aware bigrams. **Question. Construct trigrams for “I like tea” using two start markers and one end marker.** #### Given Original sentence has three words. For a trigram model each predicted token needs two prior context tokens. #### To find All trigram prediction events. #### Formula and why it applies $$ P(w_i\mid w_{i-2},w_{i-1}) $$ Two start markers provide a complete context even for the first word. #### Solution ##### Step 1. Pad with two starts I like tea ##### Step 2. Slide a three-token window (, , I), (, I, like), (I, like, tea), (like, tea, ). ##### Step 3. Count predicted events The model predicts I, like, tea and the end marker: four predictions. **Final answer.** Four trigrams. Start markers are context; they are not ordinary predicted words. **Question. Give the bag-of-bigrams counts for “a b a b” without padding.** #### Given Four tokens in their given order. #### To find Bigram positions, distinct bigram types and their counts. #### Formula and why it applies $$ \#\text{positions}=4-2+1=3 $$ A bag stores frequencies; repeated bigrams must not be discarded. #### Solution ##### Step 1. Enumerate all positions (a,b), (b,a), (a,b). ##### Step 2. Group identical pairs | Bigram type | Count | | --- | --- | | a b | 2 | | b a | 1 | **Final answer.** Three bigram positions but two distinct bigram types. **Yaad rakho:** n-gram mein n words; context mein n−1 words. --- ## M2 · Unigram estimation and complete worked examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-counts--unigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-counts/unigram-estimation Formulas explained through full worked examples **Maximum likelihood estimation (MLE)** chooses probabilities equal to observed relative frequencies. In a unigram model, every word is predicted independently of its surrounding words. If a word occurs c times in N training tokens, its estimate is c/N. To score m specified words, multiply their m unigram probabilities. $$ P_{\mathrm{MLE}}(w)=\frac{C(w)}{N}\qquad P(w_1,\ldots,w_m)\approx\prod_{i=1}^{m}P(w_i) $$ Choose a tokenization convention before counting. Lowercasing, punctuation handling and boundary markers can change the vocabulary and counts. Never count words with one convention and form the denominator with another. A unigram model gives the same product to rearrangements of the same word multiset; that is one of its limitations. **Question. Train an unpadded, case-insensitive unigram model on “I like tea” and “I like coffee”. Find the probability of “I like tea”.** #### Given Training sentences: I like tea; I like coffee. Use lowercase words; exclude punctuation and sentence boundaries. #### To find The vocabulary, word probabilities and test-sentence probability. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P(I\ like\ tea)=P(I)P(like)P(tea) $$ The unigram approximation ignores order and uses total token count for every denominator. #### Solution ##### Step 1. Count every training token Each sentence contains three words, so N = 3+3 = 6. ##### Step 2. Count types and frequencies | Word | Count | MLE probability | | --- | --- | --- | | i | 2 | 2/6 = 1/3 | | like | 2 | 2/6 = 1/3 | | tea | 1 | 1/6 | | coffee | 1 | 1/6 | Vocabulary size V = 4. ##### Step 3. Write one factor per test word $$ P=\frac{2}{6}\times\frac{2}{6}\times\frac{1}{6} $$ ##### Step 4. Multiply the fractions $$ P=\frac{4}{216}=\frac{1}{54}\approx0.018519 $$ **Final answer.** The unigram sentence probability is approximately 0.018519. **Question. A corpus contains counts a: 4, b: 3 and c: 1. Find the unigram probability of “a b a”.** #### Given C(a)=4, C(b)=3, C(c)=1. No boundary markers. #### To find P(a b a). #### Formula and why it applies $$ P(a\,b\,a)=P(a)P(b)P(a) $$ A repeated test word contributes a probability factor at every occurrence; do not include it just once. #### Solution ##### Step 1. Find the total $$ N=4+3+1=8 $$ ##### Step 2. Estimate the required probabilities $$ P(a)=4/8=1/2,\qquad P(b)=3/8 $$ ##### Step 3. Use both occurrences of a $$ P=\frac12\times\frac38\times\frac12 $$ ##### Step 4. Simplify $$ P=\frac3{32}=0.09375 $$ **Final answer.** P(a b a) = 0.09375. **Yaad rakho:** Repeated word ka probability bhi repeat multiply hoga. **Question. An unsmoothed unigram model is trained on “red blue red”. Find the probability of “red green”.** #### Given N=3. Counts: red=2, blue=1, green=0. The test contains red and green. #### To find The unsmoothed test probability and the reason for its value. #### Formula and why it applies $$ P(red\ green)=P(red)P(green) $$ MLE assigns zero to an unseen outcome. Smoothing is needed if an unseen in-vocabulary item must receive probability. #### Solution ##### Step 1. Estimate red $$ P(red)=2/3 $$ ##### Step 2. Estimate the unseen word $$ P(green)=0/3=0 $$ ##### Step 3. Multiply $$ (2/3)\times0=0 $$ **Final answer.** Unsmoothed probability = 0. This reflects missing training evidence, not proof that the phrase is impossible. --- ## M2 · Bigram counts: build the table before multiplying Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-counts--bigram-count-table/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-counts/bigram-count-table Formulas explained through full worked examples A **bigram probability table** uses the previous word as its row and the next word as its column. Divide each cell count by its row's total outgoing count. With one end marker on every sentence, each ordinary training word has a continuation, so its outgoing count equals its token frequency. Do not create a bigram that crosses from one sentence into the next. $$ P(w\mid h)=\frac{C(h,w)}{\sum_{v\in V}C(h,v)}=\frac{C(h,w)}{C(h)} $$ Add boundaries to each sentence separately → List adjacent pairs → Count repeated pairs → Divide each row by its total → Follow the test sentence through the table **Question. Train a bigram model on “I like tea”, “I like coffee” and “you like tea”. Use one start and one end marker. Find the probability of “I like tea”.** #### Given Corpus: I like tea ; I like coffee ; you like tea . Preserve these tokens. #### To find All nonzero bigram estimates and the test probability. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Every sentence starts and ends independently. Four probability factors are required for the three test words plus end. #### Solution ##### Step 1. List the training pairs Sentence 1: (,I), (I,like), (like,tea), (tea,). Sentence 2: (,I), (I,like), (like,coffee), (coffee,). Sentence 3: (,you), (you,like), (like,tea), (tea,). ##### Step 2. Create the nonzero count table | History → next | Pair count | History count | Probability | | --- | --- | --- | --- | | → I | 2 | 3 | 2/3 | | → you | 1 | 3 | 1/3 | | I → like | 2 | 2 | 1 | | you → like | 1 | 1 | 1 | | like → tea | 2 | 3 | 2/3 | | like → coffee | 1 | 3 | 1/3 | | tea → | 2 | 2 | 1 | | coffee → | 1 | 1 | 1 | ##### Step 3. Write the test path $$ P=P(I\mid\langle s\rangle)P(like\mid I)P(tea\mid like)P(\langle/s\rangle\mid tea) $$ ##### Step 4. Substitute each matching row $$ P=\frac23\times1\times\frac23\times1 $$ ##### Step 5. Multiply $$ P=\frac49\approx0.444444 $$ **Final answer.** P(I like tea) = 4/9 ≈ 0.444444. **Question. Using the same three-sentence corpus given here—“I like tea”, “I like coffee”, “you like tea”—find the bigram probabilities of “you like coffee” and “tea like I”. Include boundaries.** #### Given Corpus with one start/end marker per sentence. Counts: start→you=1/3; you→like=1; like→coffee=1/3; coffee→end=1. No training sentence starts with tea. #### To find The two test probabilities. #### Formula and why it applies $$ P(\text{sentence})=\prod P(\text{next token}\mid\text{previous token}) $$ A model can assign a nonzero probability to a sentence it never saw if all of its local bigrams were seen. #### Solution ##### Step 1. Score you like coffee $$ P=\frac13\times1\times\frac13\times1=\frac19\approx0.111111 $$ ##### Step 2. Inspect tea like I $$ P(tea\mid\langle s\rangle)=0/3=0 $$ ##### Step 3. Apply the zero factor The full product is zero, regardless of the remaining factors. **Final answer.** P(you like coffee) = 1/9; P(tea like I) = 0. A new sentence is not necessarily a zero-probability sentence. **Question. Obtain the unsmoothed bigram probability of “students are from Vellore” from this corpus: “I am from Vellore”; “I am a teacher”; “students are good and are from various cities”; “students from Vellore do engineering”. Use sentence boundaries.** #### Given Four training sentences, each padded with one and . Test sequence: students are from Vellore . #### To find The five required conditional probabilities and their product. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Count all occurrences in the entire corpus. In particular, are occurs twice inside the third sentence, and Vellore is followed by end once and do once. #### Solution ##### Step 1. Count the histories | History | Occurrences with continuation | | --- | --- | | | 4 | | students | 2 | | are | 2 | | from | 3 | | Vellore | 2 | ##### Step 2. Count the required pairs | Required pair | Count | Probability | | --- | --- | --- | | students | 2 | 2/4 | | students are | 1 | 1/2 | | are from | 1 | 1/2 | | from Vellore | 2 | 2/3 | | Vellore | 1 | 1/2 | ##### Step 3. Write the complete product $$ P=\frac24\times\frac12\times\frac12\times\frac23\times\frac12 $$ ##### Step 4. Multiply progressively $$ \frac24\times\frac12=\frac14,\quad \frac14\times\frac12=\frac18 $$ $$ \frac18\times\frac23=\frac1{12},\quad \frac1{12}\times\frac12=\frac1{24} $$ **Final answer.** P = 1/24 ≈ 0.0416667. **Yaad rakho:** Same sentence mein word do baar aaye toh count bhi do. --- ## M2 · Trigram estimates: two-word history, not two denominators Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-counts--trigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-counts/trigram-estimation Formulas explained through full worked examples A trigram model predicts the next word from the two preceding words. The numerator is the count of a triple. The denominator is the count of its two-word history with an available next token. Do not multiply two independent bigram probabilities and call that a trigram probability. $$ P(w_i\mid w_{i-2},w_{i-1})=\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})} $$ **Question. A corpus contains 12 occurrences of “prime minister”. Eight are followed by “of”, three by “said”, and one by “visited”. Estimate the three trigram probabilities.** #### Given History h = (prime, minister). Count of this history = 12. Triple counts for of, said, visited are 8, 3, 1. #### To find The conditional distribution of the next word. #### Formula and why it applies $$ P(w\mid prime,minister)=C(prime,minister,w)/12 $$ All three candidates share one two-word history, so they share denominator 12. #### Solution ##### Step 1. Calculate of $$ P(of\mid prime,minister)=8/12=2/3 $$ ##### Step 2. Calculate said $$ P(said\mid prime,minister)=3/12=1/4 $$ ##### Step 3. Calculate visited $$ P(visited\mid prime,minister)=1/12 $$ ##### Step 4. Confirm normalization $$ 8/12+3/12+1/12=12/12=1 $$ **Final answer.** The probabilities are 2/3, 1/4 and 1/12. **Question. Train a trigram model on “I like tea”, “I like coffee” and “you like tea”. Use two start markers and one end marker. Score “I like tea”.** #### Given Three sentences. Padded first sentence is I like tea ; pad the other two in the same way. #### To find Four trigram prediction probabilities and the sentence probability. #### Formula and why it applies $$ P=\prod_i P(w_i\mid w_{i-2},w_{i-1}) $$ Each prediction uses a two-token context; the two start markers supply the first history. #### Solution ##### Step 1. Count the first-word event The history (,) occurs three times, and two are followed by I. $$ P(I\mid\langle s\rangle,\langle s\rangle)=2/3 $$ ##### Step 2. Count the second-word event (,I) occurs twice; both are followed by like. $$ P(like\mid\langle s\rangle,I)=2/2=1 $$ ##### Step 3. Count the third-word event (I,like) occurs twice; one is followed by tea. $$ P(tea\mid I,like)=1/2 $$ ##### Step 4. Count the end event (like,tea) occurs twice; both are followed by end. $$ P(\langle/s\rangle\mid like,tea)=2/2=1 $$ ##### Step 5. Multiply all four $$ P=\frac23\times1\times\frac12\times1=\frac13 $$ **Final answer.** Trigram P(I like tea) = 1/3. It differs from the bigram estimate because the context and denominator changed. **Question. A trigram model supplies P(we|,)=0.4, P(study|,we)=0.5, P(NLP|we,study)=0.8 and P(|study,NLP)=0.25. Score “we study NLP”.** #### Given Four trigram conditional factors. The end marker is included. #### To find The complete sentence probability. #### Formula and why it applies $$ P=P(we\mid s,s)P(study\mid s,we)P(NLP\mid we,study)P(end\mid study,NLP) $$ Use each supplied two-word history exactly; do not swap its word order. #### Solution ##### Step 1. Insert the values $$ P=0.4\times0.5\times0.8\times0.25 $$ ##### Step 2. Multiply the first pair $$ 0.4\times0.5=0.2 $$ ##### Step 3. Include NLP $$ 0.2\times0.8=0.16 $$ ##### Step 4. Include the end $$ 0.16\times0.25=0.04 $$ **Final answer.** P = 0.04. **Yaad rakho:** Trigram: teen-word count upar, pehle do-word count neeche. **Why larger n is not automatically better.** More context can capture local order more precisely, but each longer history occurs less often. Parameter count, storage and sparsity grow. A model trained on Shakespeare may assign poor probabilities to financial news because training and test distributions differ. Keep a held-out test set; do not evaluate only on memorized training sentences. --- ## M2 · Laplace smoothing: why both numerator and denominator change Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-smoothing--laplace-basics/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-smoothing/laplace-basics Formulas explained through full worked examples **Smoothing** reallocates some probability mass to events that received zero or unreliable estimates from finite training data. **Laplace, or add-one, smoothing** adds one pseudo-count to every possible next outcome. If there are V possible outcomes, the total added count is V, so adding 1 only to the numerator would produce probabilities that do not sum to 1. $$ P_{\mathrm{Laplace}}(w)=\frac{C(w)+1}{N+V} $$ $$ P_{\mathrm{Laplace}}(w\mid h)=\frac{C(h,w)+1}{C(h)+V} $$ For unigram smoothing, N is the total training token count. For bigram smoothing, C(h) is the count of the particular history. In both cases V is the number of outcomes being predicted, not the number of training sentences and not the number of distinct bigrams. When end is a possible prediction, include it in V; the start marker is usually context only. Observed counts: 3, 1, 0 → Add one to every outcome: 4, 2, 1 → New total: 7 → Smoothed probabilities: 4/7, 2/7, 1/7 **Question. A fixed unigram vocabulary is {a,b,c}. The training counts are 3,1,0. Compute unsmoothed and Laplace-smoothed probabilities.** #### Given C(a)=3, C(b)=1, C(c)=0; N=4; V=3. #### To find The complete distributions. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P_L(w)=(C(w)+1)/(N+V) $$ The zero-count c is still in the fixed vocabulary, so it must receive one pseudo-count and contribute to V. #### Solution ##### Step 1. Compute unsmoothed values | Word | Unsmoothed | | --- | --- | | a | 3/4 = 0.75 | | b | 1/4 = 0.25 | | c | 0/4 = 0 | ##### Step 2. Calculate the new denominator $$ N+V=4+3=7 $$ ##### Step 3. Add one to each numerator $$ C(a)+1=4,\quad C(b)+1=2,\quad C(c)+1=1 $$ ##### Step 4. Divide | Word | Smoothed | | --- | --- | | a | 4/7 ≈ 0.571429 | | b | 2/7 ≈ 0.285714 | | c | 1/7 ≈ 0.142857 | ##### Step 5. Confirm the total $$ 4/7+2/7+1/7=1 $$ **Final answer.** Smoothed probabilities: a=4/7, b=2/7, c=1/7. Common a loses mass; unseen c gains mass. **Question. After history I, a corpus records like three times, study once and sleep zero times. The allowed next vocabulary is {like,study,sleep}. Compute all Laplace-smoothed bigram probabilities.** #### Given C(I,like)=3; C(I,study)=1; C(I,sleep)=0. History count C(I)=4. V=3. #### To find P\_L(like|I), P\_L(study|I), P\_L(sleep|I). #### Formula and why it applies $$ P_L(w\mid I)=\frac{C(I,w)+1}{C(I)+V} $$ This is smoothing within the I row; use its total 4, not the global corpus token count. #### Solution ##### Step 1. Form the common denominator $$ C(I)+V=4+3=7 $$ ##### Step 2. Calculate like $$ P_L(like\mid I)=(3+1)/7=4/7 $$ ##### Step 3. Calculate study $$ P_L(study\mid I)=(1+1)/7=2/7 $$ ##### Step 4. Calculate sleep $$ P_L(sleep\mid I)=(0+1)/7=1/7 $$ **Final answer.** The row is (4/7, 2/7, 1/7). An unseen bigram is no longer forced to zero. **Question. A fixed vocabulary has five possible next tokens. A history h is unseen in training. What is its add-one probability for each next token?** #### Given C(h)=0; C(h,w)=0 for each outcome; V=5. #### To find P\_L(w|h) for each token. #### Formula and why it applies $$ P_L(w\mid h)=\frac{0+1}{0+V} $$ All outcomes receive equal pseudo-counts because the training data gives no preference for this history. #### Solution ##### Step 1. Add the pseudo-count $$ 0+1=1 $$ ##### Step 2. Form the denominator $$ 0+5=5 $$ ##### Step 3. Divide and sum $$ P_L(w\mid h)=1/5=0.2,\qquad5(0.2)=1 $$ **Final answer.** Every outcome gets 0.2 under this fixed-vocabulary model. **Vocabulary warning.** Add-one does not automatically define probabilities for every word in the world. Unknown words require a declared vocabulary/UNK convention. If a question supplies V, use that stated V rather than inventing additional tokens. **Yaad rakho:** Add-one: upar +1, neeche +V; bigram mein neeche history ka count. --- ## M2 · A complete unseen-bigram question from corpus to final answer Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-smoothing--smoothed-sentence/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-smoothing/smoothed-sentence Formulas explained through full worked examples **Question. Consider the corpus “I love to code in Python. I love to learn new things. I code every day.” Obtain the bigram probabilities needed for “I love Python”. Explain the zero-probability issue, then use Laplace smoothing to score it. Treat each sentence separately, preserve case, remove punctuation, and include an end token among predicted outcomes.** #### Given Three sentences padded with one start/end marker. Distinct words: I, love, to, code, in, Python, learn, new, things, every, day. With , V=12. Start is context only. #### To find Unsmoothed and smoothed P(I love Python | ). #### Formula and why it applies $$ P(w\mid h)=\frac{C(h,w)}{C(h)},\qquad P_L(w\mid h)=\frac{C(h,w)+1}{C(h)+12} $$ The test contains an unseen transition love→Python. Smoothing assigns probability to that transition and renormalizes every required history row. #### Solution ##### Step 1. Count the contexts | History | Count | | --- | --- | | | 3 | | I | 3 | | love | 2 | | Python | 1 | ##### Step 2. Count each required pair | Pair | Count | | --- | --- | | I | 3 | | I love | 2 | | love Python | 0 | | Python | 1 | ##### Step 3. Compute the unsmoothed product $$ P=\frac33\times\frac23\times\frac02\times\frac11=0 $$ Both words love and Python were seen, but their adjacent pair was not. This is an unseen bigram, not an unknown-word case. ##### Step 4. Smooth the first event $$ P_L(I\mid\langle s\rangle)=\frac{3+1}{3+12}=\frac4{15} $$ ##### Step 5. Smooth the second event $$ P_L(love\mid I)=\frac{2+1}{3+12}=\frac3{15} $$ ##### Step 6. Smooth the unseen event $$ P_L(Python\mid love)=\frac{0+1}{2+12}=\frac1{14} $$ ##### Step 7. Smooth the end event $$ P_L(\langle/s\rangle\mid Python)=\frac{1+1}{1+12}=\frac2{13} $$ ##### Step 8. Multiply the smoothed factors $$ P_L=\frac4{15}\times\frac3{15}\times\frac1{14}\times\frac2{13} $$ $$ P_L=\frac{24}{40950}=\frac4{6825}\approx0.000586081 $$ **Final answer.** Unsmoothed P=0. With the stated V=12 and boundaries, Laplace-smoothed P≈0.000586081. If an exam states a different convention—for example excluding boundary tokens from the entire calculation—recompute both V and the set of probability factors consistently. Do not combine a boundary-aware numerator with a boundary-free denominator. --- ## M2 · Adjusted counts and relative discounting Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-smoothing--adjusted-count-discount/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-smoothing/adjusted-count-discount Formulas explained through full worked examples The slides also express smoothing through an **adjusted count** c\*. Instead of writing the smoothed probability with denominator N+V, they rescale its numerator so the denominator can remain N. Algebraically these are equivalent: $$ c^*=(c+1)\frac{N}{N+V},\qquad P_L(w)=\frac{c^*}{N} $$ $$ d=\frac{c^*}{c}\quad(c>0) $$ The relative discount d compares adjusted and original counts. It is undefined when c=0 because division by zero is undefined. Add-one smoothing moves the distribution toward uniformity: frequent events are reduced, but some rare seen events may increase. It is not correct to claim that every nonzero count must decrease under Laplace smoothing. **Question. For N=20 and V=5, a word has original count c=10. Find its adjusted count, smoothed probability and relative discount.** #### Given N=20; V=5; c=10. #### To find c\*, P\_L and d. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\quad P_L=c^*/N,\quad d=c^*/c $$ These expressions rewrite add-one smoothing without changing its probability. #### Solution ##### Step 1. Form the new total $$ N+V=20+5=25 $$ ##### Step 2. Find the adjusted count $$ c^*=(10+1)\frac{20}{25}=11(0.8)=8.8 $$ ##### Step 3. Find the probability $$ P_L=8.8/20=0.44 $$ ##### Step 4. Find the relative discount $$ d=8.8/10=0.88 $$ **Final answer.** c\*=8.8, P\_L=0.44, d=0.88. The original probability 0.5 is reduced. **Question. Under N=20, V=5, a rare word occurs once. Compute its adjusted count and compare the original and smoothed probabilities.** #### Given N=20; V=5; c=1. #### To find c\*, original P and smoothed P\_L. #### Formula and why it applies $$ c^*=(c+1)N/(N+V) $$ The same smoothing rule may increase a rare event's probability. #### Solution ##### Step 1. Original probability $$ P=1/20=0.05 $$ ##### Step 2. Adjusted count $$ c^*=(1+1)(20/25)=1.6 $$ ##### Step 3. Smoothed probability $$ P_L=1.6/20=0.08 $$ **Final answer.** c\*=1.6 and P\_L=0.08, which exceeds the original 0.05. **Question. For N=20 and V=5, an in-vocabulary word has count zero. Find its adjusted count and smoothed probability. Can a relative discount be calculated?** #### Given c=0; N=20; V=5. #### To find c\*, P\_L and whether d exists. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\qquad P_L=c^*/N $$ The pseudo-count creates positive mass for the unseen vocabulary item. #### Solution ##### Step 1. Adjusted count $$ c^*=(0+1)(20/25)=0.8 $$ ##### Step 2. Smoothed probability $$ P_L=0.8/20=0.04 $$ ##### Step 3. Relative discount d would be 0.8/0. That is undefined, so do not report a finite discount. **Final answer.** c\*=0.8 and P\_L=0.04; relative discount is undefined for c=0. --- ## M2 · Perplexity: meaning, formula and four complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-smoothing--evaluation-perplexity/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-smoothing/evaluation-perplexity Formulas explained through full worked examples **Intrinsic evaluation** evaluates a model directly on a held-out test corpus, for example using perplexity. **Extrinsic evaluation** places it inside a downstream application, such as a speech recognizer or translation system, and measures the application's performance. Extrinsic testing is closer to actual use but often slower and more expensive. A lower perplexity is not a guarantee of improvement on every downstream task. **Perplexity** is the inverse probability of a test sequence, normalized by the number of prediction events. Intuitively, it is the model's effective average number of equally plausible choices. A model that always gives the correct next word probability 1/4 has perplexity 4. Lower is better when the models are tested on the same data, vocabulary, tokenization and boundary convention. $$ \operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}} $$ | Symbol | Meaning | | --- | --- | | W | The complete test sequence. | | P(W) | Product of the probabilities assigned to the actual next tokens. | | N | Number of scored prediction events under the stated convention. | | Exponent −1/N | Take the reciprocal, then the Nth root. | For calculator entry, use parentheses: P raised to (−1 ÷ N), or (1 ÷ P) raised to (1 ÷ N). Retain several digits in intermediate products. The ordinary start marker is usually given context, not a predicted event; the end marker counts if its probability is included. **Question. A model assigns probability 0.0625 to a test sequence containing four scored tokens. Compute its perplexity.** #### Given P(W)=0.0625=1/16; N=4. #### To find PP(W). #### Formula and why it applies $$ \operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N} $$ The root removes the effect of sequence length; the reciprocal makes lower surprise correspond to lower perplexity. #### Solution ##### Step 1. Take the reciprocal $$ 1/P(W)=1/0.0625=16 $$ ##### Step 2. Use the fourth root $$ \operatorname{PP}=16^{1/4} $$ ##### Step 3. Evaluate $$ 2^4=16\quad\Rightarrow\quad\operatorname{PP}=2 $$ **Final answer.** Perplexity = 2. **Question. Two language models score the same three-token test sequence. Model A gives next-token probabilities 0.5, 0.5, 0.5. Model B gives 0.5, 0.25, 0.25. Compute their perplexities and select the better intrinsic result.** #### Given Same sequence and N=3 for both models. A factors: 0.5,0.5,0.5. B factors: 0.5,0.25,0.25. #### To find Both sequence probabilities and perplexities. #### Formula and why it applies $$ P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3} $$ A fair comparison uses identical test tokens and scoring conventions. #### Solution ##### Step 1. Calculate A's probability $$ P_A=0.5\times0.5\times0.5=0.125=1/8 $$ ##### Step 2. Calculate A's perplexity $$ \operatorname{PP}_A=8^{1/3}=2 $$ ##### Step 3. Calculate B's probability $$ P_B=0.5\times0.25\times0.25=0.03125=1/32 $$ ##### Step 4. Calculate B's perplexity $$ \operatorname{PP}_B=32^{1/3}\approx3.174802 $$ ##### Step 5. Compare 2 is lower than 3.174802; A places more probability on the observed sequence. **Final answer.** Model A is better by perplexity on this test: 2 versus about 3.174802. **Question. The bigram training corpus is “This is a dog”, “This is a cat”, “I love my cat”, “This is my name”. Score “This is my cat”, including an end marker, and compute perplexity using five predicted events.** #### Given Each training sentence has one start and end marker. Test predictions: This, is, my, cat, . Thus N=5. #### To find Bigram sentence probability and perplexity. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5} $$ Start is given context. Four words plus the end marker produce five probability factors. #### Solution ##### Step 1. Count the required transitions | Transition | Pair count / history count | | --- | --- | | →This | 3/4 | | This→is | 3/3 | | is→my | 1/3 | | my→cat | 1/2 | | cat→ | 2/2 | ##### Step 2. Form the product $$ P=\frac34\times1\times\frac13\times\frac12\times1=\frac18 $$ ##### Step 3. Take the reciprocal $$ 1/P=8 $$ ##### Step 4. Take the fifth root $$ \operatorname{PP}=8^{1/5}\approx1.515717 $$ **Final answer.** P(W)=1/8 and perplexity≈1.515717 using five predicted events. **If the question explicitly states N = 6:** use that supplied convention and report PP = 81/6 ≈ 1.414214. The class slide counts both boundary markers in N. Write the chosen N beside the formula; do not silently switch between 5 and 6. Also, This occurs three times and is follows it three times, so P(is|This)=3/3, not 3/4. **Question. A uniform model assigns probability 1/10 to each actual next word in a five-token test. Find its perplexity.** #### Given Five scored tokens, each with assigned probability 0.1. N=5. #### To find The sentence probability and perplexity. #### Formula and why it applies $$ P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5} $$ Uniform equal probabilities make the effective number of choices especially clear. #### Solution ##### Step 1. Multiply the five factors $$ P(W)=0.1^5=0.00001=10^{-5} $$ ##### Step 2. Invert $$ 1/P(W)=10^5 $$ ##### Step 3. Take the fifth root $$ \operatorname{PP}=(10^5)^{1/5}=10 $$ **Final answer.** Perplexity = 10. **Yaad rakho:** Probability high → surprise low → perplexity low. --- ## M2 · Log probabilities: replace long products with sums Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-smoothing--log-probabilities/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-smoothing/log-probabilities Formulas explained through full worked examples Probabilities are at most 1, so multiplying many of them produces very small numbers. Computers can round extremely small positive products to zero: **numerical underflow**. Log probabilities convert multiplication into addition and are more stable for long sequences. Natural logs use base e; log₂ uses base 2. Choose one base consistently. $$ \ln P(W)=\sum_{i=1}^{N}\ln p_i $$ $$ P(W)=e^{\ln P(W)},\qquad \operatorname{PP}(W)=\exp\left(-\frac1N\sum_i\ln p_i\right) $$ The log of a probability below 1 is negative. A value closer to zero means a larger probability: ln(0.5) ≈ −0.693 is larger than ln(0.1) ≈ −2.303. A zero probability has log −∞; taking logs does not solve the unseen-event problem. Smoothing handles that separate problem. **Question. The next-word probabilities are 0.5, 0.25 and 0.5. Calculate the natural log probability, ordinary probability and perplexity.** #### Given p₁=0.5; p₂=0.25; p₃=0.5; N=3. #### To find ln P(W), P(W), and PP. #### Formula and why it applies $$ \ln P=\sum_i\ln p_i,\quad P=e^{\ln P},\quad\operatorname{PP}=e^{-\ln P/N} $$ Logarithms preserve products as sums and the exponent converts the result back. #### Solution ##### Step 1. Compute individual logs $$ \ln0.5\approx-0.693147,\quad\ln0.25\approx-1.386294 $$ ##### Step 2. Add all three terms $$ \ln P=-0.693147-1.386294-0.693147\approx-2.772589 $$ ##### Step 3. Recover the probability $$ P=e^{-2.772589}\approx0.0625 $$ ##### Step 4. Find average negative log probability $$ -\ln P/3\approx2.772589/3=0.924196 $$ ##### Step 5. Exponentiate $$ \operatorname{PP}=e^{0.924196}\approx2.519842 $$ **Final answer.** ln P≈−2.772589, P=0.0625, PP≈2.519842. **Question. A model has a total natural log probability −12 over six scored tokens. Calculate its perplexity.** #### Given ln P(W)=−12; N=6. #### To find PP without forming the tiny ordinary probability. #### Formula and why it applies $$ \operatorname{PP}=\exp(-\ln P(W)/N) $$ The logarithmic formula directly uses the supplied log score. #### Solution ##### Step 1. Negate the log score $$ -\ln P=12 $$ ##### Step 2. Divide by token count $$ 12/6=2 $$ ##### Step 3. Apply exponential $$ \operatorname{PP}=e^2\approx7.389056 $$ **Final answer.** Perplexity≈7.389056. **Question. A language model's average negative log₂ probability is 3 bits per token. Find its perplexity.** #### Given Cross-entropy H=3 bits/token. Log base is 2. #### To find PP. #### Formula and why it applies $$ \operatorname{PP}=2^H $$ The inverse operation of log₂ is raising 2 to that power; use e only for natural logs. #### Solution ##### Step 1. Identify the log base Bits mean base 2. ##### Step 2. Raise the base to H $$ \operatorname{PP}=2^3=8 $$ **Final answer.** Perplexity = 8. **Yaad rakho:** ln use kiya toh e se wapas; log₂ use kiya toh 2 se wapas. --- ## M2 · Markov chains: read a transition table and multiply a path Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-hmm--markov-chain/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-hmm/markov-chain Formulas explained through full worked examples **A Markov chain** is a probabilistic model of a sequence of states. Under the first-order Markov assumption, the next state's probability depends on the current state rather than the complete previous history. The **initial probability** tells us how likely each first state is. A **transition probability** tells us how likely a move from one state to another is. $$ P(s_t\mid s_1,\ldots,s_{t-1})=P(s_t\mid s_{t-1}) $$ $$ P(s_1,\ldots,s_T)=P(s_1)\prod_{t=2}^{T}P(s_t\mid s_{t-1}) $$ For T states in a path, use one initial factor and T−1 transition factors. A row-oriented transition matrix must have each complete row sum to 1. Always read the row/column labels; some exam diagrams use a transposed arrangement. | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. **Question. For the Rain/Dry Markov chain with initial P(Rain)=0.4 and P(Dry)=0.6, and transitions Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8, find the probability of Dry, Dry, Rain, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Dry,Dry,Rain,Rain). #### Formula and why it applies $$ P(D,D,R,R)=P(D)P(D\mid D)P(R\mid D)P(R\mid R) $$ Each step conditions on the immediately preceding state. The sequence contains four states, so it needs one initial plus three transition factors. #### Solution ##### Step 1. Start in Dry $$ P(D)=0.6 $$ ##### Step 2. Move Dry to Dry $$ 0.6\times0.8=0.48 $$ ##### Step 3. Move Dry to Rain $$ 0.48\times0.2=0.096 $$ ##### Step 4. Move Rain to Rain $$ 0.096\times0.3=0.0288 $$ **Final answer.** Path probability = 0.0288. **Question. For initial probabilities Rain=0.4, Dry=0.6 and transition rows Rain=(0.3,0.7), Dry=(0.2,0.8) in column order Rain,Dry, find the probability of Rain, Dry, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Rain,Dry,Rain). #### Formula and why it applies $$ P(R,D,R)=P(R)P(D\mid R)P(R\mid D) $$ Do not replace Dry→Rain with Rain→Dry: the two directional probabilities differ. #### Solution ##### Step 1. Initial Rain $$ P(R)=0.4 $$ ##### Step 2. First transition $$ P(D\mid R)=0.7,\quad0.4(0.7)=0.28 $$ ##### Step 3. Second transition $$ P(R\mid D)=0.2,\quad0.28(0.2)=0.056 $$ **Final answer.** Path probability = 0.056. **Question. For the same fully specified chain—initial Rain=0.4, Dry=0.6; Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8—find P(Dry,Dry). Also find the conditional probability of next Dry given current Dry.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find The two-state joint probability and the one-step conditional probability. #### Formula and why it applies $$ P(D,D)=P(D)P(D\mid D) $$ The joint path includes an uncertain initial state. The conditional transition assumes the current state is already Dry. #### Solution ##### Step 1. Read the conditional transition $$ P(D\mid D)=0.8 $$ ##### Step 2. Include the initial state for the joint path $$ P(D,D)=0.6\times0.8=0.48 $$ **Final answer.** P(Dry,Dry)=0.48; P(next Dry|current Dry)=0.8. These answer different questions. **Yaad rakho:** Path: pehle start probability, phir har arrow ki probability. --- ## M2 · Estimate transition and emission probabilities from tagged data Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-hmm--transition-emission-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-hmm/transition-emission-estimation Formulas explained through full worked examples $$ P(t'\mid t)=\frac{C(t,t')}{C(t)\text{ with a following tag/end event}} $$ $$ P(w\mid t)=\frac{C(w,t)}{C(t)} $$ In a transition denominator, count the previous tag's outgoing events. In an emission denominator, count all tokens bearing that tag. With explicit end transitions the totals align conveniently. Keep the two numerator types separate: tag→tag counts are not word→tag counts. **Question. A tagged corpus contains 116454 determiner tags DT, each counted with an outgoing event. DT is followed by NN 56509 times. Estimate P(NN|DT).** #### Given C(DT)=116454; C(DT,NN)=56509. #### To find The DT→NN transition probability. #### Formula and why it applies $$ P(NN\mid DT)=C(DT,NN)/C(DT) $$ The denominator is the number of previous DT contexts because DT is the condition. #### Solution ##### Step 1. Identify the transition count DT followed by NN occurs 56509 times. ##### Step 2. Identify the conditional total There are 116454 DT contexts. ##### Step 3. Divide $$ P(NN\mid DT)=\frac{56509}{116454}\approx0.485247 $$ **Final answer.** P(NN|DT)≈0.485247. **Question. The tag VBZ occurs 21627 times, and 10073 of those tokens are the word “is”. Estimate P(is|VBZ).** #### Given C(VBZ)=21627; C(is,VBZ)=10073. #### To find The emission probability of is from VBZ. #### Formula and why it applies $$ P(is\mid VBZ)=C(is,VBZ)/C(VBZ) $$ The condition is the tag VBZ; we ask which word it emits. #### Solution ##### Step 1. Choose the word-tag count The required word is is with tag VBZ: count 10073. ##### Step 2. Choose the tag total All VBZ-tagged word tokens: count 21627. ##### Step 3. Divide $$ P(is\mid VBZ)=\frac{10073}{21627}\approx0.465760 $$ **Final answer.** P(is|VBZ)≈0.465760. This is different from P(VBZ|is). **Question. Use the tagged sentences “dogs/N chase/V cats/N” and “cats/N sleep/V”. Estimate the initial tag probabilities, all nonzero transitions including END, and emissions.** #### Given Corpus: START dogs/N chase/V cats/N END; START cats/N sleep/V END. #### To find Initial distribution, transition rows and emission rows. #### Formula and why it applies $$ \pi_t=C(START,t)/2,\quad a_{tu}=C(t,u)/C(t),\quad b_t(w)=C(w,t)/C(t) $$ The corpus explicitly tells us the tags, so we can count training frequencies directly. #### Solution ##### Step 1. Count tag tokens N occurs 3 times: dogs, cats, cats. V occurs 2 times: chase, sleep. ##### Step 2. Count sentence starts Both sentences start with N. $$ \pi_N=2/2=1,\quad \pi_V=0/2=0 $$ ##### Step 3. Count transitions | Pair | Count | | --- | --- | | N→V | 2 | | N→END | 1 | | V→N | 1 | | V→END | 1 | ##### Step 4. Normalize transition rows | Current | To N | To V | To END | | --- | --- | --- | --- | | N | 0 | 2/3 | 1/3 | | V | 1/2 | 0 | 1/2 | ##### Step 5. Count and normalize emissions | Tag | dogs | cats | chase | sleep | | --- | --- | --- | --- | --- | | N | 1/3 | 2/3 | 0 | 0 | | V | 0 | 0 | 1/2 | 1/2 | **Final answer.** π=(1,0); transitions and emissions are given in the completed tables. Each complete probability row sums to 1. **Yaad rakho:** Tag→tag table aur tag→word table ko mix mat karna. --- ## M2 · HMM path probabilities: three complete numerical examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-hmm--hot-cold-paths/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-hmm/hot-cold-paths Formulas explained through full worked examples The following model uses hidden weather states Hot (H) and Cold (C). Observations are numbers 1, 2 and 3. These numbers are observation labels, not quantities to multiply directly. Their emission probabilities come from the table. | Initial state | Probability | | --- | --- | | H | 0.8 | | C | 0.2 | | From → to | H | C | | --- | --- | --- | | H | 0.7 | 0.3 | | C | 0.4 | 0.6 | | Emission state | Observation 1 | Observation 2 | Observation 3 | | --- | --- | --- | --- | | H | 0.2 | 0.4 | 0.4 | | C | 0.5 | 0.4 | 0.1 | **Question. An HMM has π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for observations (1,2,3) are H:(0.2,0.4,0.4) and C:(0.5,0.4,0.1). Find the joint probability of hidden path H,C,H and observations 1,3,1.** #### Given Hidden path H→C→H. Observations 1→3→1. Model probabilities are specified in the question. There is no END factor. #### To find P(H,C,H; 1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HC}b_C(3)a_{CH}b_H(1) $$ There are three hidden states and observations: three emissions, two transitions, and one initial probability. #### Solution ##### Step 1. Start Hot and emit 1 $$ 0.8\times0.2=0.16 $$ ##### Step 2. Transition Hot to Cold $$ 0.16\times0.3=0.048 $$ ##### Step 3. Cold emits 3 $$ 0.048\times0.1=0.0048 $$ ##### Step 4. Transition Cold to Hot $$ 0.0048\times0.4=0.00192 $$ ##### Step 5. Hot emits the last 1 $$ 0.00192\times0.2=0.000384 $$ **Final answer.** Joint probability = 0.000384 = 3.84×10⁻⁴. **Question. For π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels (1,2,3), find the joint probability of path H,H,H and observations 1,3,1.** #### Given State path H→H→H, observations 1→3→1; no END factor. #### To find P(H,H,H;1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HH}b_H(3)a_{HH}b_H(1) $$ The state remains Hot; each transition still contributes its probability 0.7. #### Solution ##### Step 1. Initial state and first emission $$ 0.8(0.2)=0.16 $$ ##### Step 2. First self-transition $$ 0.16(0.7)=0.112 $$ ##### Step 3. Emit observation 3 $$ 0.112(0.4)=0.0448 $$ ##### Step 4. Second self-transition $$ 0.0448(0.7)=0.03136 $$ ##### Step 5. Emit final observation 1 $$ 0.03136(0.2)=0.006272 $$ **Final answer.** Joint probability = 0.006272. A self-transition is not automatically probability 1. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6) in next-state order H,C; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3, find the score of path C,C,H with observations 1,3,1.** #### Given Hidden path C→C→H. Observation sequence 1→3→1. No end-state factor. #### To find The joint path-and-observation probability. #### Formula and why it applies $$ P=\pi_C b_C(1)a_{CC}b_C(3)a_{CH}b_H(1) $$ The initial state is Cold, so use π(C), not π(H). Each emission is selected from the state at the same position. #### Solution ##### Step 1. Start Cold and emit 1 $$ 0.2(0.5)=0.1 $$ ##### Step 2. Stay Cold $$ 0.1(0.6)=0.06 $$ ##### Step 3. Emit 3 from Cold $$ 0.06(0.1)=0.006 $$ ##### Step 4. Move Cold to Hot $$ 0.006(0.4)=0.0024 $$ ##### Step 5. Emit 1 from Hot $$ 0.0024(0.2)=0.00048 $$ **Final answer.** Joint probability = 0.00048. **Yaad rakho:** Har column par: state ka emission. Columns ke beech: transition. --- ## M2 · POS HMM worked corpus: Mary, Jane, can, will and spot Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-hmm--tagging-corpus-case/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-hmm/tagging-corpus-case Formulas explained through full worked examples The class example contains words that can be either names or ordinary verbs/modal auxiliaries. The term is **modal** (M), not “model”. To avoid guessing tags from spelling, the training corpus below supplies them explicitly and lowercases words for the emission vocabulary. | Training sentence | Gold tags | | --- | --- | | mary jane can see will | N N M V N | | spot will see mary | N M V N | | will jane spot mary | M N V N | | mary will pat spot | N M V N | There are 9 noun tokens, 4 modal tokens and 4 verb tokens. Each sentence includes a start and end transition. For nouns, the next event can be another noun, modal, verb or END. | Current → next | N | M | V | END | | --- | --- | --- | --- | --- | | START | 3/4 | 1/4 | 0 | 0 | | N | 1/9 | 3/9 | 1/9 | 4/9 | | M | 1/4 | 0 | 3/4 | 0 | | V | 4/4 | 0 | 0 | 0 | | Tag → word | mary | jane | can | see | will | spot | pat | | --- | --- | --- | --- | --- | --- | --- | --- | | N | 4/9 | 2/9 | 0 | 0 | 1/9 | 2/9 | 0 | | M | 0 | 0 | 1/4 | 0 | 3/4 | 0 | 0 | | V | 0 | 0 | 0 | 2/4 | 0 | 1/4 | 1/4 | **Question. Using the fully tagged four-sentence corpus above, calculate the joint score of “will jane spot mary” assigned tags M,N,V,N, including END.** #### Given Required factors from the tables: START→M=1/4; emit will|M=3/4; M→N=1/4; jane|N=2/9; N→V=1/9; spot|V=1/4; V→N=1; mary|N=4/9; N→END=4/9. #### To find The specified tag-path score, including words and sentence end. #### Formula and why it applies $$ P=\pi_M b_M(will)a_{MN}b_N(jane)a_{NV}b_V(spot)a_{VN}b_N(mary)a_{N,END} $$ Both transitions and emissions are needed. A tag transition alone cannot tell us whether a particular word is plausible. #### Solution ##### Step 1. Start and emit will $$ \frac14\times\frac34=\frac3{16} $$ ##### Step 2. Move to N and emit jane $$ \frac3{16}\times\frac14\times\frac29=\frac1{96} $$ ##### Step 3. Move to V and emit spot $$ \frac1{96}\times\frac19\times\frac14=\frac1{3456} $$ ##### Step 4. Move to N and emit mary $$ \frac1{3456}\times1\times\frac49=\frac1{7776} $$ ##### Step 5. End the sentence $$ \frac1{7776}\times\frac49=\frac1{17496}\approx0.0000571559 $$ **Final answer.** The joint score is 1/17496 ≈ 0.0000571559. **Question. For the same tagged corpus and tables, explain the unsmoothed score of “will can spot mary” assigned tags M,V,N,N.** #### Given The proposed second word is can with hidden tag V. In the training corpus, can occurs only with M, so b\_V(can)=0. #### To find Whether the complete path score is zero and what this means. #### Formula and why it applies $$ P(\text{path,words})=(\text{other required factors})\times b_V(can) $$ A product containing one zero factor is zero even if every other factor is positive. #### Solution ##### Step 1. Find the critical emission $$ b_V(can)=C(can,V)/C(V)=0/4=0 $$ ##### Step 2. Use the full-product consequence $$ P(\text{path,words})=0 $$ ##### Step 3. Interpret correctly This unsmoothed corpus model has no evidence for can as a verb. It does not prove that a verb usage of can is impossible in English. **Final answer.** The unsmoothed path score is zero. --- ## M2 · Why exhaustive search grows exponentially Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-hmm--sequence-count/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-hmm/sequence-count Formulas explained through full worked examples If each of T observed words can receive any of N tags, there are N choices at each position. Thus the total number of complete tag sequences is N multiplied by itself T times. This counts possible assignments before grammar rules or zero probabilities eliminate any. $$ \text{Number of tag sequences}=N^T $$ **Question. A four-word sentence is tagged using three possible POS tags per word. How many complete tag sequences are possible?** #### Given N=3 possible tags per position; T=4 word positions. #### To find The number of tag sequences. #### Formula and why it applies $$ M=N^T $$ Every independent position contributes a factor of three choices. #### Solution ##### Step 1. Write one factor per position $$ M=3\times3\times3\times3 $$ ##### Step 2. Multiply $$ 3\times3=9,\quad9\times3=27,\quad27\times3=81 $$ **Final answer.** 81 possible tag sequences. **Question. A five-word sentence has two candidate tags per word. Count its tag sequences.** #### Given N=2; T=5. #### To find M. #### Formula and why it applies $$ M=N^T $$ The length is the exponent, not a multiplier. #### Solution ##### Step 1. Substitute $$ M=2^5 $$ ##### Step 2. Evaluate $$ 2\times2\times2\times2\times2=32 $$ **Final answer.** 32 possible sequences. **Question. Using 45 possible tags, compare the number of tag sequences for sentences of length 2 and 3.** #### Given N=45. First T=2, then T=3. #### To find Both counts and the growth factor. #### Formula and why it applies $$ M_2=45^2,\quad M_3=45^3 $$ Adding one position multiplies the possibilities by the number of tag choices. #### Solution ##### Step 1. Two words $$ 45^2=2025 $$ ##### Step 2. Three words $$ 45^3=2025\times45=91125 $$ ##### Step 3. Compare $$ 91125/2025=45 $$ **Final answer.** 2 words: 2025 paths. 3 words: 91125 paths. One additional word multiplies the count by 45. The slides introduce Viterbi to avoid enumerating every path. It reuses the best prefix score for each ending state. For a dense first-order HMM, it needs O(TN²) recurrence work instead of enumerating NT complete paths. **Yaad rakho:** Choices per word ki power = sentence length. --- ## M2 · Viterbi decoding: compare incoming paths, keep the best, trace back Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-hmm--viterbi-explained/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-hmm/viterbi-explained Formulas explained through full worked examples **The Viterbi algorithm** finds the most likely hidden-state sequence for a fixed observation sequence. At each position, it keeps the best path ending in each state. This is safe because, under the HMM assumptions, future extensions depend on that ending state, not the discarded details of the prefix. $$ \delta_1(j)=\pi_jb_j(o_1) $$ $$ \delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ $$ \psi_t(j)=\arg\max_i[\delta_{t-1}(i)a_{ij}] $$ | Symbol | Meaning in ordinary words | | --- | --- | | δₜ(j) | Score of the best prefix that ends in state j at position t. | | ψₜ(j) | Previous state that supplied that best prefix; keep it as a backpointer. | | max | Choose the larger candidate score; do not add. | | argmax | Remember which previous state won, rather than only its value. | Initialize: start × first emission → For each next state: compare incoming scores → Multiply winner by this state's emission → Keep score + winning predecessor → Choose best final state and trace arrows backward Do not choose the best state independently at each word. The best last state can come from a different best predecessor than a local guess. Backpointers recover one consistent complete path. If scores tie, either tied path is optimal under the model; state the tie. **Question. Use an HMM with π(H)=0.8, π(C)=0.2, transitions H:(0.7,0.3), C:(0.4,0.6) in H,C order, and emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3. Find the most likely hidden path for observations 1,3,1 using Viterbi.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find The best hidden-state sequence and its joint score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(o_1),\quad\delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ Viterbi keeps two best prefix scores per column, one ending H and one ending C. #### Solution ##### Step 1. Initialize at observation 1 $$ \delta_1(H)=0.8(0.2)=0.16 $$ $$ \delta_1(C)=0.2(0.5)=0.10 $$ ##### Step 2. At observation 3, calculate the H cell $$ H\to H:\ 0.16(0.7)=0.112 $$ $$ C\to H:\ 0.10(0.4)=0.040 $$ $$ \delta_2(H)=\max(0.112,0.040)(0.4)=0.0448 $$ Backpointer for this H cell: H. ##### Step 3. At observation 3, calculate the C cell $$ H\to C:\ 0.16(0.3)=0.048 $$ $$ C\to C:\ 0.10(0.6)=0.060 $$ $$ \delta_2(C)=\max(0.048,0.060)(0.1)=0.006 $$ Backpointer for this C cell: C. ##### Step 4. At final observation 1, calculate H $$ H\to H:\ 0.0448(0.7)=0.03136 $$ $$ C\to H:\ 0.006(0.4)=0.0024 $$ $$ \delta_3(H)=0.03136(0.2)=0.006272 $$ Backpointer: H. ##### Step 5. At final observation 1, calculate C $$ H\to C:\ 0.0448(0.3)=0.01344 $$ $$ C\to C:\ 0.006(0.6)=0.0036 $$ $$ \delta_3(C)=0.01344(0.5)=0.00672 $$ Backpointer: H. ##### Step 6. Select the final state 0.00672 > 0.006272, so the best final state is C. ##### Step 7. Trace back Final C came from H at position2. That H came from H at position1. Reading forward gives H,H,C. ##### Step 8. Verify the recovered complete path $$ 0.8(0.2)(0.7)(0.4)(0.3)(0.5)=0.00672 $$ **Final answer.** Best path H→H→C with joint score 0.00672. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 3,1.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best two-state hidden path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(3),\quad\delta_2(j)=b_j(1)\max_i[\delta_1(i)a_{ij}] $$ The same recurrence applies even though the observation order changed. #### Solution ##### Step 1. Initialize observation 3 $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.1)=0.02 $$ ##### Step 2. Calculate the final H $$ \max(0.32(0.7),0.02(0.4))(0.2) $$ $$ =\max(0.224,0.008)(0.2)=0.0448 $$ Winning predecessor H. ##### Step 3. Calculate the final C $$ \max(0.32(0.3),0.02(0.6))(0.5) $$ $$ =\max(0.096,0.012)(0.5)=0.048 $$ Winning predecessor H. ##### Step 4. Select and trace Final C wins with 0.048 and came from H. The path is H,C. **Final answer.** Best path H→C, score 0.048. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 2,2.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(2),\quad\delta_2(j)=b_j(2)\max_i[\delta_1(i)a_{ij}] $$ Both states emit2 with probability 0.4, but their initial and transition probabilities still differ. #### Solution ##### Step 1. Initialize $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.4)=0.08 $$ ##### Step 2. Final H $$ \max(0.32(0.7),0.08(0.4))(0.4) $$ $$ =\max(0.224,0.032)(0.4)=0.0896 $$ Predecessor H. ##### Step 3. Final C $$ \max(0.32(0.3),0.08(0.6))(0.4) $$ $$ =\max(0.096,0.048)(0.4)=0.0384 $$ Predecessor H. ##### Step 4. Select and trace The final H score 0.0896 is larger. Its predecessor is H. **Final answer.** Best path H→H with score 0.0896. **Yaad rakho:** Har destination ke liye incoming paths compare; winner ka arrow yaad. --- ## M2 · A path score is not the probability of the observed sentence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-hmm--joint-vs-posterior/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-hmm/joint-vs-posterior Formulas explained through full worked examples **Question. An HMM has π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3. For observations 1,3, calculate the probability of the observations by summing all four two-state paths.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find P(observations 1,3), not just the best-path score. #### Formula and why it applies $$ P(o_1,o_2)=\sum_{s_1,s_2}\pi_{s_1}b_{s_1}(o_1)a_{s_1s_2}b_{s_2}(o_2) $$ The hidden path is unspecified, so its mutually exclusive alternatives must be summed. #### Solution ##### Step 1. Score H,H $$ 0.8(0.2)(0.7)(0.4)=0.0448 $$ ##### Step 2. Score H,C $$ 0.8(0.2)(0.3)(0.1)=0.0048 $$ ##### Step 3. Score C,H $$ 0.2(0.5)(0.4)(0.4)=0.016 $$ ##### Step 4. Score C,C $$ 0.2(0.5)(0.6)(0.1)=0.006 $$ ##### Step 5. Add all paths $$ P(1,3)=0.0448+0.0048+0.016+0.006=0.0716 $$ **Final answer.** Observation probability =0.0716. The best single-path score 0.0448 is a different quantity. If asked for the posterior of path H,H given observations 1,3, normalize: 0.0448/0.0716 ≈0.625698. Viterbi chooses the best path using joint scores because the observation denominator is common to every candidate. It does not compute that denominator by taking a maximum. --- ## M2 · Lesk algorithm, from words to overlap scores Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-wsd--u5-lesk/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-wsd/u5-lesk Formulas explained through full worked examples **Lesk is a knowledge-based WSD method that chooses senses whose dictionary descriptions have the greatest overlap with contextual evidence.** Original Lesk compares definitions of neighbouring words' senses. A common simplified version compares each target-sense gloss directly with the sentence context. State which version you use. ### Simplified Lesk procedure 1. Remove the target word from the context; tokenize and normalize the remaining content words. 2. Obtain the gloss for each candidate sense and apply the same normalization. 3. Count shared words between the context and each gloss. 4. Select the sense with maximum overlap. If scores tie, report the tie or use a specified tie-break rule. $$ \operatorname{score}(s)=|C\cap G_s|,\qquad s^*=\arg\max_s\operatorname{score}(s) $$ $$ C $$ is the set of context words, $$ G_s $$ the set of words in sense $$ s $$ 's gloss, $$ \cap $$ means common elements, $$ |\cdot| $$ means count, and $$ \arg\max $$ means “the sense giving the largest score”. This version counts unique matching words once. **Question. Use simplified Lesk to disambiguate “bank” in “The river water rose near the bank.” Use C = {river, water, rose, near}; financial gloss = {institution, money, deposit}; river gloss = {land, river, water, edge}.** #### Given The normalized context and two gloss sets are given. Count exact shared words. #### To find The score of each sense and the selected meaning. #### Formula and why it applies Use $$ \operatorname{score}(s)=|C\cap G_s| $$ because the task specifies context–gloss overlap. #### Step 1. Compare the financial sense No word occurs in both {river, water, rose, near} and {institution, money, deposit}. Overlap = {}. #### Step 2. Count the financial score $$ \operatorname{score}(F)=0 $$ . #### Step 3. Compare the river sense Shared words are {river, water}. “Rose” and “near” are not in the gloss. #### Step 4. Count and select $$ \operatorname{score}(R)=2 $$ . Since $$ 2>0 $$ , choose the river-edge sense. **Answer.** Bank means land beside a river. **Question. Use simplified Lesk for “The dog barked loudly.” Treat barked as bark. Context C = {dog, loudly}; sound gloss = {sharp, cry, dog}; tree gloss = {outer, covering, tree}; ship gloss = {sailing, ship}.** #### Given Three glosses and normalized context {dog, loudly}. #### To find Select the sense of bark. #### Formula and why it applies $$ \operatorname{score}(s)=|C\cap G_s| $$ . Lemmatization lets barked match the target lemma bark; it does not create additional overlap words. #### Step 1. Sound sense Intersection = {dog}; score = 1. #### Step 2. Tree sense Intersection = {}; score = 0. #### Step 3. Ship sense Intersection = {}; score = 0. #### Step 4. Compare The sound sense has the unique maximum: 1 > 0. **Answer.** Bark is the dog's cry. **Question. Apply definition–definition Lesk to “pine cone”. Pine senses: P1 = {evergreen, tree, needle, leaf}; P2 = {waste, sorrow, illness}. Cone senses: C1 = {solid, shape, circular}; C2 = {fruit, evergreen, tree}.** #### Given Two candidate senses for each word. Tokens are already normalized; tree and trees would be reduced to the same form if necessary. #### To find The most compatible pair of senses. #### Formula and why it applies Use $$ |G_{P_i}\cap G_{C_j}| $$ for every pair because this is original-style gloss–gloss comparison. #### Step 1. P1 with C1 No shared word → 0. #### Step 2. P1 with C2 Shared {evergreen, tree} → 2. #### Step 3. P2 with each cone sense P2–C1 has no overlap → 0; P2–C2 has no overlap → 0. #### Step 4. Choose the best pair The four scores are 0, 2, 0, 0. Maximum 2 belongs to P1–C2. **Answer.** Pine is the evergreen tree; cone is its fruit-like reproductive structure. **Limitations:** exact overlap misses synonyms, short glosses may produce zeros, and different preprocessing choices change the score. Extended Lesk may include related synset definitions or longer phrase overlaps; do not silently substitute it for a simple question. Lesk: context aur dictionary mein common words gino; sabse bada overlap jeeta. --- ## M2 · Thesaurus / Walker approach, with three complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-wsd--u5-walker/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-wsd/u5-walker Formulas explained through full worked examples **The Walker approach uses thesaurus categories as semantic evidence.** Associate each candidate sense with a category. Count or weight context words that support that category. Choose the largest total. $$ \operatorname{score}(s)=\sum_{w\in C} I(w,\operatorname{category}(s)) $$ $$ I=1 $$ if the supplied category mapping says the context word supports the sense, otherwise $$ I=0 $$ . The summation symbol means add all those contributions. Weighted questions may replace 1 with a supplied weight. Category membership must be supplied or justified; it is not guessed from word frequency. **Question. Disambiguate bank using Walker scoring. Context words are money, interest, annum, fetch. The finance category supports money, interest and annum; the location category supports none of these words.** #### Given Two categories: Finance and Location. Each supported word contributes 1. #### To find The two totals and selected bank sense. #### Formula and why it applies Sum the category indicators for each sense. #### Step 1. Make a contribution table | Word | Finance | Location | | --- | --- | --- | | money | 1 | 0 | | interest | 1 | 0 | | annum | 1 | 0 | | fetch | 0 | 0 | #### Step 2. Add Finance $$ 1+1+1+0=3 $$ . #### Step 3. Add Location $$ 0+0+0+0=0 $$ . #### Step 4. Select 3 > 0, so select the financial institution. **Answer.** Finance, with score 3 versus 0. **Question. Use Walker scoring for “The fisherman caught bass beside the lake.” Context {fisherman,caught,lake}. The Fish category contains all three clues; the Music category contains none.** #### Given Fish and Music senses, binary category membership given. #### To find The correct bass sense. #### Formula and why it applies Score = sum of context-category matches. #### Step 1. Record Fish matches fisherman → 1; caught → 1; lake → 1. #### Step 2. Add Fish score $$ 1+1+1=3 $$ . #### Step 3. Add Music score $$ 0+0+0=0 $$ . #### Step 4. Choose Fish has the larger score. **Answer.** Bass refers to a fish. **Question. Score “The player used a bat in the match.” Context {player,match}. Sport supports both words with weights 2 and 1; Animal supports neither. Use the supplied weighted Walker scheme.** #### Given Sport weights {player:2, match:1}; Animal weights {player:0, match:0}. #### To find Weighted sense scores. #### Formula and why it applies Use the sum of supplied weights, not an unweighted count, because the question explicitly gives weights. #### Step 1. Sport contribution of player Add 2. #### Step 2. Sport contribution of match Add 1. Sport total = 2 + 1 = 3. #### Step 3. Animal total 0 + 0 = 0. #### Step 4. Compare 3 > 0 selects the sporting implement. **Answer.** Bat is the sports implement. Walker: dictionary ke exact words nahi, context kis topic/category ko support karta hai woh count. --- ## M2 · Multinomial Naive Bayes: the exact solving method Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-wsd--u5-bayes-steps/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-wsd/u5-bayes-steps Formulas explained through full worked examples Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. ### Always keep these counts separate Class priors use **number of examples**. Word likelihoods in this multinomial model use **number of word occurrences**. The vocabulary counts **unique words across all classes**. 1. List the class examples and calculate priors. 2. Write one shared vocabulary and count its size. 3. Count total word occurrences in each class. 4. Count each query word in each class. 5. Apply the given smoothing rule. 6. Multiply each prior by all its query likelihoods. 7. Compare scores; divide each score by their sum only if a normalized posterior is requested. $$ P(s\mid C)=\frac{\operatorname{score}(s)}{\sum_{s'}\operatorname{score}(s')}. $$ **Why smoothing is needed:** an unseen clue gives $$ 0/N_s=0 $$ ; one zero makes the whole product zero. Add-one smoothing redistributes probability across every vocabulary word and prevents these zero factors. Use it only when requested, taught for the task, or state the assumption. **Different model warning:** a Bernoulli Naive Bayes question counts presence/absence per document and may include absence factors. Do not use its denominator in a multinomial word-count question. Pehle sentences se prior; phir words se likelihood; dono counts mix mat kar. --- ## M2 · Mam's bank numerical—every count and every multiplication Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-wsd--u5-bayes-mam/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-wsd/u5-bayes-mam Formulas explained through full worked examples **Question. Classify the sense of “bank” using the training table below. The sentence is: “As the stream flowed gently past the shore, we set up camp by the bank of the river, discussing the possibility of borrowing a loan to buy a boat for future trips.” Use only the highlighted feature words stream, shore and loan, and add-one multinomial smoothing.** #### Given | Training row | Context words | Sense | | --- | --- | --- | | 1 | money, deposit, loan | Finance | | 2 | river, water, shore | Riverbank | | 3 | account, savings, investment | Finance | | 4 | boat, river, stream | Riverbank | | 5 | branch, deposit, check | Finance | | 6 | fishing, shore, river | Riverbank | Query feature set: {stream, shore, loan}. Other sentence words are not included as features in this supplied example. #### To find The most probable bank sense and normalized posteriors. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Calculate priors Finance appears in 3 of the 6 rows; Riverbank appears in 3 of 6. $$ P(F)=\frac36=\frac12,\qquad P(R)=\frac36=\frac12. $$ #### Step 2. Construct the vocabulary {money, deposit, loan, river, water, shore, account, savings, investment, boat, stream, fishing, branch, check}. Thus V=14. Repeated deposit or river is counted once in vocabulary. #### Step 3. Count word occurrences per class There are 3 finance rows × 3 words = 9 word occurrences. There are 3 river rows × 3 words = 9. Thus $$ N_F=N_R=9 $$ , and each smoothed denominator is $$ 9+14=23 $$ . #### Step 4. Count the three query words | Word | Finance count | Riverbank count | | --- | --- | --- | | stream | 0 | 1 | | shore | 0 | 2 | | loan | 1 | 0 | #### Step 5. Calculate finance likelihoods $$ P(\text{stream}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{shore}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{loan}\mid F)=\frac{1+1}{23}=\frac2{23} $$ #### Step 6. Multiply the finance score $$ \operatorname{score}(F)=\frac12\times\frac1{23}\times\frac1{23}\times\frac2{23}=\frac1{23^3}=\frac1{12167}\approx0.00008219. $$ #### Step 7. Calculate river likelihoods $$ P(\text{stream}\mid R)=\frac{1+1}{23}=\frac2{23} $$ $$ P(\text{shore}\mid R)=\frac{2+1}{23}=\frac3{23} $$ $$ P(\text{loan}\mid R)=\frac{0+1}{23}=\frac1{23} $$ #### Step 8. Multiply the river score $$ \operatorname{score}(R)=\frac12\times\frac2{23}\times\frac3{23}\times\frac1{23}=\frac3{12167}\approx0.00024657. $$ #### Step 9. Compare scores 3/12167 is larger than 1/12167. Riverbank wins even though “loan” is financial evidence: the combined clues matter. #### Step 10. Normalize if a posterior is requested $$ P(F\mid C)=\frac{1/12167}{4/12167}=\frac14=0.25 $$ $$ P(R\mid C)=\frac{3/12167}{4/12167}=\frac34=0.75. $$ **Answer.** Bank is the riverbank sense. Scores are 1/12167 and 3/12167; normalized model posteriors are 0.25 and 0.75. **Without smoothing:** stream and shore are absent in Finance; loan is absent in Riverbank. Both unsmoothed scores become zero, so there is no valid winning class from those products. Also, unnormalized scores should not be labelled as final posterior probabilities. Zero word? Add-one: count+1 upar, total words+vocabulary neeche. Last mein score bada wala sense. --- ## M2 · More WSD numericals: given likelihoods, unequal priors and repeated clues Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/formulas/f-wsd--u5-bayes-practice/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/formulas/f-wsd/u5-bayes-practice Formulas explained through full worked examples **Question. For bass, let P(Fish)=0.6 and P(Music)=0.4. Given Fish likelihoods P(caught|Fish)=0.5, P(lake|Fish)=0.4 and Music likelihoods P(caught|Music)=0.05, P(lake|Music)=0.1, classify context {caught,lake} and normalize.** #### Given Priors and likelihoods are provided directly; no smoothing or training-count estimation is requested. #### To find The winning sense and its posterior. #### Formula and why it applies $$ \operatorname{score}(s)=P(s)P(\text{caught}\mid s)P(\text{lake}\mid s) $$ , because the two features are treated as conditionally independent. #### Step 1. Fish multiplication $$ 0.6\times0.5=0.3 $$ , then $$ 0.3\times0.4=0.12 $$ . #### Step 2. Music multiplication $$ 0.4\times0.05=0.02 $$ , then $$ 0.02\times0.1=0.002 $$ . #### Step 3. Total evidence $$ 0.12+0.002=0.122 $$ . #### Step 4. Normalize $$ P(Fish\mid C)=0.12/0.122\approx0.983607 $$ $$ P(Music\mid C)=0.002/0.122\approx0.016393. $$ **Answer.** Fish, with model posterior approximately 0.983607. **Question. Disambiguate bank from the repeated context tokens {loan, loan, river}. Priors are Finance=2/3 and River=1/3. Vocabulary V=4. Finance has N=8 total tokens, loan count 3 and river count 0; River has N=4, loan count 0 and river count 2. Apply add-one smoothing.** #### Given Multinomial counts are supplied. The two appearances of loan must both contribute. #### To find Scores and normalized Finance posterior. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Finance denominator $$ 8+4=12 $$ . $$ P(loan\mid F)=(3+1)/12=1/3 $$ ; $$ P(river\mid F)=(0+1)/12=1/12 $$ . #### Step 2. River denominator $$ 4+4=8 $$ . $$ P(loan\mid R)=(0+1)/8=1/8 $$ ; $$ P(river\mid R)=(2+1)/8=3/8 $$ . #### Step 3. Finance product $$ \operatorname{score}(F)=\frac23\times\frac13\times\frac13\times\frac1{12}=\frac1{162}\approx0.00617284. $$ #### Step 4. River product $$ \operatorname{score}(R)=\frac13\times\frac18\times\frac18\times\frac38=\frac1{512}\approx0.001953125. $$ #### Step 5. Compare and normalize $$ P(F\mid C)=\frac{1/162}{1/162+1/512}=\frac{512}{674}=\frac{256}{337}\approx0.759644. $$ **Answer.** Finance wins. Repeated loan contributes a squared likelihood; do not reduce the query to a unique-word set. ### Answer-writing order in an exam Write the class priors, vocabulary size, class token totals, a likelihood table, both score products and the final class. Keep fractions until the comparison where possible. If the question only asks classification, normalization is optional; if it asks posterior probabilities, normalization is required. Repeated word aaye toh factor bhi repeat. Sirf word-set bana diya toh multinomial answer badal jayega. --- ## M2 · Q1B · Chunking and chinking Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/final-2025--final-2025-q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/final-2025/final-2025-q1b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Explain the concepts of chunking and chinking in Natural Language Processing (NLP). Discuss how Part-of-Speech (POS) tagging supports these processes, and illustrate your explanation with a suitable example. [5 marks] ### Answer **Chunking** groups adjacent tagged words into shallow phrases, such as noun phrases, without building a complete sentence parse. **Chinking** removes a specified sequence of tags or words from a chunk, splitting a larger candidate chunk when required. ### POS tags provide the pattern A POS tag describes a grammatical category: DT = determiner, JJ = adjective, NN = singular noun, VBD = past-tense verb and IN = preposition. A noun-phrase rule can be written as `NP: {
?*+}`: an optional determiner, zero or more adjectives, followed by one or more nouns. ### Worked example | Word | The | small | cat | sat | on | the | mat | | --- | --- | --- | --- | --- | --- | --- | --- | | Tag | DT | JJ | NN | VBD | IN | DT | NN | Applying the NP rule gives **[The small cat] sat on [the mat]**. The first chunk contains DT–JJ–NN; the second contains DT–NN. “sat on” is not included because it does not match the noun-phrase rule. For chinking, begin with one broad chunk: **[The small cat sat on the mat]**. Remove the VBD and IN sequence “sat on”, for example with `}+{`. The result is **[The small cat] sat on [the mat]**. POS tagging therefore supplies grammatical labels; chunking uses labels to select phrases, while chinking uses labels to exclude unwanted material. Incorrect tags can lead to incorrect chunks. **Yaad rakho:** Chunking = phrase ko jodo; chinking = beech ka unwanted part nikalo. --- ## M2 · Q1C · Hyponymy, hypernymy and WSD Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/final-2025--final-2025-q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/final-2025/final-2025-q1c M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Define hyponymy and hypernymy with suitable linguistic examples. How can hypernymy be applied in Word Sense Disambiguation (WSD) to resolve the correct meaning of polysemous words? Provide an example? [5 marks] ### Answer **Hyponymy** is the “is-a-kind-of” relation from a specific concept to a broader concept. **Hypernymy** is the inverse relation from the broader concept to the specific one. “Rose” is a hyponym of “flower”; “flower” is a hypernym of “rose”. rose / tulip → flower → plant The arrow points towards a broader category. Hyponymy is not a part–whole relation: a wheel is part of a car, but it is not a kind of car. ### Using the hierarchy for WSD WSD selects the intended sense of a word from its context. Obtain candidate senses from a lexical resource such as WordNet, inspect their hypernym chains, and favour the chain most compatible with nearby words. 1. Sentence: “The mouse clicked the icon.” 2. Candidate 1: mouse → rodent → mammal → animal. Candidate 2: mouse → pointing device → computer peripheral. 3. The context words “clicked” and “icon” belong to the computer-use domain. They support the device hierarchy, rather than the animal hierarchy. 4. Select the computer-pointing-device sense. In “The mouse ate cheese,” the animal hierarchy is more compatible. Hypernym evidence supplies semantic generalisation; it is a useful signal rather than a guarantee, because short or unusual contexts may still be ambiguous. **Yaad rakho:** Hyponym specific hai; hypernym uska bada category hai. --- ## M2 · Q2B · Laplace-smoothed bigram probability Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/final-2025--final-2025-q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/final-2025/final-2025-q2b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Consider the following training corpus: 1. artists paint murals in city 2. musicians play songs in city 3. painters paint murals on walls 4. signs are near city Using a bigram language model with Laplace smoothing, evaluate the probability of the test sentence: “artists paint murals near city”. Assume the and for each sentence. [10 marks] ### Answer ### Given - artists paint murals in city - musicians play songs in city - painters paint murals on walls - signs are near city Test: artists paint murals near city. Add one start token and one end token to every training and test sentence. ### To find The Laplace-smoothed bigram probability of the complete test sentence, including its end token. ### Formula: which one and why $$ P_{+1}(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)+1}{C(w_{i-1})+V} $$ C(previous,current) counts that adjacent pair in the training corpus. C(previous) counts its occurrences as a history. V is the vocabulary size. Add-one smoothing is necessary because an unseen pair must not force the whole sentence probability to zero. $$ P(w_1,\ldots,w_n,\langle/s\rangle\mid\langle s\rangle)=\prod_{i=1}^{n+1}P(w_i\mid w_{i-1}) $$ ### Solution ### Step 1. Fix the token and vocabulary convention Lowercase words and remove the final punctuation. The supplied course key counts both boundary tokens in V. Following that convention: are, artists, city, in, murals, musicians, near, on, paint, painters, play, signs, songs, walls, , gives **V = 16**. The start marker supplies history; it is not an extra probability factor before the first word. ### Step 2. List every required transition The complete test chain is → artists → paint → murals → near → city → . ### Step 3. Count first, then add one | Required bigram | History count | Pair count | Smoothed probability | | --- | --- | --- | --- | | → artists | 4 | 1 | (1+1)/(4+16) = 1/10 | | artists → paint | 1 | 1 | (1+1)/(1+16) = 2/17 | | paint → murals | 2 | 2 | (2+1)/(2+16) = 1/6 | | murals → near | 2 | 0 | (0+1)/(2+16) = 1/18 | | near → city | 1 | 1 | (1+1)/(1+16) = 2/17 | | city → | 3 | 3 | (3+1)/(3+16) = 4/19 | Count only adjacent words, never words that merely appear in the same sentence. The unseen pairs have count zero, so their numerator becomes one. ### Step 4. Multiply all six probabilities $$ P=\frac{2}{20}\times\frac{2}{17}\times\frac{3}{18}\times\frac{1}{18}\times\frac{2}{17}\times\frac{4}{19} $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. → artists | $$ 1\times\frac{1}{10}=\frac{1}{10} $$ | | 2. artists → paint | $$ \frac{1}{10}\times\frac{2}{17}=\frac{1}{85} $$ | | 3. paint → murals | $$ \frac{1}{85}\times\frac{1}{6}=\frac{1}{510} $$ | | 4. murals → near | $$ \frac{1}{510}\times\frac{1}{18}=\frac{1}{9180} $$ | | 5. near → city | $$ \frac{1}{9180}\times\frac{2}{17}=\frac{1}{78030} $$ | | 6. city → | $$ \frac{1}{78030}\times\frac{4}{19}=\frac{2}{741285} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{2}{741285}\approx 2.69801763\times10^{-6} $$ ### Step 5. Interpret the result This is a joint sentence probability, so a small value is normal. Do not add transition probabilities; each word must occur in sequence, so the chain multiplies them. ### Final answer **P ≈ 2.69801763 × 10-6** with the stated course vocabulary convention. **Yaad rakho:** Pair count mein 1 jodo, previous-word count mein V; phir saare fractions multiply karo. **Convention note:** Some implementations exclude from the predicted vocabulary. Their V is one smaller and their numerical probability differs. In an exam, state V explicitly and follow the convention specified in the question or class solution. --- ## M2 · Q4B · HMM emission, transition and sequence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/final-2025--final-2025-q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/final-2025/final-2025-q4b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Consider the following POS tagged corpus. Apply Hidden Markov Model (HMM) approach to: 1. Cat/Noun chase/Verb mice/Noun 2. The/Det dog/Noun runs/Verb 3. A/Det cat/Noun eats/Verb cheese/Noun i. Construct the emission probability matrix. ii. Construct the transmission probability matrix. iii. Calculate the probability of sequence of events: The/Det cat/Noun eats/Verb mice/Noun. Note: Include start and end characters for all the sentences. [10 marks] ### Answer ### Given | Sentence | Tagged sequence | | --- | --- | | 1 | cat/Noun chase/Verb mice/Noun | | 2 | the/Det dog/Noun runs/Verb | | 3 | a/Det cat/Noun eats/Verb cheese/Noun | Test words: the cat eats mice. Given tag sequence: Det → Noun → Verb → Noun. Include Start and End for all sentences. Lowercase the corpus so sentence-initial Cat and cat are one word. ### To find Emission probabilities, transition probabilities and the joint probability of the specified tag-and-word sequence. ### Formula: which one and why $$ P(w\mid t)=\frac{C(t,w)}{C(t)} $$ An emission probability asks: among tokens carrying this tag t, what fraction are the word w? $$ P(t_j\mid t_i)=\frac{C(t_i,t_j)}{\sum_k C(t_i,t_k)} $$ A transition probability asks: among transitions leaving tag tᵢ, what fraction go to tⱼ? Include End in the outgoing counts. The paper says “transmission”; the standard HMM term is transition. $$ P(\mathbf w,\mathbf t)=P(t_1\mid\mathrm{Start})\prod_{i=1}^{n}P(w_i\mid t_i)\prod_{i=2}^{n}P(t_i\mid t_{i-1})P(\mathrm{End}\mid t_n) $$ ### Solution ### Step 1. Count tokens for each tag Det has 2 tokens; Noun has 5 tokens; Verb has 3 tokens. The repeated noun cat occurs twice with Noun; every other displayed noun occurs once. The three verb types each occur once. ### Step 2. Build the emission matrix | Tag / word | a | cat | chase | cheese | dog | eats | mice | runs | the | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Det | 1/2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1/2 | | Noun | 0 | 2/5 | 0 | 1/5 | 1/5 | 0 | 1/5 | 0 | 0 | | Verb | 0 | 0 | 1/3 | 0 | 0 | 1/3 | 0 | 1/3 | 0 | Each row sums to one. A zero means no occurrence under that tag in this unsmoothed training corpus; no add-one smoothing was requested. ### Step 3. Write the three tag chains Start → Noun → Verb → Noun → End Start → Det → Noun → Verb → End Start → Det → Noun → Verb → Noun → End ### Step 4. Count outgoing transitions | From | To Det | To Noun | To Verb | To End | Outgoing total | | --- | --- | --- | --- | --- | --- | | Start | 2 | 1 | 0 | 0 | 3 | | Det | 0 | 2 | 0 | 0 | 2 | | Noun | 0 | 0 | 3 | 2 | 5 | | Verb | 0 | 2 | 0 | 1 | 3 | ### Step 5. Divide each row by its outgoing total | From / To | Det | Noun | Verb | End | | --- | --- | --- | --- | --- | | Start | 2/3 | 1/3 | 0 | 0 | | Det | 0 | 1 | 0 | 0 | | Noun | 0 | 0 | 3/5 | 2/5 | | Verb | 0 | 2/3 | 0 | 1/3 | End is terminal, so it has no estimated outgoing row. The four nonterminal rows each sum to one. ### Step 6. Choose the factors for the given sequence | Event in order | Probability | | --- | --- | | Start → Det | 2/3 | | Det emits the | 1/2 | | Det → Noun | 1 | | Noun emits cat | 2/5 | | Noun → Verb | 3/5 | | Verb emits eats | 1/3 | | Verb → Noun | 2/3 | | Noun emits mice | 1/5 | | Noun → End | 2/5 | ### Step 7. Multiply, including the end transition $$ P=\frac23\times\frac12\times1\times\frac25\times\frac35\times\frac13\times\frac23\times\frac15\times\frac25 $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. Start → Det | $$ 1\times\frac{2}{3}=\frac{2}{3} $$ | | 2. Det emits the | $$ \frac{2}{3}\times\frac{1}{2}=\frac{1}{3} $$ | | 3. Det → Noun | $$ \frac{1}{3}\times1=\frac{1}{3} $$ | | 4. Noun emits cat | $$ \frac{1}{3}\times\frac{2}{5}=\frac{2}{15} $$ | | 5. Noun → Verb | $$ \frac{2}{15}\times\frac{3}{5}=\frac{2}{25} $$ | | 6. Verb emits eats | $$ \frac{2}{25}\times\frac{1}{3}=\frac{2}{75} $$ | | 7. Verb → Noun | $$ \frac{2}{75}\times\frac{2}{3}=\frac{4}{225} $$ | | 8. Noun emits mice | $$ \frac{4}{225}\times\frac{1}{5}=\frac{4}{1125} $$ | | 9. Noun → End | $$ \frac{4}{1125}\times\frac{2}{5}=\frac{8}{5625} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac8{5625}\approx0.0014222222 $$ ### Final answer **Joint sequence probability = 8/5625 ≈ 0.0014222222.** This is the probability of the specified words and tags together. It is not the probability of the word sequence summed over every possible tag path. **Yaad rakho:** Tag se tag = transition; tag se word = emission. Dono alternate multiply, end ko mat bhoolo. **Boundary reminder:** The initial factor is P(Det | Start) = 2/3, based on three sentence starts, and the final factor is P(End | Noun) = 2/5. The overall fraction of Det tokens, 2/10, is not the start probability. Using it and omitting End, as in the handwritten key’s last line, computes a different quantity. --- ## M2 · Q5B · Naïve Bayes word-sense classification Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/final-2025--final-2025-q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/final-2025/final-2025-q5b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Consider the sentence: “The bat hit the ball hard on the cricket field.” Two possible senses of “bat”: Sense 1: Bat (Sports equipment – cricket/baseball); Sense 2: Bat (Nocturnal flying mammal). | Sentence ID | Context words | Sense | | --- | --- | --- | | 1 | ball, cricket, score, stadium | Sports | | 2 | night, cave, wings, fly | Animal | | 3 | player, run, match, bat | Sports | | 4 | nocturnal, vampire, insect | Animal | | 5 | team, field, bowl, bat | Sports | | 6 | dark, flying, cave, night | Animal | a) Using the Naïve Bayes algorithm, classify the sense of “bat” in the given sentence. Apply Laplace Smoothing wherever applicable. b) Discuss the significance of Laplace Smoothing in Naïve Bayes Classification. [10 marks] ### Answer ### Given There are three Sports examples and three Animal examples. Treat training context lists as bag-of-words token counts. Exclude the ambiguous target “bat” from the test context. Use the fixed training vocabulary; ignore out-of-vocabulary test words hit and hard. The usable test features are ball, cricket and field. ### To find The posterior-score comparison for Sports and Animal, and the role of smoothing. ### Formula: which one and why $$ \mathrm{score}(s)=P(s)\prod_{w\in\mathrm{context}}P(w\mid s) $$ $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V} $$ P(s) is the class prior, C(w,s) counts occurrences of context word w in class s, Nₛ is the total number of context tokens in that class and V is the shared vocabulary size. Conditional independence lets us multiply word probabilities. The common denominator P(context) is unnecessary when only comparing classes. ### Solution ### Step 1. Calculate the priors $$ P(S)=\frac36=0.5,\qquad P(A)=\frac36=0.5 $$ ### Step 2. Count class tokens and vocabulary Sports has 4 + 4 + 4 = 12 tokens. Animal has 4 + 3 + 4 = 11 tokens. The shared training vocabulary has 20 distinct words: ball, cricket, score, stadium, player, run, match, bat, team, field, bowl, night, cave, wings, fly, nocturnal, vampire, insect, dark, flying. “bat”, “night” and “cave” repeat but count only once each towards V. ### Step 3. Look up each usable context word | Feature | Sports count | Animal count | P(word | Sports) | P(word | Animal) | | --- | --- | --- | --- | --- | | ball | 1 | 0 | (1+1)/(12+20) = 2/32 | (0+1)/(11+20) = 1/31 | | cricket | 1 | 0 | 2/32 | 1/31 | | field | 1 | 0 | 2/32 | 1/31 | ### Step 4. Multiply the Sports score $$ \mathrm{score}(S)=0.5\times\frac2{32}\times\frac2{32}\times\frac2{32}=0.0001220703125 $$ ### Step 5. Multiply the Animal score $$ \mathrm{score}(A)=0.5\times\frac1{31}\times\frac1{31}\times\frac1{31}\approx0.00001678359236 $$ ### Step 6. Compare the scores 0.0001220703125 is greater than 0.00001678359236. Therefore select Sports. Optional normalization gives P(Sports | retained features) ≈ 0.87913; normalization is not needed for the decision. ### Final answer The intended sense is **bat as sports equipment**. Ball, cricket and field favour that sense. **Yaad rakho:** Pehle class prior, phir har context-word likelihood multiply; bigger score wali sense choose karo. ### Why Laplace smoothing matters Without smoothing, all three test features have zero count in Animal. One zero factor makes the entire class score zero, even if other evidence could support that class. Add-one smoothing assigns nonzero probability to every word in the defined vocabulary and redistributes probability mass. It does not mean every unseen word outside the vocabulary can be inserted without changing the vocabulary model. ### If following the supplied paper key exactly The key also multiplies probabilities for hit and hard using the same denominators 32 and 31. That convention produces Sports = 1.19209290 × 10⁻⁷ and Animal ≈ 1.74647163 × 10⁻⁸, again selecting Sports. Its treatment of out-of-vocabulary words is not a normalized fixed-vocabulary model. State your feature convention clearly; the complete calculation above uses a consistent fixed training vocabulary. --- ## M2 · Q1B · Content and function words Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/reexam-2026--reexam-2026-q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/reexam-2026/reexam-2026-q1b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. i. Define function words and content words. ii. Which category (function words and content words) belong to open class and closed class? Justify your answer. iii. For the sentence “The small cat is sleeping under the tree”, identify the function words and content words. [5 marks] ### Answer **Content words** carry much of a sentence’s lexical meaning, usually including nouns, main verbs, adjectives and many adverbs. **Function words** primarily express grammatical relationships, including determiners, prepositions, conjunctions, pronouns and auxiliary verbs. | Category | Typical class | Reason | | --- | --- | --- | | Content words | Open class | New nouns, verbs and adjectives enter the language readily, such as new technology names | | Function words | Closed class | The grammatical inventory is relatively stable; new determiners or pronouns are much less frequently added | | Word | Role in this sentence | Category | | --- | --- | --- | | The | Determiner | Function | | small | Adjective | Content | | cat | Noun | Content | | is | Auxiliary supporting progressive tense | Function | | sleeping | Main verb | Content | | under | Preposition | Function | | the | Determiner | Function | | tree | Noun | Content | **Final answer:** content words = small, cat, sleeping, tree; function words = The, is, under, the. “is” is auxiliary here; a word’s role must be judged in context. “Closed” means relatively resistant to new members, not that change is logically impossible. **Yaad rakho:** Content words scene batate hain; function words grammar ka connection banate hain. --- ## M2 · Q2B · Laplace smoothing on a second corpus Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/reexam-2026--reexam-2026-q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/reexam-2026/reexam-2026-q2b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Consider the following training corpus: 1. cats chase mice in the garden 2. dogs bark loudly at night 3. birds sing in the morning 4. children play games in the park Using a bigram language model with Laplace smoothing, evaluate the probability of the test sentence: “cats bark in the park.” Assume and as start and end of the sentence. [10 marks] ### Answer ### Given - cats chase mice in the garden - dogs bark loudly at night - birds sing in the morning - children play games in the park Test: cats bark in the park. Add one start token and one end token to every training and test sentence. ### To find The Laplace-smoothed bigram probability of the complete test sentence, including its end token. ### Formula: which one and why $$ P_{+1}(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)+1}{C(w_{i-1})+V} $$ C(previous,current) counts that adjacent pair in the training corpus. C(previous) counts its occurrences as a history. V is the vocabulary size. Add-one smoothing is necessary because an unseen pair must not force the whole sentence probability to zero. $$ P(w_1,\ldots,w_n,\langle/s\rangle\mid\langle s\rangle)=\prod_{i=1}^{n+1}P(w_i\mid w_{i-1}) $$ ### Solution ### Step 1. Fix the token and vocabulary convention Lowercase words and remove the final punctuation. The supplied course key counts both boundary tokens in V. Following that convention: at, bark, birds, cats, chase, children, dogs, games, garden, in, loudly, mice, morning, night, park, play, sing, the, , gives **V = 20**. The start marker supplies history; it is not an extra probability factor before the first word. ### Step 2. List every required transition The complete test chain is → cats → bark → in → the → park → . ### Step 3. Count first, then add one | Required bigram | History count | Pair count | Smoothed probability | | --- | --- | --- | --- | | → cats | 4 | 1 | (1+1)/(4+20) = 1/12 | | cats → bark | 1 | 0 | (0+1)/(1+20) = 1/21 | | bark → in | 1 | 0 | (0+1)/(1+20) = 1/21 | | in → the | 3 | 3 | (3+1)/(3+20) = 4/23 | | the → park | 3 | 1 | (1+1)/(3+20) = 2/23 | | park → | 1 | 1 | (1+1)/(1+20) = 2/21 | Count only adjacent words, never words that merely appear in the same sentence. The unseen pairs have count zero, so their numerator becomes one. ### Step 4. Multiply all six probabilities $$ P=\frac{2}{24}\times\frac{1}{21}\times\frac{1}{21}\times\frac{4}{23}\times\frac{2}{23}\times\frac{2}{21} $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. → cats | $$ 1\times\frac{1}{12}=\frac{1}{12} $$ | | 2. cats → bark | $$ \frac{1}{12}\times\frac{1}{21}=\frac{1}{252} $$ | | 3. bark → in | $$ \frac{1}{252}\times\frac{1}{21}=\frac{1}{5292} $$ | | 4. in → the | $$ \frac{1}{5292}\times\frac{4}{23}=\frac{1}{30429} $$ | | 5. the → park | $$ \frac{1}{30429}\times\frac{2}{23}=\frac{2}{699867} $$ | | 6. park → | $$ \frac{2}{699867}\times\frac{2}{21}=\frac{4}{14697207} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{4}{14697207}\approx 2.72160554\times10^{-7} $$ ### Step 5. Interpret the result This is a joint sentence probability, so a small value is normal. Do not add transition probabilities; each word must occur in sequence, so the chain multiplies them. ### Final answer **P ≈ 2.72160554 × 10-7** with the stated course vocabulary convention. **Yaad rakho:** Pair count mein 1 jodo, previous-word count mein V; phir saare fractions multiply karo. **Convention note:** Some implementations exclude from the predicted vocabulary. Their V is one smaller and their numerical probability differs. In an exam, state V explicitly and follow the convention specified in the question or class solution. --- ## M2 · Q3A · Top-down parsing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/reexam-2026--reexam-2026-q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/reexam-2026/reexam-2026-q3a M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. In the context of natural language processing, explain the concept of top-down and bottom-up parsing approaches. Apply top-down syntax parsing approach to the sentence “The tall boy saw a girl in the park.” and draw a parse tree using the CFG grammar rules: S → NP VP; NP → Det Adj N | Det N; VP → V NP | VP PP; PP → P NP. Lexicon: Det → {the | a}; Adj → {tall}; N → {boy | girl | park}; V → {saw}; P → {in}. [10 marks] ### Answer | Aspect | Top-down | Bottom-up | | --- | --- | --- | | Start | The start symbol S | The input words | | Operation | Expand a nonterminal using grammar productions | Combine words and constituents using productions in reverse | | Success | Leaves match the complete input | All input is reduced to S | | Possible difficulty | Naive recursive descent can loop on left recursion or need backtracking | A naive shift-reduce parser may face shift/reduce conflicts | ### Given The sentence contains the tokens the / tall / boy / saw / a / girl / in / the / park. Lowercase the initial “The” to match the lexicon. The full stop is punctuation outside the listed grammar. ### To find A derivation beginning at S and a parse tree matching the whole sentence. ### Rule: which one and why A context-free production replaces one nonterminal with its right-hand sequence. A top-down derivation begins at S and expands the desired structure. The supplied grammar attaches the PP through VP → VP PP; it does not provide NP → NP PP. ### Step 1. Expand the sentence S ⇒ NP VP. Choose NP → Det Adj N for “the tall boy”. ### Step 2. Match the subject words Det ⇒ the, Adj ⇒ tall, N ⇒ boy. The remaining material must be generated by VP. ### Step 3. Allow a verb phrase followed by a PP Expand the outer VP using VP → VP PP. Then expand the inner VP using VP → V NP. This finite derivation selects the recursive rule once. ### Step 4. Match the verb and object V ⇒ saw. Choose NP → Det N, then Det ⇒ a and N ⇒ girl. ### Step 5. Expand the prepositional phrase PP ⇒ P NP. P ⇒ in. NP ⇒ Det N, with Det ⇒ the and N ⇒ park. ### Step 6. Check the complete leaf order The leaves read “the tall boy saw a girl in the park”, exactly the normalized input. The PP “in the park” attaches to the VP under this supplied grammar. SNPVPDetAdjNVPPPVNPPNPDetNDetNThetallboysawagirlinthepark Top-down parse tree showing the PP attached to VP. ### Complete derivation S ⇒ NP VP ⇒ Det Adj N VP ⇒ the tall boy VP ⇒ the tall boy VP PP ⇒ the tall boy V NP PP ⇒ the tall boy saw Det N PP ⇒ the tall boy saw a girl P NP ⇒ the tall boy saw a girl in Det N ⇒ the tall boy saw a girl in the park. **Implementation caution:** VP → VP PP is left-recursive. A naive recursive-descent implementation may loop; it must transform or otherwise handle the grammar. A valid finite top-down derivation, as shown above, still exists. **Yaad rakho:** Rule jo diya hai wahi use karo: yahan “in the park” VP ka part hai. --- ## M2 · Q3B · Lesk word-sense disambiguation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/reexam-2026--reexam-2026-q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/reexam-2026/reexam-2026-q3b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Consider the sentence: “The bank can ensure your deposits are safe.” The word “bank” has the following two possible senses (WordNet-style glosses): Bank₁: Sloping land especially beside a body of water. Bank₂: A financial institution that accepts deposits and channels the money into lending activities. i. Apply the LESK algorithm to determine the correct sense of the word “bank” in the given sentence. Show the steps clearly. ii. Briefly discuss limitations of the LESK algorithm in handling ambiguous words in real-world contexts. [10 marks] ### Answer ### Given Context sentence and the two supplied dictionary glosses. Use a simplified Lesk method: lowercase, remove function words, exclude the target bank and compare distinct content-word overlap. Do not add synonyms that the question does not supply. ### To find Which candidate gloss has the greatest overlap with the local context. ### Formula: which one and why $$ \mathrm{score}(s)=|C\cap G_s|,\qquad \hat s=\arg\max_s\mathrm{score}(s) $$ C is the set of context content words. Gₛ is the content-word set of the gloss for sense s. The intersection contains words appearing in both; its size is the score. This is an overlap count, not a probability. ### Solution ### Step 1. Prepare the context Remove The, can, your, are and the target bank. A reasonable content set is C = {ensure, deposits, safe}. If “can” is retained under another fixed stop-list it matches neither gloss and does not change the result. ### Step 2. Prepare Bank₁’s gloss G₁ = {sloping, land, especially, beside, body, water}. There is no shared word with {ensure, deposits, safe}. ### Step 3. Calculate Bank₁’s score $$ C\cap G_1=\varnothing,\qquad \mathrm{score}(1)=0 $$ ### Step 4. Prepare Bank₂’s gloss and count overlap G₂ = {financial, institution, accepts, deposits, channels, money, lending, activities}. The shared word is deposits. ### Step 5. Calculate Bank₂’s score and select $$ C\cap G_2=\{\mathrm{deposits}\},\qquad \mathrm{score}(2)=1 $$ 1 > 0, so choose the financial-institution sense. ### Final answer **Bank₂: financial institution.** The explicit overlap “deposits” supports this choice. **Yaad rakho:** Lesk: context aur har meaning ki definition mein common words count karo; highest overlap choose karo. ### ii. Limitations - Short contexts or short glosses may produce no overlaps or ties. - Exact word matching misses related words and synonyms, such as deposit versus savings, unless an extension is defined. - Irrelevant frequent words can create misleading overlaps if preprocessing is poor. - Dictionary glosses may omit a domain-specific sense or fail to express the context’s wording. - Simplified Lesk uses limited local information and may miss discourse or real-world knowledge. Lemmatization, richer glosses and semantic similarity can improve variants, but the exact variant and tie-breaking rule should be stated. --- ## M2 · Q4B · A second HMM sequence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/papers/reexam-2026--reexam-2026-q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/papers/reexam-2026/reexam-2026-q4b M2-relevant questions from an actual TEE paper · original marks retained for longer practice Question. Consider the following POS tagged corpus. Apply Hidden Markov Model (HMM) approach to: 1. Dog/Noun eats/Verb bread/Noun 2. The/Det cat/Noun sleeps/Verb 3. A/Det dog/Noun eats/Verb fish/Noun i. Construct the emission probability matrix. ii. Construct the transmission probability matrix. iii. Calculate the probability of the given sequence of events: The/Det dog/Noun eats/Verb bread/Noun. Note: Include start and end characters for all the sentences. [10 marks] ### Answer The repeated verb eats has a different emission count from the final paper; recalculate it rather than copying the previous numerical result. ### Given | Sentence | Tagged sequence | | --- | --- | | 1 | dog/Noun eats/Verb bread/Noun | | 2 | the/Det cat/Noun sleeps/Verb | | 3 | a/Det dog/Noun eats/Verb fish/Noun | Lowercase the corpus. Test sequence: the/Det dog/Noun eats/Verb bread/Noun. Add Start and End markers. ### To find Emission and transition matrices, then the probability of the supplied tagged sequence. ### Formula: which one and why $$ P(w\mid t)=\frac{C(t,w)}{C(t)} $$ $$ P(t_j\mid t_i)=\frac{C(t_i,t_j)}{\sum_kC(t_i,t_k)} $$ Emission divides the word’s tag-specific count by the total tokens with that tag. Transition divides the tag-pair count by all transitions leaving the previous tag, including End. $$ P(\mathbf w,\mathbf t)=P(t_1\mid\mathrm{Start})\prod_iP(w_i\mid t_i)\prod_{i=2}^{n}P(t_i\mid t_{i-1})P(\mathrm{End}\mid t_n) $$ ### Solution ### Step 1. Count tag totals and word frequencies Det: the=1, a=1, total2. Noun: dog=2, bread=1, cat=1, fish=1, total5. Verb: eats=2, sleeps=1, total3. ### Step 2. Construct emission matrix | Tag / word | a | bread | cat | dog | eats | fish | sleeps | the | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Det | 1/2 | 0 | 0 | 0 | 0 | 0 | 0 | 1/2 | | Noun | 0 | 1/5 | 1/5 | 2/5 | 0 | 1/5 | 0 | 0 | | Verb | 0 | 0 | 0 | 0 | 2/3 | 0 | 1/3 | 0 | ### Step 3. Count transitions from the three chains Start → Noun → Verb → Noun → End Start → Det → Noun → Verb → End Start → Det → Noun → Verb → Noun → End | From | Det count | Noun count | Verb count | End count | Total | | --- | --- | --- | --- | --- | --- | | Start | 2 | 1 | 0 | 0 | 3 | | Det | 0 | 2 | 0 | 0 | 2 | | Noun | 0 | 0 | 3 | 2 | 5 | | Verb | 0 | 2 | 0 | 1 | 3 | ### Step 4. Divide outgoing counts | From / to | Det | Noun | Verb | End | | --- | --- | --- | --- | --- | | Start | 2/3 | 1/3 | 0 | 0 | | Det | 0 | 1 | 0 | 0 | | Noun | 0 | 0 | 3/5 | 2/5 | | Verb | 0 | 2/3 | 0 | 1/3 | All nonterminal rows sum to one. No smoothing is requested. ### Step 5. Write every sequence factor | Event | Probability | | --- | --- | | Start → Det | 2/3 | | Det emits the | 1/2 | | Det → Noun | 1 | | Noun emits dog | 2/5 | | Noun → Verb | 3/5 | | Verb emits eats | 2/3 | | Verb → Noun | 2/3 | | Noun emits bread | 1/5 | | Noun → End | 2/5 | ### Step 6. Multiply all factors $$ P=\frac23\times\frac12\times1\times\frac25\times\frac35\times\frac23\times\frac23\times\frac15\times\frac25 $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. Start → Det | $$ 1\times\frac{2}{3}=\frac{2}{3} $$ | | 2. Det emits the | $$ \frac{2}{3}\times\frac{1}{2}=\frac{1}{3} $$ | | 3. Det → Noun | $$ \frac{1}{3}\times1=\frac{1}{3} $$ | | 4. Noun emits dog | $$ \frac{1}{3}\times\frac{2}{5}=\frac{2}{15} $$ | | 5. Noun → Verb | $$ \frac{2}{15}\times\frac{3}{5}=\frac{2}{25} $$ | | 6. Verb emits eats | $$ \frac{2}{25}\times\frac{2}{3}=\frac{4}{75} $$ | | 7. Verb → Noun | $$ \frac{4}{75}\times\frac{2}{3}=\frac{8}{225} $$ | | 8. Noun emits bread | $$ \frac{8}{225}\times\frac{1}{5}=\frac{8}{1125} $$ | | 9. Noun → End | $$ \frac{8}{1125}\times\frac{2}{5}=\frac{16}{5625} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{16}{5625}\approx0.0028444444 $$ ### Final answer **P = 16/5625 ≈ 0.0028444444** for these words and specified tags jointly. The two occurrences of eats give P(eats | Verb) = 2/3. **Yaad rakho:** Same tag structure ho sakta hai, lekin word emission counts dobara count karna. **Boundary reminder:** Include P(Det | Start) = 2/3 and P(End | Noun) = 2/5. The handwritten key’s final line instead inserts the overall Det fraction 2/10 and omits End; the complete boundary-aware probability requested here is 0.0028444444. --- ## M2 · Paper instructions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-a--instructions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-a/instructions M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice **Practice paper · 10 marks · 60 minutes.** Q1 and Q2 are compulsory (2 marks each). Attempt any two of Q3, Q4 and Q5 (3 marks each). All five solutions are included below their questions. These are newly authored practice questions, not predictions of the actual paper. Use about 8 minutes each for Q1–Q2, 16 minutes each for the two chosen 3-mark questions, and the remaining 12 minutes to read, choose and check. Write the formula and substitutions even if a final calculation gets stuck. --- ## M2 · Q1 (2 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-a--q1/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-a/q1 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q1 For “we study language models”, list the unigrams, bigrams and trigrams. Do not add boundary markers. (2 marks) **Given** Four tokens in the stated order. **To find** All contiguous n-grams for n=1,2,3. ### Formula and why it applies $$ \#\text{ n-grams}=T-n+1 $$ T is the number of tokens. Slide a window of n consecutive tokens, one position at a time. ### Solution #### Step 1. Unigrams, window size 1 (we), (study), (language), (models); count=4. #### Step 2. Bigrams, window size 2 (we study), (study language), (language models); count=4−2+1=3. #### Step 3. Trigrams, window size 3 (we study language), (study language models); count=4−3+1=2. 4 unigrams, 3 bigrams and 2 trigrams, listed above. --- ## M2 · Q2 (2 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-a--q2/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-a/q2 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q2 A corpus has C(read)=10, C(read books)=6 and C(read papers)=4. Find both bigram conditional probabilities. (2 marks) **Given** read has 10 continuation opportunities; 6 are books and 4 are papers. **To find** P(books | read) and P(papers | read). ### Formula and why it applies $$ P(w\mid h)=\frac{C(hw)}{C(h)} $$ h is the preceding word. Divide each pair count by the count of its shared history. ### Solution #### Step 1. Probability of books $$ P(\text{books}\mid\text{read})=\frac6{10}=0.6 $$ #### Step 2. Probability of papers $$ P(\text{papers}\mid\text{read})=\frac4{10}=0.4 $$ #### Step 3. Check the complete row $$ 0.6+0.4=1 $$ P(books | read)=0.6 and P(papers | read)=0.4. --- ## M2 · Q3 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-a--q3/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-a/q3 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q3 Training corpus: “I like tea” and “I like coffee”. Lowercase all words and add and to each sentence. Using add-one bigram smoothing and a vocabulary containing both boundary symbols, find P(“i like tea”). (3 marks) **Given** Padded corpus: i like tea ; i like coffee . V={, i, like, tea, coffee, }, |V|=6. **To find** Product of the four smoothed transition probabilities, including the end transition. ### Formula and why it applies $$ P_{+1}(w\mid h)=\frac{C(hw)+1}{C(h)+|V|} $$ Every possible vocabulary continuation gets one pseudocount. Both boundaries are counted in V because the question explicitly requires the course convention. ### Solution #### Step 1. Count histories and pairs | Transition | Pair count | History count | | --- | --- | --- | | → i | 2 | 2 | | i → like | 2 | 2 | | like → tea | 1 | 2 | | tea → | 1 | 1 | #### Step 2. Compute first probability $$ P(i\mid\langle s\rangle)=\frac{2+1}{2+6}=\frac38 $$ #### Step 3. Compute middle probabilities $$ P(\text{like}\mid i)=\frac38 $$ $$ P(\text{tea}\mid\text{like})=\frac{1+1}{2+6}=\frac14 $$ #### Step 4. Compute end probability $$ P(\langle/ s\rangle\mid\text{tea})=\frac{1+1}{1+6}=\frac27 $$ #### Step 5. Multiply $$ P=\frac38\times\frac38\times\frac14\times\frac27=\frac9{896}\approx0.010044643 $$ Sentence probability = 9/896 ≈ 0.010044643. --- ## M2 · Q4 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-a--q4/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-a/q4 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q4 An HMM has P(N|start)=0.6, P(V|N)=0.5, P(end|V)=0.4, P(dogs|N)=0.5 and P(run|V)=0.8. Find the joint probability of words “dogs run” with tag path N,V, including the end transition. (3 marks) **Given** The five required transition/emission factors are supplied. **To find** P(dogs run, N V, end). ### Formula and why it applies $$ P(w,t)=P(t_1\mid start)P(w_1\mid t_1)P(t_2\mid t_1)P(w_2\mid t_2)P(end\mid t_2) $$ Transitions move between tags. Emissions generate words from tags. Include both types of factor. ### Solution #### Step 1. Place the factors along the path start → N: 0.6 → emit dogs: 0.5 → N → V: 0.5 → emit run: 0.8 → V → end: 0.4 #### Step 2. First word contribution $$ 0.6\times0.5=0.3 $$ #### Step 3. Enter V and emit run $$ 0.3\times0.5=0.15 $$ $$ 0.15\times0.8=0.12 $$ #### Step 4. End the sentence $$ 0.12\times0.4=0.048 $$ Joint path-and-word probability = 0.048. This is not a normalised posterior over tag paths. --- ## M2 · Q5 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-a--q5/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-a/q5 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q5 For a two-sense Naive Bayes WSD model, P(F)=P(R)=0.5. After smoothing, P(loan|F)=0.4, P(money|F)=0.3, P(loan|R)=0.1 and P(money|R)=0.1. Classify “bank” using context {loan, money} and obtain the winning sense’s posterior. (3 marks) **Given** F=financial sense; R=river sense. Two exhaustive senses; conditional independence of the context words is assumed. **To find** Two scores, selected sense and normalised posterior. ### Formula and why it applies $$ s(c)=P(c)P(\text{loan}\mid c)P(\text{money}\mid c) $$ $$ P(c\mid x)=\frac{s(c)}{s(F)+s(R)} $$ Multiply the prior by the two likelihoods, then normalise because a posterior is requested. ### Solution #### Step 1. Financial score $$ s(F)=0.5\times0.4\times0.3=0.06 $$ #### Step 2. River score $$ s(R)=0.5\times0.1\times0.1=0.005 $$ #### Step 3. Total evidence score $$ 0.06+0.005=0.065 $$ #### Step 4. Normalise winner $$ P(F\mid x)=\frac{0.06}{0.065}=\frac{12}{13}\approx0.923077 $$ Financial sense; posterior ≈0.923077. --- ## M2 · Paper instructions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-b--instructions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-b/instructions M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice **Practice paper · 10 marks · 60 minutes.** Q1 and Q2 are compulsory (2 marks each). Attempt any two of Q3, Q4 and Q5 (3 marks each). All five solutions are included below their questions. These are newly authored practice questions, not predictions of the actual paper. Use about 8 minutes each for Q1–Q2, 16 minutes each for the two chosen 3-mark questions, and the remaining 12 minutes to read, choose and check. Write the formula and substitutions even if a final calculation gets stuck. --- ## M2 · Q1 (2 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-b--q1/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-b/q1 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q1 A language model gives a test sequence probability of 0.0016 across four predicted tokens. Calculate perplexity. (2 marks) **Given** P=0.0016; N=4. **To find** PP. ### Formula and why it applies $$ PP=P^{-1/N}=\left(\frac1P\right)^{1/N} $$ Use the number of predicted events, not an unrelated training token count. ### Solution #### Step 1. Reciprocal $$ \frac1{0.0016}=625 $$ #### Step 2. Fourth root $$ PP=625^{1/4}=5 $$ Perplexity=5. --- ## M2 · Q2 (2 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-b--q2/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-b/q2 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q2 A first-order Markov model starts in state A with probability 0.7. P(B|A)=0.4 and P(A|B)=0.3. Compute the path probability A→B→A, with no end-state factor. (2 marks) **Given** Start A=0.7; A→B=0.4; B→A=0.3. **To find** Probability of this state path. ### Formula and why it applies $$ P(A,B,A)=P(A)P(B\mid A)P(A\mid B) $$ First-order dependence uses only the previous state for each transition. ### Solution #### Step 1. First transition contribution $$ 0.7\times0.4=0.28 $$ #### Step 2. Second transition contribution $$ 0.28\times0.3=0.084 $$ Path probability=0.084. --- ## M2 · Q3 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-b--q3/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-b/q3 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q3 For a fixed vocabulary of five possible continuations, C(very)=7 and C(very bright)=0. Calculate the unsmoothed and Laplace-smoothed probabilities of bright after very. (3 marks) **Given** History count=7; pair count=0; |V|=5. **To find** MLE and add-one probabilities. ### Formula and why it applies $$ P_{MLE}(w\mid h)=\frac{C(hw)}{C(h)} $$ $$ P_{+1}(w\mid h)=\frac{C(hw)+1}{C(h)+|V|} $$ MLE uses only observations. Laplace reserves probability for all known vocabulary outcomes, including unseen pairs. ### Solution #### Step 1. Unsmoothed estimate $$ P_{MLE}(\text{bright}\mid\text{very})=\frac07=0 $$ #### Step 2. Add one to the pair count $$ 0+1=1 $$ #### Step 3. Add vocabulary size to the history count $$ 7+5=12 $$ #### Step 4. Divide $$ P_{+1}(\text{bright}\mid\text{very})=\frac1{12}\approx0.083333 $$ MLE=0; Laplace=1/12≈0.083333. The zero count means unseen in this corpus, not impossible language. --- ## M2 · Q4 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-b--q4/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-b/q4 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q4 A two-state HMM has initial probabilities P(N)=0.6, P(V)=0.4. Transition rows (N→N,N→V)=(0.3,0.7) and (V→N,V→V)=(0.8,0.2). Emissions are P(fish|N)=0.5, P(fish|V)=0.4, P(swim|N)=0.1 and P(swim|V)=0.6. Find the best tag path for “fish swim”, without end transitions. (3 marks) **Given** Two states N,V; all start, transition and relevant emission probabilities are supplied. **To find** Largest joint score and its tag sequence. ### Formula and why it applies $$ \delta_1(t)=P(t)P(w_1\mid t) $$ $$ \delta_2(t)=P(w_2\mid t)\max_s[\delta_1(s)P(t\mid s)] $$ Viterbi keeps the best path ending at each tag and stores its predecessor. Enumerating four paths gives the same result for this tiny example. ### Solution #### Step 1. Initialise fish $$ \delta_1(N)=0.6\times0.5=0.3 $$ $$ \delta_1(V)=0.4\times0.4=0.16 $$ #### Step 2. Score both paths ending in N $$ NN:0.3\times0.3\times0.1=0.009 $$ $$ VN:0.16\times0.8\times0.1=0.0128 $$ Keep VN at N; predecessor V. #### Step 3. Score both paths ending in V $$ NV:0.3\times0.7\times0.6=0.126 $$ $$ VV:0.16\times0.2\times0.6=0.0192 $$ Keep NV at V; predecessor N. #### Step 4. Choose and backtrack 0.126 > 0.0128. End at V and follow its stored predecessor N. Best sequence: N,V, with joint score 0.126. --- ## M2 · Q5 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-b--q5/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-b/q5 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q5 Apply simplified Lesk to “bank” with preprocessed context {river, water, flooded}. Gloss R is {sloping, land, river, water}; gloss F is {financial, institution, money, loan}. (3 marks) **Given** Two already preprocessed gloss sets and one context set. **To find** Overlap scores and selected sense. ### Formula and why it applies $$ \text{score}(s)=|\text{context}\cap\text{gloss}(s)| $$ Simplified Lesk chooses the sense whose gloss shares the largest number of context words, using the specified set-overlap convention. ### Solution #### Step 1. Intersect the river gloss {river, water, flooded} ∩ {sloping, land, river, water} = {river, water}. #### Step 2. Count the river overlap $$ score(R)=2 $$ #### Step 3. Intersect and count the financial gloss No shared words; intersection is empty, so score(F)=0. #### Step 4. Compare 2 > 0, hence select the river-bank sense. River-bank sense, with overlap 2 versus 0. --- ## M2 · Paper instructions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-c--instructions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-c/instructions M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice **Practice paper · 10 marks · 60 minutes.** Q1 and Q2 are compulsory (2 marks each). Attempt any two of Q3, Q4 and Q5 (3 marks each). All five solutions are included below their questions. These are newly authored practice questions, not predictions of the actual paper. Use about 8 minutes each for Q1–Q2, 16 minutes each for the two chosen 3-mark questions, and the remaining 12 minutes to read, choose and check. Write the formula and substitutions even if a final calculation gets stuck. --- ## M2 · Q1 (2 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-c--q1/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-c/q1 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q1 A corpus contains five occurrences tagged Noun. Three of them are “cat”. It contains four Verb tokens, one of which is “cat”. Calculate P(cat|Noun) and P(cat|Verb). (2 marks) **Given** C(N)=5; C(cat,N)=3; C(V)=4; C(cat,V)=1. **To find** The two emission probabilities. ### Formula and why it applies $$ P(w\mid t)=\frac{C(w,t)}{C(t)} $$ An emission conditions on the tag, so the denominator counts tokens of that tag. ### Solution #### Step 1. Noun emission $$ P(\text{cat}\mid N)=\frac35=0.6 $$ #### Step 2. Verb emission $$ P(\text{cat}\mid V)=\frac14=0.25 $$ P(cat|N)=0.6 and P(cat|V)=0.25. --- ## M2 · Q2 (2 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-c--q2/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-c/q2 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q2 Each of five words may receive any of three POS tags. How many possible tag sequences exist before applying constraints? (2 marks) **Given** T=5 token positions; K=3 choices at each position. **To find** Total tag sequences. ### Formula and why it applies $$ K^T $$ Choose one tag independently for each position when counting candidates; multiply the number of options five times. ### Solution #### Step 1. Expand the power $$ 3^5=3\times3\times3\times3\times3 $$ #### Step 2. Multiply $$ 3\times3=9,\quad9\times3=27 $$ $$ 27\times3=81,\quad81\times3=243 $$ 243 tag sequences. --- ## M2 · Q3 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-c--q3/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-c/q3 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q3 For trigram prediction, C(i like)=5, C(i like tea)=3 and C(i like coffee)=2. Find the MLE probability of each continuation. Then find add-one P(tea|i like) using a continuation vocabulary of size 4. (3 marks) **Given** Two-word history i like occurs 5 times; continuations tea=3, coffee=2; |V|=4. **To find** Two MLE values and the smoothed tea probability. ### Formula and why it applies $$ P(w_i\mid w_{i-2},w_{i-1})=\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})} $$ $$ P_{+1}(w\mid h)=\frac{C(hw)+1}{C(h)+|V|} $$ A trigram conditions on two preceding words. The same add-one count rule applies to that longer history. ### Solution #### Step 1. Tea MLE $$ P(\text{tea}\mid i,\text{like})=\frac35=0.6 $$ #### Step 2. Coffee MLE $$ P(\text{coffee}\mid i,\text{like})=\frac25=0.4 $$ #### Step 3. Smooth tea $$ P_{+1}(\text{tea}\mid i,\text{like})=\frac{3+1}{5+4}=\frac49\approx0.444444 $$ MLE tea=0.6; coffee=0.4; smoothed tea=4/9≈0.444444. --- ## M2 · Q4 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-c--q4/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-c/q4 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q4 Training sentences with tags are: “the/D cat/N sleeps/V” and “a/D dog/N runs/V”. Estimate start/end-inclusive transition probabilities and the joint probability of “the cat sleeps” with path D,N,V. (3 marks) **Given** Two paths start→D→N→V→end. Four distinct words the,a,cat,dog and two verbs sleeps,runs. Use unsmoothed estimates and lowercase tokens. **To find** Transitions, required emissions and full path score. ### Formula and why it applies $$ P(t_j\mid t_i)=\frac{C(t_i\to t_j)}{\sum_k C(t_i\to t_k)} $$ $$ P(w\mid t)=\frac{C(w,t)}{C(t)} $$ Transition denominators count outgoing transitions from the previous state. Emission denominators count all tokens tagged with the emitting state. ### Solution #### Step 1. Count transitions start→D, D→N, N→V and V→end each occur twice. Each state has just one observed next state. #### Step 2. Divide to get probabilities $$ P(D\mid start)=P(N\mid D)=P(V\mid N)=P(end\mid V)=\frac22=1 $$ #### Step 3. Estimate emissions $$ P(\text{the}\mid D)=\frac12 $$ $$ P(\text{cat}\mid N)=\frac12 $$ $$ P(\text{sleeps}\mid V)=\frac12 $$ #### Step 4. Multiply in word order $$ P=1\times\frac12\times1\times\frac12\times1\times\frac12\times1=\frac18=0.125 $$ Joint probability=0.125. Each of the four observed transition probabilities is 1. --- ## M2 · Q5 (3 marks) Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/mocks/mock-c--q5/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/mocks/mock-c/q5 M2 · confirmed 2 + 2 + any two of 3 + 3 + 3 · worked practice ### Q5 In a WSD training set, financial bank and river bank each have two labelled contexts. Token totals are N(F)=6 and N(R)=6. Vocabulary size is 8. Counts are C(loan,F)=2, C(loan,R)=0, C(water,F)=0, C(water,R)=3. Use add-one Naive Bayes to classify context {loan,water}. (3 marks) **Given** P(F)=P(R)=2/4=0.5; |V|=8; both class token totals=6. **To find** Smoothed scores and winning sense. ### Formula and why it applies $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ $$ s(c)=P(c)\prod_wP(w\mid c) $$ Class priors use labelled-context counts; word likelihood denominators use class token totals. Do not exchange those two totals. ### Solution #### Step 1. Find the denominator $$ 6+8=14 $$ #### Step 2. Financial likelihoods $$ P(loan\mid F)=\frac3{14},\quad P(water\mid F)=\frac1{14} $$ #### Step 3. River likelihoods $$ P(loan\mid R)=\frac1{14},\quad P(water\mid R)=\frac4{14} $$ #### Step 4. Compute financial score $$ s(F)=\frac12\times\frac3{14}\times\frac1{14}=\frac3{392}\approx0.007653061 $$ #### Step 5. Compute river score $$ s(R)=\frac12\times\frac1{14}\times\frac4{14}=\frac4{392}\approx0.010204082 $$ #### Step 6. Choose 4/392 > 3/392; the water evidence outweighs the loan evidence under these counts. River sense wins. If normalised, P(R|context)=4/7≈0.571429. --- ## M2 · Last-day reading order Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/revision/m2-revision--order/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/revision/m2-revision/order M2 · quick revision after completing the lessons Start with the families most likely to require arithmetic: **smoothed bigrams, HMM probabilities and Naive Bayes WSD**. Then revise perplexity, n-gram counts and Lesk. Parsing, POS approaches, WordNet and lexical relations remain in the confirmed portion; do not remove them merely because the expected paper is mostly numerical. | Time block | Read and work | What your written solution should contain | | --- | --- | --- | | First 60–90 min | N-grams, MLE and Laplace examples | Padded corpus → vocabulary → count table → each fraction → full product. | | Next 60–90 min | HMM transition/emission and path examples | Tag totals → transition counts → matrices → alternating transition/emission factors. | | Next 60–90 min | WSD and Lesk examples | Class priors → vocabulary → class totals → smoothed word likelihoods → score comparison. | | Next 45–60 min | Perplexity, POS, parsing, WordNet, NER | Definitions, one meaningful example, a tree or table where needed. | | Final 60 min | One M2 mock under exam time | Both 2-mark questions and any two 3-mark questions; compare every step with the answers. | ### The writing format to repeat Given → To find → Formula + meaning → Numbered calculations → Answer If the question supplies the corpus, draw the count table before using the calculator. A long product should be built from fractions that you have already justified. Keep fractions or stored calculator values until the end. --- ## M2 · Recognise the question and start correctly Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/revision/m2-revision--method/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/revision/m2-revision/method M2 · quick revision after completing the lessons | Question clue | Use this method | First thing to write | | --- | --- | --- | | “next word”, “sentence probability” | N-gram model | List the needed word transitions. | | “unseen pair”, “Laplace” | Add-one smoothing | State V and count each history. | | “perplexity” | Reciprocal Nth root | State P(test sequence) and N. | | “transition/emission matrix” | HMM count estimation | Separate tag-to-tag counts from tag-to-word counts. | | “given tag sequence” | HMM joint path | Initial × emission × transition × emission … | | “best tag sequence” | Viterbi or enumerate tiny paths | Initialise one score per tag; retain predecessor maxima. | | “gloss overlap” | Simplified Lesk | Preprocess context and glosses consistently. | | “classify word sense with counts” | Naive Bayes WSD | Priors from examples, likelihoods from tokens. | | “parse tree” | Given CFG + chosen parser | Show grammar expansions or stack reductions. | ### Errors that change the answer 1. Dividing a conditional count by the whole corpus instead of its history count. 2. Adding 1 to the numerator without adding |V| to the denominator. 3. Forgetting the end transition. 4. Counting repeated words as new vocabulary types. 5. Using P(tag|word) in place of P(word|tag). 6. Calling an unnormalised score a posterior. 7. Silently mixing case, lemmatisation or boundary conventions. ### Short theory answer format For 2 marks: give the exact definition and one specific example or distinction. For 3 marks: add the mechanism or two ordered steps; include a small diagram/table where it carries information. The actual allocation is examiner-dependent, so these are writing guides, not an official marking scheme. --- ## M2 · What this NLP guide covers Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/about/scope--sources/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/about/scope/sources NLP · M2 and TEE · edition 3 October 2026 **M2 portion confirmed by you:** Unit 3 from PDF slide 62 to 160; Unit 4 complete (128 slides); Unit 5 slides 1–57. Slide 62 of Unit 3 starts N-grams. Slide 57 of Unit 5 completes the Naive Bayes WSD example. Vector-space models start on slide 58 and belong to TEE here. **M2 pattern confirmed by you:** both 2-mark questions compulsory; any two of the three 3-mark questions. Total attempted 10. Course policy gives one hour. Three newly authored, fully solved M2 practice papers use that pattern. **TEE:** all seven units of the official AY 2026–27 syllabus. The new ZIP supplies faculty slide packs for Units 1–5 plus a regular-expressions handout. Units 6–7 are syllabus-based explanations supplemented with the standard references below; their faculty slide packs were not supplied. The official syllabus includes GloVe and spelling correction, and both are retained even though the policy’s lecture table abbreviates them. **Previous papers found:** final exam dated 2 December 2025 and re-exam dated 9 February 2026, with answer keys, inside Semester V.zip. Programme copies were deduplicated. Every supplied question subpart is mapped to the current syllabus; no question was excluded as out of syllabus. Only wholly M2-eligible subparts appear in the M2 paper tab; mixed questions remain in the complete TEE papers. The current syllabus lists course code 702AI0C037. The supplied policy and old papers use 702AI0C009; the policy also retains an older programme/semester header. Topic coverage follows the current official syllabus and your exact M2 cutoff, rather than treating the old header as a different course. ### Conventions that must stay visible Course bigram answer keys count both start and end symbols in vocabulary size. Worked PYQs state and follow that convention; other exercises state their own vocabulary explicitly. A start symbol is conditioning context, so alternative textbook conventions may exclude it from prediction vocabulary and produce a different smoothed number. HMM path scores and Naive Bayes class scores are labelled as unnormalised when applicable. Any normalisation, case-folding or lemmatisation assumption is stated before counting. ### Reading references - Your uploaded course policy, official Semester V syllabus, five NLP slide packs and regular-expressions handout. - Your uploaded final and re-exam questions and answer keys (AY 2025–26). - [Jurafsky & Martin · Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/) — terminology and standard methods. - [NLTK Book](https://www.nltk.org/book/) — corpus, tagging, parsing and information-extraction concepts. - [Tetreault · A Corpus-Based Evaluation of Centering and Pronoun Resolution](https://aclanthology.org/J01-4003.pdf) — precise Hobbs traversal. Examples are teaching problems unless explicitly identified by their paper and question number. Practice-paper marks indicate intended answer length, not an official forecast. Reading alone cannot guarantee a score; writing the solutions independently is what makes the procedures usable under exam time. ### TEE depth review · 10 October 2026 The core TEE route retains the faculty topics CYK/CKY, grammar conversion, interpolation and Stupid Backoff. Add-k and HMM forward/posterior calculations are available separately under Optional extensions; they are not part of the core reading or formula sequence. The two TEE timed-practice entries reproduce the supplied actual papers and are not additional unseen mock papers. Method references: [Jurafsky and Martin: Hidden Markov Models](https://web.stanford.edu/~jurafsky/slp3/A.pdf); [Jurafsky and Martin: Constituency Parsing](https://web.stanford.edu/~jurafsky/slp3/E.pdf). New numerical inputs and worked traces are teaching examples. --- ## M2 · Video companions in your NotebookLM Canonical: https://deep-learning-study-room.vercel.app/read/nlp/m2/about/scope--videos/ Reader: https://deep-learning-study-room.vercel.app/nlp/#m2/about/scope/videos NLP · M2 and TEE · edition 3 October 2026 Your M2 notebook contains the exact three slide ranges and checked numerical teaching examples. The requested video topics are N-grams and Laplace smoothing, HMM/Viterbi, and Naive Bayes WSD. Use the written solutions here for the complete calculations. [Open your NLP M2 notebook →](https://notebook.google.com/notebook/a270ef1e-c285-4d31-9db1-302090c5f8b1) The notebook is in your signed-in Google account. Video generation and availability are shown in its Studio panel. Generated narration is a supplementary explanation; the numerical results in this guide have been independently checked. ## Complete video catalog updated\_utc : 2026-10-10T11:17:01.698545+00:00 notebooks : m2 : tee : lesson\_map : m2 : foundations/language : - nlp-m2-01-foundations-counts-probability foundations/notation : - nlp-m2-01-foundations-counts-probability foundations/probability : - nlp-m2-01-foundations-counts-probability m2-language-models/words-counts-probability : - nlp-m2-01-foundations-counts-probability foundations/bayes : - nlp-m2-02-bayes-generative-discriminative m2-pos-tagging/generative-discriminative : - nlp-m2-02-bayes-generative-discriminative m2-language-models/n-grams-construction : - nlp-m2-03-ngrams-language-models m2-language-models/collocations-and-lm : - nlp-m2-03-ngrams-language-models m2-language-models/chain-rule-markov : - nlp-m2-03-ngrams-language-models m2-count-to-probability/unigram-estimation : - nlp-m2-03-ngrams-language-models m2-count-to-probability/bigram-count-table : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/trigram-estimation : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/class-count-table-questions : - nlp-m2-04-bigram-trigram-tables m2-evaluation-smoothing/laplace-basics : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/smoothed-sentence : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/adjusted-count-discount : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/evaluation-perplexity : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/log-probabilities : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/class-perplexity-characters : - nlp-m2-06-perplexity-logs-calculator foundations/calculator : - nlp-m2-06-perplexity-logs-calculator m2-pos-tagging/word-classes-pos : - nlp-m2-07-pos-tags-tagger-methods m2-pos-tagging/tagger-methods : - nlp-m2-07-pos-tags-tagger-methods m2-hmm/markov-chain : - nlp-m2-08-markov-hmm-components m2-hmm/hmm-components : - nlp-m2-08-markov-hmm-components m2-hmm/transition-emission-estimation : - nlp-m2-08-markov-hmm-components m2-hmm/hot-cold-paths : - nlp-m2-09-hmm-joint-paths m2-hmm/tagging-corpus-case : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/joint-vs-posterior : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/sequence-count : - nlp-m2-10-viterbi-decoding m2-hmm-decoding/viterbi-explained : - nlp-m2-10-viterbi-decoding m2-chunks-grammar/chunking-chinking : - nlp-m2-11-chunking-chinking m2-chunks-grammar/cfg-foundation : - nlp-m2-12-cfg-top-down-parsing m2-chunks-grammar/structural-ambiguity : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-theory : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-examples : - nlp-m2-12-cfg-top-down-parsing m2-parsing/shift-reduce-theory : - nlp-m2-13-bottom-up-shift-reduce m2-parsing/shift-reduce-examples : - nlp-m2-13-bottom-up-shift-reduce m2-ner/ner-definition : - nlp-m2-14-named-entity-recognition m2-ner/ner-approaches : - nlp-m2-14-named-entity-recognition m2-ner/ner-worked-examples : - nlp-m2-14-named-entity-recognition u5-meaning/u5-words : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-relations : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-wordnet : - nlp-m2-15-lexical-meaning-wordnet u5-wsd/u5-wsd-basics : - nlp-m2-16-lesk-wsd u5-wsd/u5-lesk : - nlp-m2-16-lesk-wsd u5-wsd/u5-walker : - nlp-m2-17-walker-graph-wsd u5-wsd/u5-graph : - nlp-m2-17-walker-graph-wsd u5-naive-bayes/u5-bayes-meaning : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-steps : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-mam : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-practice : - nlp-m2-18-naive-bayes-wsd tee : foundations/language : - nlp-m2-01-foundations-counts-probability foundations/notation : - nlp-m2-01-foundations-counts-probability foundations/probability : - nlp-m2-01-foundations-counts-probability m2-language-models/words-counts-probability : - nlp-m2-01-foundations-counts-probability foundations/bayes : - nlp-m2-02-bayes-generative-discriminative m2-pos-tagging/generative-discriminative : - nlp-m2-02-bayes-generative-discriminative m2-language-models/n-grams-construction : - nlp-m2-03-ngrams-language-models m2-language-models/collocations-and-lm : - nlp-m2-03-ngrams-language-models m2-language-models/chain-rule-markov : - nlp-m2-03-ngrams-language-models m2-count-to-probability/unigram-estimation : - nlp-m2-03-ngrams-language-models m2-count-to-probability/bigram-count-table : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/trigram-estimation : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/class-count-table-questions : - nlp-m2-04-bigram-trigram-tables m2-evaluation-smoothing/laplace-basics : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/smoothed-sentence : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/adjusted-count-discount : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/evaluation-perplexity : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/log-probabilities : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/class-perplexity-characters : - nlp-m2-06-perplexity-logs-calculator foundations/calculator : - nlp-m2-06-perplexity-logs-calculator m2-pos-tagging/word-classes-pos : - nlp-m2-07-pos-tags-tagger-methods m2-pos-tagging/tagger-methods : - nlp-m2-07-pos-tags-tagger-methods m2-hmm/markov-chain : - nlp-m2-08-markov-hmm-components m2-hmm/hmm-components : - nlp-m2-08-markov-hmm-components m2-hmm/transition-emission-estimation : - nlp-m2-08-markov-hmm-components m2-hmm/hot-cold-paths : - nlp-m2-09-hmm-joint-paths m2-hmm/tagging-corpus-case : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/joint-vs-posterior : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/sequence-count : - nlp-m2-10-viterbi-decoding m2-hmm-decoding/viterbi-explained : - nlp-m2-10-viterbi-decoding m2-chunks-grammar/chunking-chinking : - nlp-m2-11-chunking-chinking m2-chunks-grammar/cfg-foundation : - nlp-m2-12-cfg-top-down-parsing m2-chunks-grammar/structural-ambiguity : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-theory : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-examples : - nlp-m2-12-cfg-top-down-parsing m2-parsing/shift-reduce-theory : - nlp-m2-13-bottom-up-shift-reduce m2-parsing/shift-reduce-examples : - nlp-m2-13-bottom-up-shift-reduce m2-ner/ner-definition : - nlp-m2-14-named-entity-recognition m2-ner/ner-approaches : - nlp-m2-14-named-entity-recognition m2-ner/ner-worked-examples : - nlp-m2-14-named-entity-recognition u5-meaning/u5-words : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-relations : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-wordnet : - nlp-m2-15-lexical-meaning-wordnet u5-wsd/u5-wsd-basics : - nlp-m2-16-lesk-wsd u5-wsd/u5-lesk : - nlp-m2-16-lesk-wsd u5-wsd/u5-walker : - nlp-m2-17-walker-graph-wsd u5-wsd/u5-graph : - nlp-m2-17-walker-graph-wsd u5-naive-bayes/u5-bayes-meaning : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-steps : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-mam : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-practice : - nlp-m2-18-naive-bayes-wsd u1-introduction/u1-nlp : - tee-01 u1-introduction/u1-language : - tee-01 u1-introduction/u1-approaches : - tee-01 u1-introduction/u1-nlu-nlg : - tee-02 u1-introduction/u1-pipeline : - tee-02 u1-introduction/u1-challenges : - tee-02 u2-preprocess/u2-tokenize : - tee-03 u2-preprocess/u2-normalization : - tee-03 u2-preprocess/u2-stem-lemma : - tee-03 u2-preprocess/u2-ttr : - tee-03 u2-representation/u2-vocab : - tee-04 u2-representation/u2-label-onehot : - tee-04 u2-representation/u2-bow : - tee-04 u2-representation/u2-tfidf : - tee-05 u2-similarity/u2-cosine : - tee-05 u2-representation/u2-slide-combined : - tee-05 u2-similarity/u2-jaccard : - tee-06 u2-similarity/u2-distances : - tee-06 u2-similarity/u2-choice : - tee-06 u3-corpora/u3-corpus : - tee-07 u3-corpora/u3-types : - tee-07 u3-corpora/u3-access : - tee-07 u3-corpora/u3-resources : - tee-07 u3-corpora/u3-acquisition : - tee-07 u3-corpora/u3-links : - tee-07 u3-regex/regex-language : - tee-08 u3-regex/regex-functions : - tee-08 u3-regex/regex-worked : - tee-08 u3-regex/regex-reference : - tee-08 u5-vectors/u5-distribution : - tee-09 u5-vectors/u5-representation : - tee-09 u5-vectors/u5-sentence : - tee-09 u5-vectors/u5-word2vec : - tee-10 u5-glove/u5-glove-theory : - tee-11 u5-glove/u5-glove-loss : - tee-11 u6/reference : - tee-12 u6/resolution : - tee-12 u6/hobbs : - tee-13 u7/classification : - tee-14 u7/sentiment : - tee-14 u7/evaluation : - tee-14 u7/spelling : - tee-15 u7/translation : - tee-16 u7/qa : - tee-17 u7/extraction : - tee-18 u7/retrieval : - tee-18 written\_only : m2 tee : u3-smoothing-extensions/add-k : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u3-smoothing-extensions/interpolation : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u3-smoothing-extensions/stupid-backoff : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u4-forward-cky/forward-probability : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u4-forward-cky/cnf : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u4-forward-cky/cky : This worked lesson was added after the video plan. No matching video has been generated or reviewed. service\_limit : status : blocked reset\_local : 2026-10-03T22:06:00+05:30 second\_account\_reset\_local : 2026-10-03T22:51:00+05:30 remaining\_not\_submitted : 12 ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Bayes Rule & Models id : nlp-m2-02-bayes-generative-discriminative title : Bayes Rule & Models title\_confirmed : true topic\_title : Bayes rule and generative versus discriminative models mode : m2 kind : focused status : available duration : 10:51 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule id : nlp-m2-07-pos-tags-tagger-methods title : NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule title\_confirmed : true topic\_title : POS tags, ambiguity and tagging methods mode : m2 kind : focused status : available duration : 9:06 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Tagger methods, Brill net gain and the full conditional rule; clarified tagging and imperative explanations. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 10 Viterbi Decoding id : nlp-m2-10-viterbi-decoding title : M2 10 Viterbi Decoding title\_confirmed : true topic\_title : Viterbi decoding with backpointers mode : m2 kind : focused status : available duration : 9:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Chunking and Chinking | Reviewed rules and worked examples id : nlp-m2-11-chunking-chinking title : NLP M2 — Chunking and Chinking | Reviewed rules and worked examples title\_confirmed : true topic\_title : Chunking and chinking from POS-tagged words mode : m2 kind : focused status : available duration : 8:22 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Correct POS labels including JJ = adjective, full NP rule application and chinking boundaries. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 13 — Shift-reduce parsing and backtracking id : nlp-m2-13-bottom-up-shift-reduce title : M2 13 — Shift-reduce parsing and backtracking title\_confirmed : false topic\_title : Shift-reduce parsing and backtracking mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 16 WSD and Lesk Scoring id : nlp-m2-16-lesk-wsd title : M2 16 WSD and Lesk Scoring title\_confirmed : true topic\_title : Word-sense disambiguation and Lesk overlap scoring mode : m2 kind : focused status : available duration : 10:17 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 17 — Walker category scoring and graph-based WSD id : nlp-m2-17-walker-graph-wsd title : M2 17 — Walker category scoring and graph-based WSD title\_confirmed : false topic\_title : Walker category scoring and graph-based WSD mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 01 — NLP foundations: language units, tasks and learning approaches id : tee-01 title : TEE 01 — NLP foundations: language units, tasks and learning approaches title\_confirmed : false topic\_title : NLP foundations: language units, tasks and learning approaches mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLU, NLG, the NLP pipeline and language challenges id : tee-02 title : NLU, NLG, the NLP pipeline and language challenges title\_confirmed : false topic\_title : NLU, NLG, the NLP pipeline and language challenges mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR id : tee-03 title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR title\_confirmed : false topic\_title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Vocabulary, label IDs, one-hot encoding and bag of words id : tee-04 title : Vocabulary, label IDs, one-hot encoding and bag of words title\_confirmed : false topic\_title : Vocabulary, label IDs, one-hot encoding and bag of words mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 05 — TF-IDF and cosine: from the corpus to the final similarity id : tee-05 title : TEE 05 — TF-IDF and cosine: from the corpus to the final similarity title\_confirmed : false topic\_title : TF-IDF and cosine: from the corpus to the final similarity mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski id : tee-06 title : TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski title\_confirmed : false topic\_title : Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Regular expressions: build and explain exact text patterns id : tee-08 title : Regular expressions: build and explain exact text patterns title\_confirmed : false topic\_title : Regular expressions: build and explain exact text patterns mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors id : tee-09 title : TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors title\_confirmed : false topic\_title : Distributional semantics, co-occurrence matrices and sentence vectors mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 10 — Word2Vec: CBOW and skip-gram, training pairs and softmax numericals id : tee-10 title : TEE 10 — Word2Vec: CBOW and skip-gram, training pairs and softmax numericals title\_confirmed : false topic\_title : Word2Vec: CBOW and skip-gram, training pairs and softmax numericals mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### GloVe: global counts, probability ratios and weighted loss id : tee-11 title : GloVe: global counts, probability ratios and weighted loss title\_confirmed : false topic\_title : GloVe: global counts, probability ratios and weighted loss mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Reference and coreference: mentions, antecedents and pronoun resolution id : tee-12 title : Reference and coreference: mentions, antecedents and pronoun resolution title\_confirmed : false topic\_title : Reference and coreference: mentions, antecedents and pronoun resolution mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Hobbs algorithm: parse-tree traversal with three complete traces id : tee-13 title : Hobbs algorithm: parse-tree traversal with three complete traces title\_confirmed : false topic\_title : Hobbs algorithm: parse-tree traversal with three complete traces mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Text classification, sentiment and evaluation numericals id : tee-14 title : Text classification, sentiment and evaluation numericals title\_confirmed : false topic\_title : Text classification, sentiment and evaluation numericals mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 15 — Spelling correction: full edit-distance tables and noisy-channel ranking id : tee-15 title : TEE 15 — Spelling correction: full edit-distance tables and noisy-channel ranking title\_confirmed : false topic\_title : Spelling correction: full edit-distance tables and noisy-channel ranking mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Machine translation: meaning preservation and worked processing id : tee-16 title : Machine translation: meaning preservation and worked processing title\_confirmed : false topic\_title : Machine translation: meaning preservation and worked processing mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Question answering: identify, retrieve and justify the answer id : tee-17 title : Question answering: identify, retrieve and justify the answer title\_confirmed : false topic\_title : Question answering: identify, retrieve and justify the answer mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Information extraction and retrieval: records, indexes and precision/recall id : tee-18 title : Information extraction and retrieval: records, indexes and precision/recall title\_confirmed : false topic\_title : Information extraction and retrieval: records, indexes and precision/recall mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · What NLP is doing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/foundations--language/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/foundations/language Shared language and probability foundations · TEE **Natural Language Processing (NLP)** is the study of computational methods for analysing, interpreting and producing human language. An NLP system receives text or speech and produces an output such as a tag, a meaning, a translation or a probability. A central question in NLP is: **how do counts from a training corpus become probabilities, and how do those probabilities help choose a sentence, a tag or a word sense?** You will learn the vocabulary first, then do the arithmetic with complete working. Training textexamples already observed → Countshow often things occur → Probabilitiescounts divided by totals → Decisioncompare possible answers | Term | Meaning | Concrete example | | --- | --- | --- | | Corpus | A collection of language data. The plural is corpora. | Three sentences saved together form a tiny corpus. | | Document | One text item within a collection. | One review or one article. | | Token | One occurrence produced by tokenization. | “cats like cats” has 3 word tokens. | | Type | A distinct token value. | The same text has 2 types: cats and like. | | Vocabulary | The set of token types recognised by the model. | V = {cats, like}; vocabulary size is 2. | | Training | Estimating rules or numbers using examples. | Count how often like follows cats. | | Test | Applying the learned model to a new example. | Calculate a probability for a sentence not used to build the counts. | ### Three levels you will repeatedly see | Level | Question it answers | Example | | --- | --- | --- | | Word sequence | Which word is likely next? | After “drink”, is “water” likely? | | Syntax | What role or structure does the word have? | “book” is a noun in “a book” and a verb in “book tickets”. | | Semantics | Which meaning is intended? | “bank” near “deposit” means a financial institution. | **Yaad rakho:** Word ka order = language model; word ka role = POS; word ka matlab = sense. ### Your TEE route Read these foundations, then continue through Units 1–7 in order. TEE includes preprocessing and vectors, corpora and regular expressions, language models, syntax, semantics, discourse and applications. The dedicated formula lessons and eight recovered previous papers follow the same methods. Original ambiguities are explained beside the affected answers. Latest TEE pattern:100 marks in three hours; Q1 compulsory plus any four of Q2–Q7. Each full question carries20 marks. Historical December2023 ends at Q6; its choices are labelled separately. The M2 tab remains restricted to your confirmed midterm portion. [Open the complete paper-by-paper map, original marks and full answers →](https://deep-learning-study-room.vercel.app/read/nlp/tee/about/pyq-analysis--all-papers/) ### Prepare the question forms seen in the latest papers first The 2 December 2025 final and 9 February 2026 re-exam both combine theory with applied work. Use their complete questions in [Solved papers](https://deep-learning-study-room.vercel.app/read/nlp/tee/index/papers/); the two Timed papers are those actual papers, not new predictions. - **Repeated calculations:** add-one bigram sentence probability (Q2b in both); HMM emission/transition tables and the joint probability of a specified path (Q4b in both); co-occurrence vectors, cosine and a plot (final Q3b / re-exam Q5b). - **Other observed forms:** TF–IDF matrix (final Q5a); BoW and pairwise cosines (re-exam Q4a); Naive Bayes WSD (final Q5b); Lesk with supplied glosses (re-exam Q3b). - **Theory and diagrams remain essential:** definitions, comparisons, preprocessing, corpus uses, CFG trees, Hobbs traversal, embeddings and application workflows. An explanation question is not automatically a demand for a long calculation. Other faculty topics remain in the full syllabus route, but their authored exercises do not establish past-paper marks or frequency. [Optional extensions](https://deep-learning-study-room.vercel.app/read/nlp/tee/about-chapter/optional-extensions/) can wait until the actual paper methods are secure. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Tokens, counts and symbols Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/foundations--notation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/foundations/notation Shared language and probability foundations · TEE A formula is a short way of stating a calculation. First name the quantities; only then insert the numbers. In this guide, C means a count and P means a probability. | Notation | Read it as | What to do | | --- | --- | --- | | $$ C(\text{cat})=3 $$ | Count of cat is three | Count every occurrence, including repeats. | | $$ P(\text{cat}) $$ | Probability of cat | A number from 0 to 1. | | $$ P(\text{cat}\mid\text{the}) $$ | Probability of cat given the | Restrict attention to occasions where the history is “the”. | | $$ \sum_i x_i $$ | Sum of the x values | Add the terms: x₁ + x₂ + … | | $$ \prod_i x_i $$ | Product of the x values | Multiply the terms: x₁ × x₂ × … | | $$ \arg\max_c s(c) $$ | The choice c with the largest score | Return a label, not the numeric score. | | $$ \max_c s(c) $$ | Largest score | Return a number. | ### Example 1 · tokens and types For the corpus “we study nlp” and “we like nlp”, find the token count, vocabulary and count of “we”. **Given** Two lowercase sentences. Ignore punctuation; do not add boundary markers. **To find** Token count N, vocabulary V, vocabulary size |V| and C(we). ### Formula and why it applies A token is an occurrence; a type is a distinct word. Add sentence lengths to count tokens; remove duplicates only when finding the vocabulary. ### Solution #### Step 1. List every occurrence we, study, nlp, we, like, nlp. #### Step 2. Count tokens $$ N=3+3=6 $$ #### Step 3. Keep one copy of each type V = {we, study, nlp, like}; |V| = 4. #### Step 4. Count the requested word “we” appears once in each sentence, hence C(we) = 2. N = 6; |V| = 4; C(we) = 2. **Yaad rakho:** Token mein repeat gin; type mein duplicate hata. ### Example 2 · sum and product Given probabilities 0.5, 0.2 and 0.1, compute their sum and their product. **Given** Three numbers: 0.5, 0.2, 0.1. **To find** Sum and product; recognise that these are different operations. ### Formula and why it applies $$ \sum_{i=1}^{3}p_i=p_1+p_2+p_3 $$ $$ \prod_{i=1}^{3}p_i=p_1p_2p_3 $$ Add for a sum. Multiply for a product. A sentence-chain calculation normally uses the product of its conditional probabilities. ### Solution #### Step 1. Add the first two numbers $$ 0.5+0.2=0.7 $$ #### Step 2. Complete the sum $$ 0.7+0.1=0.8 $$ #### Step 3. Multiply the first two numbers $$ 0.5\times0.2=0.1 $$ #### Step 4. Complete the product $$ 0.1\times0.1=0.01 $$ Sum = 0.8; product = 0.01. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Probability, without skipped steps Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/foundations--probability/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/foundations/probability Shared language and probability foundations · TEE **Empirical probability** estimates how likely an event is from its observed frequency. Divide the number of relevant observations by the total number of observations in the selected group. For a conditional probability, the denominator is the count of the condition, not the size of the entire corpus. $$ P(A)=\frac{C(A)}{N} $$ $$ P(B\mid A)=\frac{C(A,B)}{C(A)} $$ Here A is the conditioning event, B is the event being predicted, C(A,B) counts observations containing both in the required relationship, and N is the total number of observations. “Given A” tells you which denominator to use. Select the condition“given A” → Count matching outcomesC(A,B) → Divide by condition countC(A) ### Example 1 · word probability A corpus contains 20 tokens; “rain” occurs 5 times. Find the unigram probability of “rain”. **Given** N = 20 and C(rain) = 5. **To find** P(rain). ### Formula and why it applies $$ P(w)=\frac{C(w)}{N} $$ Unigram probability ignores neighbouring words. Every token in the corpus is one observation. ### Solution #### Step 1. Insert the count and total $$ P(\text{rain})=\frac{5}{20} $$ #### Step 2. Divide $$ \frac{5}{20}=0.25 $$ P(rain) = 0.25, or 25%. ### Example 2 · conditional probability In a corpus, “drink” is followed by another word 8 times. It is followed by “water” 6 times. Find P(water | drink). **Given** C(drink as a history) = 8; C(drink, water) = 6. **To find** Probability that the next word is water, given drink. ### Formula and why it applies $$ P(\text{water}\mid\text{drink})=\frac{C(\text{drink water})}{C(\text{drink})} $$ The condition is drink, so use the 8 drink-histories as the denominator. ### Solution #### Step 1. Identify the favourable continuations Six of the eight continuations are water. #### Step 2. Divide $$ P(\text{water}\mid\text{drink})=\frac68=0.75 $$ P(water | drink) = 0.75. **Yaad rakho:** Bar ke right mein jo hai, denominator uska count. ### Example 3 · multiply a sequence A model gives P(a | start) = 0.4, P(cat | a) = 0.5 and P(end | cat) = 0.2. Find the probability of the complete sentence “a cat”. **Given** Start → a = 0.4; a → cat = 0.5; cat → end = 0.2. **To find** Probability of the three-transition path. ### Formula and why it applies $$ P(\text{a cat, end})=P(\text{a}\mid\text{start})P(\text{cat}\mid\text{a})P(\text{end}\mid\text{cat}) $$ Every transition must happen in sequence, so multiply the model’s conditional probabilities. ### Solution #### Step 1. Write the three factors in order $$ P=0.4\times0.5\times0.2 $$ #### Step 2. Multiply the first pair $$ 0.4\times0.5=0.2 $$ #### Step 3. Multiply by the last factor $$ 0.2\times0.2=0.04 $$ Complete-sentence probability = 0.04. ### Example 4 · compare labels Two tag candidates have scores noun = 0.036 and verb = 0.048. Which tag does argmax return? If these are the only candidates, find the normalised noun probability. **Given** s(noun) = 0.036 and s(verb) = 0.048. **To find** Winning label and normalised noun probability. ### Formula and why it applies $$ \hat c=\arg\max_c s(c) $$ $$ P(c)=\frac{s(c)}{\sum_j s(j)} $$ A score need not sum to 1. Dividing each score by the sum gives a distribution over the candidates. ### Solution #### Step 1. Compare scores 0.048 > 0.036, so verb wins. #### Step 2. Find their total $$ 0.036+0.048=0.084 $$ #### Step 3. Divide the noun score by the total $$ P(\text{noun})=\frac{0.036}{0.084}=\frac37\approx0.428571 $$ argmax returns “verb”. The normalised noun probability is approximately 0.428571. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Bayes’ rule: prior, evidence and answer Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/foundations--bayes/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/foundations/bayes Shared language and probability foundations · TEE **Bayes’ rule** reverses a conditional probability. A word-sense classifier asks “which sense is most likely given these context words?” Training data more easily supplies “how likely are these words when this sense is used?” Bayes connects the two. $$ P(c\mid x)=\frac{P(x\mid c)P(c)}{P(x)} $$ | Symbol | Meaning | | --- | --- | | c | A candidate class, tag or word sense. | | x | The observed evidence, such as a context word. | | P(c) | Prior: how common the class is before seeing the evidence. | | P(x | c) | Likelihood: how compatible the evidence is with that class. | | P(c | x) | Posterior: probability of the class after observing the evidence. | | P(x) | Evidence normaliser, shared by all classes for the same observation. | For ranking classes, calculate **prior × likelihood**. The common denominator does not affect which class is largest. For an actual posterior probability, divide by the sum of all class scores. ### A complete Bayes calculation A word has two senses: financial bank and river bank. Priors are 0.6 and 0.4. The probability of observing “loan” is 0.5 under the financial sense and 0.1 under the river sense. Find the most likely sense and its posterior. **Given** P(F) = 0.6; P(R) = 0.4; P(loan | F) = 0.5; P(loan | R) = 0.1. These two senses are exhaustive. **To find** Scores and P(F | loan). ### Formula and why it applies $$ s(c)=P(c)P(\text{loan}\mid c) $$ $$ P(F\mid\text{loan})=\frac{s(F)}{s(F)+s(R)} $$ Multiply prior and likelihood to reverse the conditioning. Then normalise because a posterior was requested. ### Solution #### Step 1. Score the financial sense $$ s(F)=0.6\times0.5=0.30 $$ #### Step 2. Score the river sense $$ s(R)=0.4\times0.1=0.04 $$ #### Step 3. Add the scores $$ P(\text{loan})=0.30+0.04=0.34 $$ #### Step 4. Normalise the financial score $$ P(F\mid\text{loan})=\frac{0.30}{0.34}=\frac{15}{17}\approx0.882353 $$ Financial bank wins; its posterior probability is about 0.882353. **Yaad rakho:** Pehle se kitna common × evidence kitna suit karta = score. ### Bayes Rule & Models id : nlp-m2-02-bayes-generative-discriminative title : Bayes Rule & Models title\_confirmed : true topic\_title : Bayes rule and generative versus discriminative models mode : m2 kind : focused status : available duration : 10:51 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Small probabilities, logs and calculator use Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/foundations--calculator/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/foundations/calculator Shared language and probability foundations · TEE Repeated multiplication of probabilities produces small numbers. A calculator may show 0.0000125 as **1.25 × 10⁻⁵**; they are the same value. A logarithm turns a product into a sum. Log probabilities are often negative because probabilities between 0 and 1 have negative logs. $$ \log(ab)=\log a+\log b $$ $$ \log_b x=y\quad\Longleftrightarrow\quad b^y=x $$ Use the log base specified in the question. Natural log is ln (base e); ordinary calculator log is usually base 10. Perplexity is simplest to evaluate using the direct power formula. Do not mix bases in the same calculation. ### Calculator example · roots in perplexity A model assigns probability 1/16 to a test sequence with 4 predicted tokens. Calculate perplexity. **Given** P = 1/16; N = 4 predicted tokens. **To find** Perplexity PP. ### Formula and why it applies $$ PP=P^{-1/N}=\left(\frac1P\right)^{1/N} $$ Perplexity takes the reciprocal probability and then its Nth root. N must match the number of predicted events in the product. ### Solution #### Step 1. Take the reciprocal $$ \frac{1}{1/16}=16 $$ #### Step 2. Take the fourth root $$ PP=16^{1/4}=2 $$ #### Step 3. Interpret A perplexity of 2 means an effective average uncertainty comparable to two equally likely alternatives. It is not 2% error. PP = 2. ### Calculator entry order Enter the full product with parentheses. For the Nth root enter **(1 ÷ P)^(1 ÷ N)**. Keep at least 6 significant digits while working; round the final answer. If your calculator offers Ans, reuse the stored value instead of typing a heavily rounded result. **Yaad rakho:** Probability ke factors multiply; log mein add; perplexity mein ulta karke root. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · What natural language processing actually does Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u1-introduction--u1-nlp/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u1-introduction/u1-nlp TEE · Unit 1 full **Natural language processing (NLP) is the field concerned with computational methods for analysing, interpreting and generating human language.** Natural languages such as English, Hindi and Marathi evolved through human use; their meanings are not specified by the rigid rules of a programming language. An NLP system may receive text, speech converted to text, or another linguistic representation. Its output depends on the task: a category, a tag for every word, a translated sentence, a ranked document list or a generated answer. It does not always “understand” in the human sense; its performance must be evaluated for the specific task. | Task | Input | Output | | --- | --- | --- | | Sentiment analysis | “The service was excellent.” | Positive sentiment | | Text classification | An email | Spam / not spam | | POS tagging | “Birds fly.” | Birds/NOUN fly/VERB | | Named entity recognition | “Asha joined Google in Mumbai.” | Asha: person; Google: organisation; Mumbai: location | | Machine translation | A Hindi sentence | A corresponding English sentence | | Information retrieval | A search query | Ranked relevant documents | | Question answering | Question plus knowledge/context | An answer | | Summarization | A long article | A shorter text preserving key information | ### How NLP relates to AI, ML and DL **AI** is the broader study of systems performing tasks associated with intelligent behaviour. **Machine learning** learns patterns from data. **Deep learning** uses multiple trainable neural layers. NLP is an application/research field that can use hand-written rules, statistical ML or deep learning. Therefore, not every NLP system is a neural network. Linguistics supplies knowledge about language structure. Statistics supplies uncertainty models. Computer science supplies algorithms, storage and efficient implementation. A practical NLP system combines these ideas. NLP = human language ko computational task mein badlo: label, meaning, search ya generated sentence. ### TEE 01 — NLP foundations: language units, tasks and learning approaches id : tee-01 title : TEE 01 — NLP foundations: language units, tasks and learning approaches title\_confirmed : false topic\_title : NLP foundations: language units, tasks and learning approaches mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Building blocks and levels of linguistic analysis Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u1-introduction--u1-language/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u1-introduction/u1-language TEE · Unit 1 full **A phoneme** is a sound unit that can distinguish meanings in a language, such as the /p/ and /b/ contrast in “pat” and “bat”. A phoneme is a sound, not necessarily one alphabet letter. **A morpheme** is the smallest meaning-bearing grammatical unit: cats = cat + plural -s; unhappy = un- + happy. **A lexeme** is an abstract vocabulary item grouping its inflected forms: write, writes, wrote and writing belong to WRITE. A **phrase** is a group of words functioning together; a **clause** normally contains a predicate and its associated participants; a **sentence** expresses a complete grammatical unit. | Level | What it analyses | Concrete example | | --- | --- | --- | | Phonetics / phonology | Physical sounds / sound patterns in a language | Distinguish bat from pat | | Morphology / lexical analysis | Word structure and lexical items | un + help + ful → unhelpful | | Syntax | Grammatical arrangement and relationships | The dog chased the cat: subject, verb, object | | Semantics | Meaning of words and composed expressions | The dog is the chaser; the cat is chased | | Discourse | Connections across sentences | “Riya arrived. She smiled.” → She refers to Riya | | Pragmatics | Intended meaning in the situation | “It is cold here” may request closing the window | word forms → grammatical structure → sentence meaning → cross-sentence context → speaker intention ### Three worked interpretations **“The cats are sleeping.”** Morphology identifies cat + s as plural. Syntax connects “cats” to “are sleeping”. Semantics represents animals engaged in sleeping. **“Aman bought a laptop. It was expensive.”** Sentence-level syntax alone does not identify “it”. Discourse connects “it” to the laptop across the sentence boundary. **“Wonderful, another power cut!”** The word wonderful is positive literally, but situational context can indicate sarcasm and frustration. This is a pragmatic interpretation. The analysis levels are conceptual descriptions. Modern systems may learn several jointly rather than executing a compulsory separate module for every level. Morphology = word ke andar; syntax = sentence ka structure; semantics = meaning; discourse = sentences ka link; pragmatics = intention. ### TEE 01 — NLP foundations: language units, tasks and learning approaches id : tee-01 title : TEE 01 — NLP foundations: language units, tasks and learning approaches title\_confirmed : false topic\_title : NLP foundations: language units, tasks and learning approaches mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Rule-based, statistical and deep-learning approaches Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u1-introduction--u1-approaches/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u1-introduction/u1-approaches TEE · Unit 1 full **A rule-based system** applies explicit patterns, dictionaries or linguistic rules written by people. Example: mark an email as a candidate company message when its sender matches a known domain. Rules are transparent but can be brittle under paraphrasing. **A statistical/ML system** estimates useful patterns from examples. A supervised classifier learns from texts with known labels. Unsupervised learning finds structure without those labels, such as clusters of news articles. **A deep-learning system** learns layered representations and can model context using neural architectures. | Criterion | Rule based | Statistical ML | Deep learning | | --- | --- | --- | --- | | Main knowledge source | Expert-written rules | Features and training examples | Data and learned representations | | Example | Regex + sentiment lexicon | TF–IDF + Naive Bayes/SVM | Transformer text classifier | | Typical benefit | Interpretable, little training data | Effective with suitable features | Learns complex context | | Typical limitation | Rule maintenance; exceptions | Feature dependence; data mismatch | Compute/data needs; harder interpretation | ### Understanding the learning terms A **training example** is one input instance. A **label** is its desired output. A **feature** is an input property used by the model. **Training** estimates parameters; **inference** applies fixed learned parameters to new input. Separate training, validation and test sets prevent evaluation from merely measuring memorization of examples already seen. **Supervised example:** review → positive/negative, using labelled reviews. **Unsupervised example:** cluster documents by word usage without topic labels. **Reinforcement learning:** choose actions using feedback or reward over interactions; it is not the same as training an ordinary classifier from labelled text. Rules: hum likhte hain. ML: examples se pattern. DL: representation bhi layers se seekhta hai. ### TEE 01 — NLP foundations: language units, tasks and learning approaches id : tee-01 title : TEE 01 — NLP foundations: language units, tasks and learning approaches title\_confirmed : false topic\_title : NLP foundations: language units, tasks and learning approaches mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · NLU versus NLG, explained through one assistant Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u1-introduction--u1-nlu-nlg/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u1-introduction/u1-nlu-nlg TEE · Unit 1 full **Natural language understanding (NLU)** maps language into an interpretation useful for a task: intents, entities, semantic relations or another meaning representation. **Natural language generation (NLG)** produces human-language output from data, a representation or another source. “Book a ticket to Pune tomorrow.” → NLU: intent=book; destination=Pune; date=tomorrow → Application: search available journeys → NLG: “Here are the available journeys to Pune.” | Feature | NLU | NLG | | --- | --- | --- | | Direction | Language → interpretation | Data/representation → language | | Main concern | Meaning, intent, ambiguity | Content choice, organization, fluent wording | | Example | Find destination in a request | Write the booking confirmation | | Common failure | Wrong destination or sense | Unclear, unsupported or ungrammatical output | ### NLG stages in a classical pipeline 1. **Content determination:** choose facts to communicate. 2. **Data interpretation:** identify relationships or trends. 3. **Document planning:** organize facts in a useful order. 4. **Sentence aggregation:** combine related facts without repetition. 5. **Lexical and grammatical realization:** choose words and construct grammatical sentences. 6. **Presentation:** format the generated message for its audience. For a weather report, facts {city=Pune, temperature=28°C, rain=yes} can become “Pune will have rain with a temperature of 28°C.” NLU would instead extract facts or intent from an incoming language message. NLU = language se meaning andar. NLG = data se language bahar. ### NLU, NLG, the NLP pipeline and language challenges id : tee-02 title : NLU, NLG, the NLP pipeline and language challenges title\_confirmed : false topic\_title : NLU, NLG, the NLP pipeline and language challenges mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Generic NLP pipeline, evaluation and deployment Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u1-introduction--u1-pipeline/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u1-introduction/u1-pipeline TEE · Unit 1 full **An NLP pipeline is an organized sequence of operations that transforms raw language into a task output and checks whether that output is useful.** Not every task needs every cleaning operation. Data acquisition → Preprocessing → Feature representation → Model → Evaluation → Deployment and monitoring 1. **Data acquisition:** collect relevant text and, when needed, labels. 2. **Cleaning and preprocessing:** handle markup, tokenize, normalize and decide which information to retain. 3. **Feature engineering:** represent text numerically using BoW, TF–IDF or embeddings. 4. **Modelling:** apply rules or train a statistical/neural model. 5. **Evaluation:** test on held-out data using task-appropriate measures. 6. **Deployment:** make the system available, log suitable feedback and monitor errors or data changes. ### Concrete spam-filter example Collect labelled emails → separate train/test data → tokenize text while preserving meaningful terms → fit vocabulary and IDF on training data → train classifier → evaluate spam detection on the held-out set → deploy an email service filter → monitor legitimate emails wrongly blocked. **Intrinsic evaluation** measures model/task quality, such as precision, recall or tagging accuracy. **Extrinsic evaluation** measures effect in the wider application, such as whether users find answers faster. A higher internal metric does not automatically guarantee a better user outcome. **Data leakage:** learning vocabulary statistics, labels or preprocessing decisions from the test set can make reported performance too optimistic. Fit learned transformations on training data, then apply them to validation/test data. Collect → clean → numbers → model → test → use. Test data ko training ki information mat banao. ### NLU, NLG, the NLP pipeline and language challenges id : tee-02 title : NLU, NLG, the NLP pipeline and language challenges title\_confirmed : false topic\_title : NLU, NLG, the NLP pipeline and language challenges mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Ambiguity and other challenges with precise examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u1-introduction--u1-challenges/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u1-introduction/u1-challenges TEE · Unit 1 full **Ambiguity** means an expression supports more than one interpretation. Good exam answers name the ambiguity and explain both readings. | Challenge | Example | Why a system can fail | | --- | --- | --- | | Lexical/sense ambiguity | bank: finance or river edge | One written word has several senses | | Syntactic ambiguity | I saw the man with a telescope. | The telescope may modify saw or the man | | Reference ambiguity | Riya told Neha that she won. | She may refer to either person without more context | | Sarcasm | Great, another delay. | Literal positive word conveys frustration | | Negation | The film was not good. | Dropping not reverses meaning | | Domain shift | “Positive” in a medical report | Meaning/use differs by domain | | Code mixing | Kal meeting reschedule karna. | Several languages and spelling conventions | | Noisy text | tmrw, repeated letters, typos | Vocabulary/token boundaries become uncertain | | Low-resource setting | Few labelled examples in a language | Models lack representative training evidence | | Bias and privacy | Skewed data or personal details | Errors and harms can persist in deployed systems | **Homophones** sound alike but may differ in spelling (pair/pear). **Homographs** share spelling but can differ in meaning or pronunciation (lead the team / lead metal). These are different dimensions from syntactic ambiguity. Long-distance dependencies, cultural knowledge, idioms, morphology and multilingual writing systems also complicate processing. For “kick the bucket”, literal word-by-word interpretation can miss the idiomatic meaning. For “The book that the students recommended is useful”, the verb is agrees with book across an intervening phrase. Exam mein challenge ka naam + ek sentence + dono meanings/actual failure likho. ### NLU, NLG, the NLP pipeline and language challenges id : tee-02 title : NLU, NLG, the NLP pipeline and language challenges title\_confirmed : false topic\_title : NLU, NLG, the NLP pipeline and language challenges mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Tokenization: sentence, word, character and subword Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-tokenize/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-preprocess/u2-tokenize TEE · Unit 2 PDF pages 1–50 **Tokenization divides text into units called tokens for further processing.** A token may be a word, punctuation mark, character or subword, depending on the tokenizer. **Sentence segmentation** identifies sentence boundaries. Tokenization is not necessarily just splitting on spaces. | Method | Example | Trade-off | | --- | --- | --- | | Word | “cats sleep” → [cats,sleep] | Easy to interpret; unfamiliar words can be unseen tokens | | Character | cat → [c,a,t] | Handles spelling units; creates longer sequences | | Subword | unhappiness → pieces such as [un,happi,ness] in an illustrative vocabulary | Balances vocabulary size and rare-word representation | | Sentence | “Birds fly. Fish swim.” → two sentences | Abbreviations and punctuation complicate boundaries | Actual subword splits depend on the learned vocabulary; the example is illustrative, not a universal tokenizer output. Word segmentation is especially important where spaces do not reliably separate words or in compound-rich languages. **Question. Tokenize “Dr. Rao can't come today.” into words and punctuation under this rule: preserve Dr. and can't as whole tokens, and split only the final full stop.** #### Given The tokenization rule is explicitly supplied. #### To find The resulting token sequence. #### Formula and why it applies Apply the stated boundary rule; do not assume every period or apostrophe is a split point. #### Step 1. Handle abbreviation Dr. remains one token. #### Step 2. Handle contraction can't remains one token. #### Step 3. Write sequence [Dr., Rao, can't, come, today, .] **Answer.** 6 tokens under the specified rule. Another tokenizer may legitimately produce a different sequence. ### Three difficult cases **New York** may be two tokens but one named entity. **3.14** should not normally end a sentence at the decimal point. **don't** may remain whole or split into do + n't, depending on the tokenizer. Always state the convention when calculating counts. Token = chosen unit. Space split ko universal rule mat samajh. ### Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR id : tee-03 title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR title\_confirmed : false topic\_title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Cleaning, case folding, punctuation and stopwords Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-normalization/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-preprocess/u2-normalization TEE · Unit 2 PDF pages 1–50 **Normalization reduces superficial variation so comparable forms can be processed consistently.** Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization. **Stopwords** are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us. **Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.** #### Given Operations and stopword set are fixed. Do not remove not. #### To find The final token list. #### Formula and why it applies Apply the requested operations in order, keeping a visible intermediate result after each transformation. #### Step 1. Lowercase the cats are not sleeping!!! #### Step 2. Tokenize and remove punctuation [the,cats,are,not,sleeping] #### Step 3. Remove specified stopwords Remove the and are → [cats,not,sleeping]. #### Step 4. Lemmatize cats→cat; sleeping→sleep → [cat,not,sleep]. **Answer.** [cat,not,sleep]. The negative meaning is retained. ### Task decides preprocessing For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them. Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi. ### Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR id : tee-03 title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR title\_confirmed : false topic\_title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Stemming versus lemmatization Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-stem-lemma/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-preprocess/u2-stem-lemma TEE · Unit 2 PDF pages 1–50 **Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma. | Criterion | Stemming | Lemmatization | | --- | --- | --- | | Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis | | Output | May be nonword, e.g. studi | Dictionary form, e.g. study | | Context/POS | Usually little or none | Often important | | Irregular forms | May miss went→go | Can map went→go with verb analysis | | Cost | Generally simpler | Usually more linguistic processing | Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation. ### Three worked reductions 1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked. 2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation. 3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter. **Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice. Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan. ### Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR id : tee-03 title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR title\_confirmed : false topic\_title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Type–token ratio: three fully worked sums Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-preprocess--u2-ttr/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-preprocess/u2-ttr TEE · Unit 2 PDF pages 1–50 **A token is one occurrence; a type is a distinct token form under the chosen normalization.** In “cat cat dog”, there are three tokens and two types. Type–token ratio (TTR) measures observed lexical diversity. $$ \operatorname{TTR}=\frac{\text{number of distinct types}}{\text{number of tokens}}. $$ A larger TTR means more distinct words relative to text length in that sample. It is strongly affected by text length and preprocessing, so comparisons should use comparable samples. **Question. Find TTR for “the cat saw the cat” after lowercasing and ignoring punctuation. Keep all words.** #### Given Tokens [the,cat,saw,the,cat]. #### To find Type count, token count and TTR. #### Formula and why it applies $$ TTR=V/N $$ , where $$ V $$ counts unique forms and $$ N $$ counts all occurrences. #### Step 1. Count tokens There are 5 positions → N=5. #### Step 2. List types {the,cat,saw} → V=3. #### Step 3. Divide $$ TTR=3/5=0.6 $$ . **Answer.** TTR=0.6, or 60%. **Question. Find TTR for “Run run RUN” before and after lowercasing.** #### Given Before lowercasing, case distinguishes forms. There are 3 occurrences in either case. #### To find Two TTR values. #### Formula and why it applies $$ TTR=V/N $$ . Recount types after normalization. #### Step 1. Before case folding {Run,run,RUN} has 3 types; N=3. #### Step 2. First ratio $$ 3/3=1 $$ . #### Step 3. After case folding Tokens [run,run,run]; types {run} → V=1. #### Step 4. Second ratio $$ 1/3\approx0.333333 $$ . **Answer.** Before=1; after≈0.333333. **Question. A text has 80 tokens and 20 types. Another has 50 tokens and 25 types. Find and compare their TTRs.** #### Given Text A: N=80,V=20. Text B: N=50,V=25. #### To find Both ratios. #### Formula and why it applies $$ TTR=V/N $$ . #### Step 1. Text A $$ 20/80=0.25 $$ . #### Step 2. Text B $$ 25/50=0.5 $$ . #### Step 3. Interpret B has a larger observed type proportion, but unequal text lengths prevent treating this alone as an unbiased measure of richer language. **Answer.** A=0.25, B=0.5. Tokens = sab occurrences; types = unique names; TTR = unique ÷ total. ### Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR id : tee-03 title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR title\_confirmed : false topic\_title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Vocabulary and vectors—what each coordinate means Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-representation--u2-vocab/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-representation/u2-vocab TEE · Unit 2 PDF pages 51–92 **A vocabulary** is the ordered collection of distinct terms allowed by a representation. Its order defines what every coordinate means. A **vector** is simply an ordered number-list. A **document–term matrix** puts one document in each row and one vocabulary term in each column. With vocabulary [cat,dog,fish], the count vector [2,0,1] means cat appears twice, dog zero times and fish once. With [dog,cat,fish], the same text becomes [0,2,1]. A vector without its vocabulary order is incomplete. | Document | cat | dog | fish | | --- | --- | --- | --- | | cat cat fish | 2 | 0 | 1 | | dog fish | 0 | 1 | 1 | **Nominal categories** have no inherent order, such as cities. **Ordinal categories** do have an order, such as low < medium < high. Assigning arbitrary word IDs does not create meaningful numerical distances between words. Vector ka har box ek fixed word ka hai. Pehle column order fix, phir numbers. ### Vocabulary, label IDs, one-hot encoding and bag of words id : tee-04 title : Vocabulary, label IDs, one-hot encoding and bag of words title\_confirmed : false topic\_title : Vocabulary, label IDs, one-hot encoding and bag of words mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Label and one-hot encoding: three complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-representation--u2-label-onehot/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-representation/u2-label-onehot TEE · Unit 2 PDF pages 51–92 **Label encoding** assigns each category or word a unique integer ID. **One-hot encoding** assigns a length- $$ V $$ vector with one 1 at the word's vocabulary position and zeros elsewhere. $$ \operatorname{onehot}(w)_j=\begin{cases}1,&j=\operatorname{index}(w)\\0,&\text{otherwise.}\end{cases} $$ The index is an ID, not a meaning score. One-hot represents one token/category; a multiword document generally needs multiple token vectors or an aggregation scheme. **Question. Encode “dog cat dog” using vocabulary [cat,dog,fish], IDs starting at 0. Give label IDs and a one-hot vector for every token.** #### Given cat→0, dog→1, fish→2. Vector columns have the same order. #### To find The ID sequence and three one-hot rows. #### Formula and why it applies Replace each word by its ID for labels; set its vocabulary position to 1 for one-hot. #### Step 1. Label IDs dog→1, cat→0, dog→1 → [1,0,1]. #### Step 2. First dog [0,1,0]. #### Step 3. Cat [1,0,0]. #### Step 4. Second dog [0,1,0]. Repeated tokens keep the same representation. **Answer.** IDs [1,0,1]; one-hot rows [0,1,0], [1,0,0], [0,1,0]. **Question. Encode the city sequence Mumbai, Delhi, Mumbai, Pune using vocabulary [Delhi,Mumbai,Pune] and label IDs starting at 1.** #### Given Delhi→1, Mumbai→2, Pune→3. #### To find IDs and one-hot rows. #### Formula and why it applies One-hot uses three coordinates regardless of whether label IDs begin at 0 or 1. #### Step 1. IDs [2,1,2,3]. #### Step 2. Mumbai rows Each Mumbai → [0,1,0]. #### Step 3. Delhi and Pune Delhi→[1,0,0]; Pune→[0,0,1]. **Answer.** Rows [0,1,0], [1,0,0], [0,1,0], [0,0,1]. **Question. One-hot encode categories [red,green,blue] for a linear regression with an intercept. Then give dummy coding after dropping blue.** #### Given Full column order [red,green,blue]. Blue is the baseline. #### To find Full and reduced codes. #### Formula and why it applies Full one-hot rows sum to 1. With an intercept this creates a linear dependency; dropping one category column gives k−1 independent dummy indicators. #### Step 1. Full encoding | Category | red | green | blue | | --- | --- | --- | --- | | red | 1 | 0 | 0 | | green | 0 | 1 | 0 | | blue | 0 | 0 | 1 | #### Step 2. Drop blue column | Category | red | green | | --- | --- | --- | | red | 1 | 0 | | green | 0 | 1 | | blue | 0 | 0 | #### Step 3. Interpret zero row [0,0] now identifies the baseline blue, not missing data. **Answer.** Full width 3; reduced dummy width 2. Dropping a column is model-dependent, not a requirement for every neural embedding input. **Limitations:** one-hot is large and sparse and does not express semantic similarity. Unseen words need a defined unknown-token or vocabulary-update policy. Label IDs are compact but should not be treated as an ordered scalar feature for nominal words. Label = ID number. One-hot = us ID ke box mein 1, baaki 0. ### Vocabulary, label IDs, one-hot encoding and bag of words id : tee-04 title : Vocabulary, label IDs, one-hot encoding and bag of words title\_confirmed : false topic\_title : Vocabulary, label IDs, one-hot encoding and bag of words mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Bag of words and bag of n-grams Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-representation--u2-bow/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-representation/u2-bow TEE · Unit 2 PDF pages 51–92 **Bag of words (BoW) represents a document by the counts of vocabulary terms, discarding their positions.** A binary BoW variant stores only presence/absence; a count BoW stores frequencies. State which is used. $$ x_{d,j}=C(v_j,d). $$ $$ x_{d,j} $$ is document $$ d $$ 's coordinate for vocabulary term $$ v_j $$ . $$ C(v_j,d) $$ is its occurrence count. A **bag of n-grams** uses short contiguous token sequences as features. Bigrams can distinguish “not good” from “good”, but increasing n enlarges the vocabulary and creates more rare features. **Question. Build count and binary BoW for D1=“cat cat dog” and D2=“dog fish”. Use vocabulary [cat,dog,fish] and no stopword removal.** #### Given Two tokenized documents, fixed vocabulary order. #### To find Count vectors and binary vectors. #### Formula and why it applies Count BoW uses $$ C(v_j,d) $$ . Binary BoW is 1 when that count is positive and 0 otherwise. #### Step 1. D1 counts cat=2, dog=1, fish=0 → [2,1,0]. #### Step 2. D2 counts cat=0, dog=1, fish=1 → [0,1,1]. #### Step 3. Convert to presence D1 [1,1,0]; D2 [0,1,1]. **Answer.** Count rows [2,1,0], [0,1,1]; binary rows [1,1,0], [0,1,1]. **Question. After lowercasing and removing {is,a,means}, obtain BoW vectors for D1=“Continuous learning is learning lifetime”, D2=“Continuous learning means lifetime learning”, D3=“Learning is a good practice”. Use [continuous,good,learning,lifetime,practice].** #### Given Three documents and exact stopword set from the supplied slides. #### To find The document–term count matrix. #### Formula and why it applies Remove only the listed stopwords, then count every remaining token in the fixed column order. #### Step 1. Clean D1 [continuous,learning,learning,lifetime] → [1,0,2,1,0]. #### Step 2. Clean D2 [continuous,learning,lifetime,learning] → [1,0,2,1,0]. #### Step 3. Clean D3 [learning,good,practice] → [0,1,1,0,1]. #### Step 4. Compare D1 and D2 They have identical BoW vectors despite different original wording/order. **Answer.** Rows [1,0,2,1,0], [1,0,2,1,0], [0,1,1,0,1]. **Question. Build a bag of bigrams for “not good but not bad” with vocabulary [not good,good but,but not,not bad]. Do not add sentence boundary tokens.** #### Given Five tokens; use adjacent pairs, stride 1. #### To find The bigram count vector. #### Formula and why it applies A sequence of $$ L $$ tokens has $$ L-1 $$ adjacent bigrams when no boundaries are added. #### Step 1. Enumerate pairs (not,good), (good,but), (but,not), (not,bad). #### Step 2. Count each vocabulary bigram Each appears once. #### Step 3. Write vector [1,1,1,1]. **Answer.** Four bigram features, each with count 1. BoW: order bhool, word-count rakh. Bag of bigrams: neighbouring word-pair count. ### Vocabulary, label IDs, one-hot encoding and bag of words id : tee-04 title : Vocabulary, label IDs, one-hot encoding and bag of words title\_confirmed : false topic\_title : Vocabulary, label IDs, one-hot encoding and bag of words mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · TF–IDF from the exact slide convention Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-representation--u2-tfidf/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-representation/u2-tfidf TEE · Unit 2 PDF pages 51–92 **TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus.** The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency: $$ \operatorname{TF}(t,d)=\frac{C(t,d)}{N_d} $$ $$ \operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)} $$ $$ \operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t). $$ $$ N_d $$ is the document's token count after the specified preprocessing; $$ D $$ is number of documents; $$ \operatorname{df}(t) $$ is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency. **Why the log?** It compresses the ratio's scale. If a term occurs in all documents, $$ D/df=1 $$ , so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default. **Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.** #### Given D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10. #### To find The TF–IDF weight. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ , then multiply. #### Step 1. TF $$ 3/12=0.25 $$ . #### Step 2. Document ratio $$ 10/2=5 $$ . #### Step 3. IDF $$ \log_{10}5\approx0.698970 $$ . #### Step 4. Weight $$ 0.25\times0.698970\approx0.174743 $$ . **Answer.** TF=0.25, IDF≈0.698970, TF–IDF≈0.174743. **Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.** #### Given C=4, Nd=20, D=8, df=8. #### To find Its weight under unsmoothed base-10 IDF. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ . #### Step 1. TF $$ 4/20=0.2 $$ . #### Step 2. IDF $$ \log_{10}(8/8)=\log_{10}1=0 $$ . #### Step 3. Multiply $$ 0.2\times0=0 $$ . **Answer.** Weight 0: this term does not distinguish documents under this convention. **Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.** #### Given No further preprocessing. Each term occurs in exactly two of the three documents. #### To find All three vectors. #### Formula and why it applies First count df by documents, then compute each document's TF, then multiply by its term's IDF. #### Step 1. Common IDF $$ D=3 $$ , $$ df(cat)=df(dog)=df(fish)=2 $$ . Each IDF $$ =\log_{10}(3/2)\approx0.176091 $$ . #### Step 2. D1 term frequencies Length 3 → $$ [2/3,1/3,0] $$ . #### Step 3. D1 weights $$ [(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0] $$ . #### Step 4. D2 weights Length 2; TF $$ [0,1/2,1/2] $$ . Vector $$ \approx[0,0.088046,0.088046] $$ . #### Step 5. D3 weights Length 3; TF $$ [1/3,0,2/3] $$ . Vector $$ \approx[0.058697,0,0.117394] $$ . **Answer.** D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394]. TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply. ### TEE 05 — TF-IDF and cosine: from the corpus to the final similarity id : tee-05 title : TEE 05 — TF-IDF and cosine: from the corpus to the final similarity title\_confirmed : false topic\_title : TF-IDF and cosine: from the corpus to the final similarity mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Combined slide numerical: TF–IDF followed by cosine Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-representation--u2-slide-combined/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-representation/u2-slide-combined TEE · Unit 2 PDF pages 51–92 **Question. For D1=“Continuous learning is learning lifetime”, D2=“Continuous learning means lifetime learning”, D3=“Learning is a good practice”, find the TF–IDF cosine similarity of D1 and D2. Lowercase and remove {is,a,means}. Use normalized TF and log10 IDF.** #### Given Vocabulary [continuous,good,learning,lifetime,practice]. D1 and D2 each have 4 retained tokens; D3 has 3. #### To find TF–IDF vectors and cosine(D1,D2). #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ ; $$ \cos(x,y)=(x\cdot y)/(\|x\|\|y\|) $$ . First build the vectors; cosine cannot be applied directly to words. #### Step 1. Count documents containing each term | Term | df | IDF | | --- | --- | --- | | continuous | 2 | log10(3/2)≈0.176091 | | good | 1 | log10(3)≈0.477121 | | learning | 3 | log10(1)=0 | | lifetime | 2 | ≈0.176091 | | practice | 1 | ≈0.477121 | #### Step 2. Term frequencies D1=D2=[1/4,0,2/4,1/4,0]. D3=[0,1/3,1/3,0,1/3]. #### Step 3. Multiply TF by IDF Let $$ a=\tfrac14\log_{10}(3/2)\approx0.0440228 $$ . Then D1=D2=[a,0,0,a,0]. D3≈[0,0.159040,0,0,0.159040]. #### Step 4. Dot product $$ x\cdot y=a^2+0+0+a^2+0=2a^2 $$ . #### Step 5. Vector lengths $$ \|x\|=\|y\|=\sqrt{2a^2} $$ . Their product is $$ 2a^2 $$ . #### Step 6. Divide $$ \cos(x,y)=2a^2/(2a^2)=1 $$ . **Answer.** Cosine(D1,D2)=1. Their BoW Euclidean distance is also 0 because their counts are identical. The learning coordinate becomes zero because learning occurs in every document. D1–D3 cosine is 0 under this exact TF–IDF convention: their only originally shared word was learning, whose IDF is zero. This is a consequence of the model, not proof that the sentences have unrelated meanings. Combined sum: pehle cleaned words → counts → TF-IDF → similarity. Beech ka stage skip mat kar. ### TEE 05 — TF-IDF and cosine: from the corpus to the final similarity id : tee-05 title : TEE 05 — TF-IDF and cosine: from the corpus to the final similarity title\_confirmed : false topic\_title : TF-IDF and cosine: from the corpus to the final similarity mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Cosine: products, lengths and division Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-similarity--u2-cosine/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-similarity/u2-cosine Unit 2 · PDF pages 93–103 **Cosine similarity compares the direction of two nonzero vectors.** Multiply matching coordinates and add to obtain the dot product. To obtain each vector's norm, square its coordinates, add them and take a square root. $$ \cos(x,y)=\frac{\sum_i x_i y_i}{\sqrt{\sum_i x_i^2}\sqrt{\sum_i y_i^2}}. $$ The vectors must use the same coordinate/vocabulary order. For nonnegative BoW or TF–IDF vectors, the result lies from 0 to 1; for general real-valued vectors, it can range from −1 to 1. A zero vector makes this formula undefined. **Question. Find cosine similarity for x=[1,1] and y=[1,0].** #### Given Nonzero vectors, matching coordinate order. #### To find Find cosine similarity for x=[1,1] and y=[1,0]. #### Formula and why it applies Use $$ \cos(x,y)=(x\cdot y)/(\|x\|\|y\|) $$ because direction similarity is required. #### Step 1. Multiply matching coordinates 1 × 1 = 1; 1 × 0 = 0. #### Step 2. Add products 1 + 0 = 1. #### Step 3. Length of x $$ \sqrt{(1)^2+(1)^2}=\sqrt{2}. $$ #### Step 4. Length of y $$ \sqrt{(1)^2+(0)^2}=\sqrt{1}. $$ #### Step 5. Divide $$ \frac{1}{\sqrt{2}\sqrt{1}}\approx 0.707107. $$ **Answer.** Cosine≈0.707107. **Question. Find cosine similarity for x=[1,2] and y=[2,4].** #### Given Nonzero vectors, matching coordinate order. #### To find Find cosine similarity for x=[1,2] and y=[2,4]. #### Formula and why it applies Use $$ \cos(x,y)=(x\cdot y)/(\|x\|\|y\|) $$ because direction similarity is required. #### Step 1. Multiply matching coordinates 1 × 2 = 2; 2 × 4 = 8. #### Step 2. Add products 2 + 8 = 10. #### Step 3. Length of x $$ \sqrt{(1)^2+(2)^2}=\sqrt{5}. $$ #### Step 4. Length of y $$ \sqrt{(2)^2+(4)^2}=\sqrt{20}. $$ #### Step 5. Divide $$ \frac{10}{\sqrt{5}\sqrt{20}}\approx 1.000000. $$ **Answer.** Cosine≈1.000000. **Question. Find cosine similarity for x=[1,2,0] and y=[0,1,2].** #### Given Nonzero vectors, matching coordinate order. #### To find Find cosine similarity for x=[1,2,0] and y=[0,1,2]. #### Formula and why it applies Use $$ \cos(x,y)=(x\cdot y)/(\|x\|\|y\|) $$ because direction similarity is required. #### Step 1. Multiply matching coordinates 1 × 0 = 0; 2 × 1 = 2; 0 × 2 = 0. #### Step 2. Add products 0 + 2 + 0 = 2. #### Step 3. Length of x $$ \sqrt{(1)^2+(2)^2+(0)^2}=\sqrt{5}. $$ #### Step 4. Length of y $$ \sqrt{(0)^2+(1)^2+(2)^2}=\sqrt{5}. $$ #### Step 5. Divide $$ \frac{2}{\sqrt{5}\sqrt{5}}\approx 0.400000. $$ **Answer.** Cosine≈0.400000. Proportional vectors [1,2] and [2,4] have cosine 1 but nonzero Euclidean distance. Direction similarity does not require identical magnitudes. Cosine: matching products add upar; dono lengths multiply neeche. ### TEE 05 — TF-IDF and cosine: from the corpus to the final similarity id : tee-05 title : TEE 05 — TF-IDF and cosine: from the corpus to the final similarity title\_confirmed : false topic\_title : TF-IDF and cosine: from the corpus to the final similarity mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Jaccard: intersection, union and distance Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-similarity--u2-jaccard/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-similarity/u2-jaccard Unit 2 · PDF pages 93–103 **Jaccard similarity is the fraction of shared distinct elements among all distinct elements in either set.** $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|}. $$ Intersection means common elements; union means all distinct elements. Repetitions do not count again. Jaccard distance is $$ 1-J $$ , not the same quantity as similarity. Two empty sets require a declared convention because the fraction becomes 0/0. **Question. Find Jaccard similarity and distance for A={cat, dog} and B={dog, fish}.** #### Given Sets contain unique elements. #### To find Find Jaccard similarity and distance for A={cat, dog} and B={dog, fish}. #### Formula and why it applies $$ J=|A\cap B|/|A\cup B| $$ ; $$ d=1-J $$ . #### Step 1. Find intersection {dog}; count 1. #### Step 2. Find union {cat, dog, fish}; count 3. #### Step 3. Divide J=1/3≈0.333333. #### Step 4. Subtract from one Distance=1−0.333333≈0.666667. **Answer.** Similarity≈0.333333; distance≈0.666667. **Question. Find Jaccard similarity and distance for A={red, blue} and B={red, blue}.** #### Given Sets contain unique elements. #### To find Find Jaccard similarity and distance for A={red, blue} and B={red, blue}. #### Formula and why it applies $$ J=|A\cap B|/|A\cup B| $$ ; $$ d=1-J $$ . #### Step 1. Find intersection {red, blue}; count 2. #### Step 2. Find union {red, blue}; count 2. #### Step 3. Divide J=2/2≈1.000000. #### Step 4. Subtract from one Distance=1−1.000000≈0.000000. **Answer.** Similarity≈1.000000; distance≈0.000000. **Question. Find Jaccard similarity and distance for A={apple} and B={orange, pear}.** #### Given Sets contain unique elements. #### To find Find Jaccard similarity and distance for A={apple} and B={orange, pear}. #### Formula and why it applies $$ J=|A\cap B|/|A\cup B| $$ ; $$ d=1-J $$ . #### Step 1. Find intersection {}; count 0. #### Step 2. Find union {apple, orange, pear}; count 3. #### Step 3. Divide J=0/3≈0.000000. #### Step 4. Subtract from one Distance=1−0.000000≈1.000000. **Answer.** Similarity≈0.000000; distance≈1.000000. Jaccard = common unique ÷ total unique. Distance = 1 minus similarity. ### TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski id : tee-06 title : TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski title\_confirmed : false topic\_title : Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Manhattan, Euclidean and Minkowski: three complete comparisons Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-similarity--u2-distances/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-similarity/u2-distances Unit 2 · PDF pages 93–103 **Distance measures separation; smaller means closer.** Absolute values prevent positive and negative differences from cancelling. $$ d_1=\sum_i|x_i-y_i|\quad\text{Manhattan} $$ $$ d_2=\sqrt{\sum_i(x_i-y_i)^2}\quad\text{Euclidean} $$ $$ d_p=\left(\sum_i|x_i-y_i|^p\right)^{1/p},\quad p\ge1\quad\text{Minkowski}. $$ Manhattan is p=1; Euclidean is p=2. For p=3, cube absolute differences, add, then take a cube root. **Question. Find Manhattan, Euclidean and Minkowski p=3 distances for x=[1,2], y=[4,6].** #### Given Vectors share coordinate order; negative inputs are allowed. #### To find Find Manhattan, Euclidean and Minkowski p=3 distances for x=[1,2], y=[4,6]. #### Formula and why it applies Use $$ d_1=\sum|\Delta_i| $$ , $$ d_2=\sqrt{\sum|\Delta_i|^2} $$ , $$ d_3=\sqrt[3]{\sum|\Delta_i|^3} $$ . #### Step 1. Absolute differences |1 − (4)| = 3; |2 − (6)| = 4. #### Step 2. Manhattan 3 + 4 = 7. #### Step 3. Square and add 3²=9; 4²=16. Sum=25. #### Step 4. Euclidean root $$ \sqrt{25}\approx 5.000000. $$ #### Step 5. Cube and add 3³=27; 4³=64. Sum=91. #### Step 6. Minkowski root $$ \sqrt[3]{91}\approx 4.497941. $$ **Answer.** Manhattan=7; Euclidean≈5.000000; p=3≈4.497941. **Question. Find Manhattan, Euclidean and Minkowski p=3 distances for x=[0,1,2], y=[2,1,0].** #### Given Vectors share coordinate order; negative inputs are allowed. #### To find Find Manhattan, Euclidean and Minkowski p=3 distances for x=[0,1,2], y=[2,1,0]. #### Formula and why it applies Use $$ d_1=\sum|\Delta_i| $$ , $$ d_2=\sqrt{\sum|\Delta_i|^2} $$ , $$ d_3=\sqrt[3]{\sum|\Delta_i|^3} $$ . #### Step 1. Absolute differences |0 − (2)| = 2; |1 − (1)| = 0; |2 − (0)| = 2. #### Step 2. Manhattan 2 + 0 + 2 = 4. #### Step 3. Square and add 2²=4; 0²=0; 2²=4. Sum=8. #### Step 4. Euclidean root $$ \sqrt{8}\approx 2.828427. $$ #### Step 5. Cube and add 2³=8; 0³=0; 2³=8. Sum=16. #### Step 6. Minkowski root $$ \sqrt[3]{16}\approx 2.519842. $$ **Answer.** Manhattan=4; Euclidean≈2.828427; p=3≈2.519842. **Question. Find Manhattan, Euclidean and Minkowski p=3 distances for x=[-1,3], y=[2,-1].** #### Given Vectors share coordinate order; negative inputs are allowed. #### To find Find Manhattan, Euclidean and Minkowski p=3 distances for x=[-1,3], y=[2,-1]. #### Formula and why it applies Use $$ d_1=\sum|\Delta_i| $$ , $$ d_2=\sqrt{\sum|\Delta_i|^2} $$ , $$ d_3=\sqrt[3]{\sum|\Delta_i|^3} $$ . #### Step 1. Absolute differences |-1 − (2)| = 3; |3 − (-1)| = 4. #### Step 2. Manhattan 3 + 4 = 7. #### Step 3. Square and add 3²=9; 4²=16. Sum=25. #### Step 4. Euclidean root $$ \sqrt{25}\approx 5.000000. $$ #### Step 5. Cube and add 3³=27; 4³=64. Sum=91. #### Step 6. Minkowski root $$ \sqrt[3]{91}\approx 4.497941. $$ **Answer.** Manhattan=7; Euclidean≈5.000000; p=3≈4.497941. Manhattan: absolute-add. Euclidean: square-add-root. Minkowski: p-power-add-p-root. ### TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski id : tee-06 title : TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski title\_confirmed : false topic\_title : Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Choose the representation before the metric Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u2-similarity--u2-choice/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u2-similarity/u2-choice Unit 2 · PDF pages 93–103 | Question | Measure | Closer means | | --- | --- | --- | | Compare vector direction | Cosine similarity | Higher | | Compare term sets | Jaccard similarity | Higher | | Straight-line separation | Euclidean distance | Lower | | Sum coordinate changes | Manhattan distance | Lower | | General p specified | Minkowski distance | Lower | BoW and TF–IDF may give different similarities. First show preprocessing, vocabulary and vectors; then use the metric. Arbitrary label IDs do not define meaningful word distances. Never compare vectors using different term orders. Keep intermediate precision until the final result. Representation + metric dono question ke hisaab se. Formula pe seedha jump nahi. ### TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski id : tee-06 title : TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski title\_confirmed : false topic\_title : Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Corpus structure and annotations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-corpora--u3-corpus/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-corpora/u3-corpus Unit 3 · PDF pages 1–61; pronouncing dictionary from current syllabus **A corpus is a collection of language data assembled for study or computational use.** Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. **Annotation** adds linguistic information such as POS tags, entities, parse structures or sense labels. | Structure | Meaning | Use | | --- | --- | --- | | Plain | Collection without category labels | Word counts, language modelling | | Categorized | Documents grouped by genre/topic | Classification | | Overlapping categories | One document has several categories | Multi-topic news | | Temporal | Documents indexed by time | Language change | | Annotated | Linguistic labels attached | Tagging, parsing, WSD | | Parallel | Aligned texts in different languages | Translation | A **tagset** is the allowed label inventory. A **treebank** is a corpus annotated with syntax trees. **Metadata** records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions. **Raw example:** “Riya works in Pune.” **POS annotation:** Riya/PROPN works/VERB in/ADP Pune/PROPN. **NER annotation:** Riya/PERSON, Pune/LOCATION. Each annotation answers a different task. A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use. Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Gutenberg, web/chat, Brown, Reuters and Inaugural Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-corpora--u3-types/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-corpora/u3-types Unit 3 · PDF pages 1–61; pronouncing dictionary from current syllabus | Resource | Contents | Typical use | | --- | --- | --- | | Gutenberg selection | Literary/book texts | Long-form language and vocabulary | | Web text | Informal domain-specific text | Study usage beyond edited books | | Chat corpus | Conversational posts | Dialogue, informal forms | | Brown | English texts across genres | Genre comparison | | Reuters | News with topic labels | Text classification; overlapping topics | | Inaugural | Speeches indexed by time | Vocabulary change across years | | Wordlist | Words/names/stopwords | Lexical filtering | | Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme | **Gutenberg versus web/chat:** book language does not represent every spelling or conversational pattern in messages. **Brown versus Reuters:** Brown emphasizes genres; Reuters news can have overlapping topic labels. **Inaugural addresses:** dates provide a time dimension. Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property. Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Raw text, tokens and sentences are different views Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-corpora--u3-access/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-corpora/u3-access Unit 3 · PDF pages 1–61; pronouncing dictionary from current syllabus Corpus readers expose several views. **raw** returns a character string, **words** returns token occurrences, and **sents** returns a list of sentences where each sentence contains tokens. | Conceptual method | Returned view | Its length counts | | --- | --- | --- | | fileids() | Document IDs | Files | | raw(fileid) | Character string | Characters | | words(fileid) | Token list | Token occurrences | | sents(fileid) | Sentence lists | Sentences | | categories() | Genre/topic labels | Categories | | tagged\_words(fileid) | Word–tag pairs | Annotated tokens | **Question. For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types.** #### Given Punctuation is included in token count but excluded from word types. #### To find For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types. #### Formula and why it applies Count every token occurrence, then distinct alphabetic forms. Different counting rules give different totals. #### Step 1. Token count 4+4=8. #### Step 2. Sentence count Two outer lists → 2. #### Step 3. Types {the,cat,sleeps,dog,runs} → 5. **Answer.** 8 tokens, 2 sentences, 5 word types. For “Cats run. Dogs sleep.”, a reader may return six tokens including two periods, but only four alphabetic word tokens. State the convention before calculating TTR or probabilities. raw=characters, words=tokens, sents=sentences. Teen alag counts. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Wordlists, pronunciation and semantic resources Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-corpora--u3-resources/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-corpora/u3-resources Unit 3 · PDF pages 1–61; pronouncing dictionary from current syllabus **A wordlist** stores lexical entries without necessarily giving usage context. A stopword list gives task-specific filtering terms. Names lists can support lexical analysis but cannot reliably infer a real person's identity or gender. **A pronouncing dictionary** maps a written word to one or more phonetic symbol sequences. CMU-style dictionaries use ARPAbet; vowel stress suffixes distinguish unstressed, primary-stressed and secondary-stressed vowels. A spelling can have more than one pronunciation, as “read” does. | Question | Resource | Reason | | --- | --- | --- | | Which words rhyme with night? | Pronouncing dictionary | Compare sounds, not just spelling | | Which terms should be filtered? | Specified stopword list | Apply task's filtering convention | | What broader concept includes car? | WordNet | Follow hypernym relation | | How frequently does car occur? | Text corpus | Count actual usage occurrences | **WordNet** groups word senses in synsets, supplies glosses and connects meanings by semantic relations. Its full treatment is in Unit 5. A corpus supplies usage evidence; a lexical resource supplies organized knowledge. A system can combine both. Sound→pronouncing dictionary. Meaning→WordNet. Occurrence→corpus. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Acquiring data and preserving labels during augmentation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-corpora--u3-acquisition/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-corpora/u3-acquisition Unit 3 · PDF pages 1–61; pronouncing dictionary from current syllabus **Data acquisition** obtains language examples relevant to the task from existing data, public resources or appropriate new collection. Product instrumentation can gather useful feedback. Supervised learning additionally needs an annotation process when labels are absent. **Data augmentation** creates training variants intended to preserve the correct label. It is not simply changing words arbitrarily. | Method | Procedure | Caution | | --- | --- | --- | | Synonym replacement | Use suitable synonyms for selected words | Choose the right sense | | Back translation | Translate to another language and back | May change meanings/entities | | TF–IDF-informed replacement | Protect informative words while varying others | Do not remove label-bearing clues | | Bigram flipping | Swap adjacent words | Can break grammar or meaning | | Entity replacement | Replace by same-type entity | Must preserve the task label | | Noise injection | Simulate typos/keyboard errors | Match expected noise | | Active learning | Select informative cases for human labels | This acquires labels; it is not synthetic text | ### Three label decisions “The film is good”→“The film is excellent” may preserve a positive sentiment label. “The film is not good”→“The film is good” reverses the label and is not valid label-preserving augmentation. “Flights to Pune”→“Flights to Mumbai” may preserve a travel-topic label but changes the destination label. Split train/test data before creating related augmented variants. Otherwise near-duplicates can leak between sets and produce misleading evaluation. Keep a representative evaluation set of genuine examples. Sentence badal sakta hai; required correct label nahi. Label badla toh augmentation invalid. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · How corpus evidence powers later units Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-corpora--u3-links/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-corpora/u3-links Unit 3 · PDF pages 1–61; pronouncing dictionary from current syllabus Word-sequence counts support n-gram language models and smoothing. POS-annotated sentences support supervised taggers. Sense-labelled contexts support Naive Bayes WSD. Word-context counts support distributional semantics and GloVe. Parallel aligned sentences support translation. For a question asking the importance of corpora, give three points: empirical language evidence, examples for training/evaluation, and reproducible comparison. For an annotated-corpus question, define the added labels and give a concrete example. For resource comparisons, distinguish running text, lexical lists, pronunciation entries and semantic relations. Same text se task ke hisaab se counts, tags, senses ya context patterns nikaalte hain. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Characters, classes, anchors and repetition Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-regex--regex-language/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-regex/regex-language TEE · current Unit 3 syllabus and supplied Regular Expressions handout; outside confirmed M2 page cutoff **A regular expression (regex) is a pattern language for specifying classes of text strings.** A regex engine tests or finds strings matching a pattern. It can extract numbers, locate words, clean text and implement rule-based NLP features. A match is a string satisfying the pattern; it is not an interpretation of the string's meaning. Start with a literal: `cat` matches those three consecutive letters. Without boundary constraints it also matches the cat part inside “catalog” or “wildcat”. Add restrictions only when the question asks for them. | Pattern | Meaning | Example | | --- | --- | --- | | `[aeiou]` | Exactly one listed lowercase vowel | a, e; not b | | `[A-Z]` | One uppercase ASCII letter | A through Z | | `[^0-9]` | One character not in the digit range | a or a space; not 5 | | `\d` | One digit under the regex engine's character rules | Use [0-9] if only ASCII digits are intended | | `\w` | Word character, usually letters/digits/underscore | Unicode details depend on engine | | `\s` | Whitespace character | Space, tab, newline | | `\D, \W, \S` | Complement classes | Non-digit, non-word, non-whitespace | | `.` | One arbitrary character except newline by default | DOTALL flag can include newlines | | `\.` | Literal full stop | Escaping removes the wildcard meaning | | `^` | Start of string/line depending on mode | ^The | | `$` | End anchor under the engine's rules | Spain$ | | `\b` | Boundary between word and non-word positions | Whole-word matching | | `*` | Zero or more repetitions of previous item | ab\* matches a, ab, abb | | `+` | One or more | ab+ requires at least one b | | `?` | Zero or one | colou?r allows color and colour | | `{m}, {m,n}, {m,}` | Exact, bounded or minimum repetitions | [0-9]{3} | | `(...)` | Group a sequence | (ab)+ repeats the pair | | `|` | Choose an alternative | cat|dog | **Critical distinction:** a character class chooses one character, whereas alternatives can choose whole strings. `[cat]` matches one c, a or t; `(cat|dog)` matches either complete word. A quantifier acts on the immediately preceding atom/group. `abc*` means ab followed by any number of c; `(abc)*` repeats the entire abc group. Regex left-to-right padho: kaunsa character, kitni baar, aur start/end/word boundary kya hai? ### Regular expressions: build and explain exact text patterns id : tee-08 title : Regular expressions: build and explain exact text patterns title\_confirmed : false topic\_title : Regular expressions: build and explain exact text patterns mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Search, match, fullmatch, findall, substitution and flags Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-regex--regex-functions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-regex/regex-functions TEE · current Unit 3 syllabus and supplied Regular Expressions handout; outside confirmed M2 page cutoff | Python operation | What it does | Typical use | | --- | --- | --- | | re.search(pattern,text) | Finds the first match anywhere | Locate a clue inside a sentence | | re.match(pattern,text) | Tests only from the start | Check a prefix | | re.fullmatch(pattern,text) | Requires the complete string to match | Validate an exact format | | re.findall(pattern,text) | Returns non-overlapping matches | Extract all numbers | | re.finditer(pattern,text) | Returns match objects with spans | Locate match positions | | re.sub(pattern,replacement,text) | Replaces matches | Normalize repeated whitespace | | re.split(pattern,text) | Splits at matches | Custom token boundaries | Python raw strings such as `r"\bcat\b"` keep the backslashes for the regex engine. In an ordinary Python string, `\b` is a backspace escape, so using a raw string prevents a common mistake. Capturing groups can change what `findall` returns; use a noncapturing group `(?:...)` when grouping is needed but capturing is not. ### Flags and engine differences **IGNORECASE / i** ignores case. **MULTILINE / m** lets ^ and $ refer to line boundaries. **DOTALL / s** lets dot match newline. JavaScript's **g** flag requests repeated/global matching; Python has no separate global flag for `findall`, which already finds successive non-overlapping matches. Do not mix JavaScript slash delimiters with Python's string syntax. ### Greedy versus lazy quantifiers A greedy quantifier tries to consume as much as possible while permitting the whole match; a lazy quantifier tries as little as possible. On `onetwo`, `<.*>` can match the entire string from the first < to last >. The lazy `<.*?>` returns shorter tag-shaped matches during repeated matching. Regex is useful for restricted patterns; robust arbitrary HTML processing should use an HTML parser. search=kahin bhi; match=start se; fullmatch=poora text; findall=sab matches. Greedy zyada, lazy kam. ### Regular expressions: build and explain exact text patterns id : tee-08 title : Regular expressions: build and explain exact text patterns title\_confirmed : false topic\_title : Regular expressions: build and explain exact text patterns mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Six full regex questions and solutions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-regex--regex-worked/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-regex/regex-worked TEE · current Unit 3 syllabus and supplied Regular Expressions handout; outside confirmed M2 page cutoff **Question. From “cat catalog wildcat Cat”, extract only the lowercase whole word cat using Python regex.** #### Given Case-sensitive matching; whole words only. #### To find A pattern and its matches. #### Rule and why it applies Use `\b` boundaries around the literal word. This prevents substring matches inside longer word tokens. #### Step 1. Write the literal `cat` alone would also match catalog and wildcat. #### Step 2. Add both word boundaries `r"\bcat\b"`. #### Step 3. Check candidates cat matches. catalog fails at the ending boundary. wildcat fails at the starting boundary. Cat fails because matching is case sensitive. #### Step 4. Extract `re.findall(r"\bcat\b", text)` returns `["cat"]`. **Answer.** Pattern `\bcat\b`; one match: cat. With IGNORECASE, Cat would also match. **Question. Write a regex that validates exactly ten ASCII digits, then test 9876543210, 98765 and A9876543210.** #### Given No country-prefix restriction; digits only; entire input must match. #### To find A validation pattern and each result. #### Rule and why it applies `[0-9]` permits one ASCII digit; `{10}` repeats it exactly ten times; fullmatch prevents extra characters. #### Step 1. Choose the character class `[0-9]`. #### Step 2. Specify exact length `[0-9]{10}`. #### Step 3. Require the whole input `re.fullmatch(r"[0-9]{10}", text)`. In a simple single-line exam pattern, `^[0-9]{10}$` expresses the intended format. #### Step 4. Test all examples 9876543210: 10 digits → valid. 98765: only 5 → invalid. A9876543210: contains an extra letter → invalid. **Answer.** Use fullmatch with `[0-9]{10}`. Only 9876543210 passes. This checks format, not whether a phone number exists. **Question. Extract date-shaped strings in DD/MM/YYYY format from “On 05/10/2026 and 7/10/2026 we met.” Require two digits for day and month and four for year.** #### Given Format recognition only; no calendar-validity test. #### To find Pattern and extracted strings. #### Rule and why it applies Use fixed repetition counts separated by literal slashes, and boundaries around the full pattern. #### Step 1. Day part `[0-9]{2}`. #### Step 2. Month and year `[0-9]{2}/[0-9]{2}/[0-9]{4}`. #### Step 3. Add boundaries `r"\b[0-9]{2}/[0-9]{2}/[0-9]{4}\b"`. #### Step 4. Evaluate the two candidates 05/10/2026 has 2/2/4 digits and matches. 7/10/2026 has a one-digit day and fails. **Answer.** Only 05/10/2026 is extracted. A date such as 99/99/2026 has the right shape but needs a calendar parser to reject it semantically. **Question. Use a regex to check a simple email shape: one or more username characters from letters, digits, dot, underscore, percent, plus or hyphen; @; a domain of letters, digits, dots or hyphens; a literal dot; a letter-only suffix of length at least two.** #### Given This is the simplified handout format, not full standards-compliant validation. #### To find The pattern and an explanation of every part. #### Rule and why it applies Build the requested components and use fullmatch so no extra prefix or suffix is accepted. #### Step 1. Username `[A-Za-z0-9._%+-]+`: one or more allowed username characters. The hyphen is last, so it is literal. #### Step 2. Separator and domain `@[A-Za-z0-9.-]+`: literal @ followed by domain characters. #### Step 3. Suffix `\.[A-Za-z]{2,}`: escaped dot, then at least two letters. #### Step 4. Combine and test `re.fullmatch(r"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}", text)`. `riya+nlp@example.com` matches; `riya.example.com` lacks @ and fails. **Answer.** The combined pattern meets the stated classroom shape. It neither proves address existence nor enforces every real email/domain constraint. **Question. For text “The rain in Spain”, write patterns that test starts with The, ends with Spain, and satisfies both.** #### Given Single-line, case-sensitive text. #### To find Three patterns and outcomes. #### Rule and why it applies Use ^ for beginning, $ for ending, and .\* for the intervening sequence in this simple one-line task. #### Step 1. Starts with `^The` succeeds. #### Step 2. Ends with `Spain$` succeeds. #### Step 3. Both `^The.*Spain$` succeeds because the text begins with The and ends with Spain. #### Step 4. Why dot-star? `.` consumes one intervening character; `*` allows zero or more, including spaces here. **Answer.** All three checks succeed. Without anchors a substring match alone would not enforce the required position. **Question. Construct a case-sensitive pattern for full filenames ending in .jpg or .png and test photo.jpg, chart.png and photo.jpg.exe.** #### Given Filename must have at least one character before the dot. Treat the filename as a single line. #### To find Pattern and three results. #### Rule and why it applies Use a noncapturing alternative for the two complete suffixes and fullmatch to disallow trailing text. #### Step 1. Filename stem `.+` requires one or more characters. #### Step 2. Literal dot `\.`. #### Step 3. Choose suffix `(?:jpg|png)` chooses jpg or png. #### Step 4. Full match and test `re.fullmatch(r".+\.(?:jpg|png)", filename)`. photo.jpg passes; chart.png passes; photo.jpg.exe fails. **Answer.** Full pattern `.+\.(?:jpg|png)`; use IGNORECASE only if uppercase extensions should also be accepted. Pattern likhke ek valid, ek short/wrong, ek extra-character case zaroor mentally run karo. ### Regular expressions: build and explain exact text patterns id : tee-08 title : Regular expressions: build and explain exact text patterns title\_confirmed : false topic\_title : Regular expressions: build and explain exact text patterns mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Pattern reference for the handout's remaining tasks Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-regex--regex-reference/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-regex/regex-reference TEE · current Unit 3 syllabus and supplied Regular Expressions handout; outside confirmed M2 page cutoff | Task | Pattern / operation | Read it correctly | | --- | --- | --- | | Extract hashtags | `#[A-Za-z0-9_]+` | # followed by one or more permitted tag characters | | Extract integer sequences | `[0-9]+` | Does not include decimal dots or signs | | Extract words ending ing | `\b[A-Za-z]*ing\b` | Zero or more prefix letters, then ing | | Currency amounts, simple integers | `[$₹][0-9]+` | One currency symbol followed by digits; decimals need extension | | Words containing exact uppercase AI | `\b[A-Za-z]*AI[A-Za-z]*\b` | Case-sensitive; letters before/after may be absent | | Capital-initial alphabetic words | `\b[A-Z][A-Za-z]*\b` | One capital then zero or more letters | | Extract mentions | `@[A-Za-z0-9_]+` | Simple username rule, not any platform's complete policy | | Exactly five-letter words | `\b[A-Za-z]{5}\b` | Word boundaries plus exact length | | Starts N and ends P | `\bN[A-Za-z]*P\b` | NP itself is allowed; require + to demand middle letters | | Alphabetic words only | `\b[A-Za-z]+\b` | ASCII letter-only token matches | | Basic http/https URL extraction | `https?://[^\s<>]+` | Matches a URL-shaped span; trailing punctuation may need removal | | Collapse whitespace | `re.sub(r"\s+", " ", text)` | Replace each whitespace run with one space | ### IPv4 addresses: shape versus numeric validity `(?:[0-9]{1,3}\.){3}[0-9]{1,3}` recognizes four dotted digit groups, but permits 999.999.999.999. For numeric octets 0–255 without leading zeros, define one octet as `(?:25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])`, join four with escaped dots, and require fullmatch. An IP-address parser is clearer for production validation. **Limits:** regex recognizes surface patterns; it cannot infer the intended sense of bank, establish a person's identity from capitalization, or decide arbitrary grammatical meaning. A date/email/URL-shaped match is not automatically a valid or real-world existing item. Regex shape verify karta hai; real meaning/existence ke liye extra logic lagta hai. ### Regular expressions: build and explain exact text patterns id : tee-08 title : Regular expressions: build and explain exact text patterns title\_confirmed : false topic\_title : Regular expressions: build and explain exact text patterns mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Words, counts and probabilities: the notation you need Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-language-models--words-counts-probability/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-language-models/words-counts-probability Unit 3, pages 62–108; essential probability revision **A token** is one occurrence of a word. A **type** is one distinct word. A **corpus** is a collection of texts used for analysis or training. In “cats chase cats”, there are three word tokens but only two word types: cats and chase. A language model learns regularities from such a corpus to assign probabilities to word sequences. It does not simply count whether a sentence is grammatically correct. | Symbol | Read it as | Concrete meaning | | --- | --- | --- | | C(cat) | count of cat | Number of times cat occurs. | | C(the, cat) | count of the followed by cat | Count adjacent ordered pairs; cat the is different. | | P(cat) | probability of cat | A number from 0 to 1. | | P(cat | the) | probability of cat given the | Among contexts ending in the, how often cat follows. | | V | vocabulary size | Number of distinct outcomes the model may predict. | | N | number of tokens or predictions | Its exact meaning must be stated for the calculation. | The vertical bar **| means “given”**. The condition is on its right. In P(cat | the), the model has already seen “the” and is predicting “cat”. This direction matters: P(cat | the) and P(the | cat) usually have different denominators. **Question. A corpus contains 20 word tokens. The word cat occurs 5 times. Find the maximum-likelihood unigram probability of cat.** #### Given Total tokens N = 20; count C(cat) = 5. #### To find P(cat). #### Formula and why it applies $$ P(w)=\frac{C(w)}{N} $$ Relative frequency estimates the probability of drawing this word from the corpus. The denominator includes every token, not just distinct words. #### Solution ##### Step 1. Select the matching count The required outcome is cat, which occurs 5 times. ##### Step 2. Divide by the total number of trials $$ P(\text{cat})=\frac{5}{20}=0.25 $$ ##### Step 3. Interpret the result Under this unigram model, cat receives probability 0.25, or 25%. **Final answer.** P(cat) = 0.25. **Yaad rakho:** Jo chahiye uska count upar; total possible trials neeche. **Question. The word the has 10 outgoing occurrences in a corpus. It is followed by cat on 4 occasions and dog on 6. Find P(cat | the) and P(dog | the).** #### Given C(the, cat) = 4; C(the, dog) = 6; C(the) = 10 outgoing contexts. #### To find Both next-word probabilities. #### Formula and why it applies $$ P(w\mid h)=\frac{C(h,w)}{C(h)} $$ The condition h = the restricts attention to its 10 continuations. We do not divide by the total corpus size. #### Solution ##### Step 1. Use the cat continuation $$ P(\text{cat}\mid\text{the})=\frac{4}{10}=0.4 $$ ##### Step 2. Use the dog continuation $$ P(\text{dog}\mid\text{the})=\frac{6}{10}=0.6 $$ ##### Step 3. Check the complete row $$ 0.4+0.6=1 $$ **Final answer.** P(cat | the) = 0.4 and P(dog | the) = 0.6. **Yaad rakho:** Given waala word denominator choose karata hai. **Question. A collection has 8 sentences; 6 start with I and 2 start with we. Estimate each starting-word probability.** #### Given Each sentence receives a start marker . C() = 8; C(, I) = 6; C(, we) = 2. #### To find P(I | ) and P(we | ). #### Formula and why it applies $$ P(w_1\mid\langle s\rangle)=\frac{C(\langle s\rangle,w_1)}{C(\langle s\rangle)} $$ The start marker supplies the context for the first word; one start context is available per sentence. #### Solution ##### Step 1. Calculate the first option $$ P(I\mid\langle s\rangle)=6/8=0.75 $$ ##### Step 2. Calculate the second option $$ P(we\mid\langle s\rangle)=2/8=0.25 $$ **Final answer.** The estimated probabilities are 0.75 and 0.25. ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Unigrams, bigrams and trigrams: construct them correctly Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-language-models--n-grams-construction/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-language-models/n-grams-construction Unit 3, pages 62–108; essential probability revision **Definition.** An n-gram is a contiguous sequence of n items. For word n-grams, the items are consecutive word tokens. A unigram contains one word; a bigram contains two; a trigram contains three. Character n-grams use characters instead, so always identify the unit being counted. Tokens: we | study | natural | language → Move a window one token at a time → Bigrams: we study / study natural / natural language For m tokens without padding, the number of n-gram *positions* is max(0, m−n+1). Repeated positions may contain the same n-gram type. A bag-of-n-grams representation counts each n-gram type and loses the overall sentence order beyond the local order inside each n-gram. $$ \text{Number of unpadded n-gram positions}=\max(0,m-n+1) $$ **Question. List all word unigrams, bigrams and trigrams for “the prime minister of our country”. Do not add boundary tokens.** #### Given Six tokens: the, prime, minister, of, our, country. n = 1, 2 and 3. #### To find All three lists and their counts. #### Formula and why it applies $$ \#\text{positions}=m-n+1 $$ Each window must contain consecutive words and move one position. No wraparound or invented final word is allowed. #### Solution ##### Step 1. Take one word at a time Unigrams: (the), (prime), (minister), (of), (our), (country). Count = 6−1+1 = 6. ##### Step 2. Take two consecutive words Bigrams: (the, prime), (prime, minister), (minister, of), (of, our), (our, country). Count = 6−2+1 = 5. ##### Step 3. Take three consecutive words Trigrams: (the, prime, minister), (prime, minister, of), (minister, of, our), (of, our, country). Count = 6−3+1 = 4. **Final answer.** 6 unigrams, 5 bigrams and 4 trigrams. **Question. Construct bigrams for “I like tea” using one start and one end marker.** #### Given Original tokens: I, like, tea. Add and once each. #### To find The boundary-aware bigram sequence. #### Formula and why it applies $$ \#\text{bigrams}=(m+2)-1=m+1 $$ The start context predicts the first word, and the last word predicts that the sentence ends. #### Solution ##### Step 1. Write the padded sequence I like tea ##### Step 2. Move a two-token window (, I), (I, like), (like, tea), (tea, ). ##### Step 3. Count the windows $$ 3+1=4 $$ **Final answer.** There are four boundary-aware bigrams. **Question. Construct trigrams for “I like tea” using two start markers and one end marker.** #### Given Original sentence has three words. For a trigram model each predicted token needs two prior context tokens. #### To find All trigram prediction events. #### Formula and why it applies $$ P(w_i\mid w_{i-2},w_{i-1}) $$ Two start markers provide a complete context even for the first word. #### Solution ##### Step 1. Pad with two starts I like tea ##### Step 2. Slide a three-token window (, , I), (, I, like), (I, like, tea), (like, tea, ). ##### Step 3. Count predicted events The model predicts I, like, tea and the end marker: four predictions. **Final answer.** Four trigrams. Start markers are context; they are not ordinary predicted words. **Question. Give the bag-of-bigrams counts for “a b a b” without padding.** #### Given Four tokens in their given order. #### To find Bigram positions, distinct bigram types and their counts. #### Formula and why it applies $$ \#\text{positions}=4-2+1=3 $$ A bag stores frequencies; repeated bigrams must not be discarded. #### Solution ##### Step 1. Enumerate all positions (a,b), (b,a), (a,b). ##### Step 2. Group identical pairs | Bigram type | Count | | --- | --- | | a b | 2 | | b a | 1 | **Final answer.** Three bigram positions but two distinct bigram types. **Yaad rakho:** n-gram mein n words; context mein n−1 words. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Collocations and the purpose of a language model Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-language-models--collocations-and-lm/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-language-models/collocations-and-lm Unit 3, pages 62–108; essential probability revision **A collocation** is a conventional word combination whose words occur together more strongly than ordinary chance or independent frequencies would suggest. Examples include “strong tea”, “heavy rain” and “red wine”. A bigram is merely two consecutive items; it need not be a collocation. “of the” can be frequent because both words are frequent, while a collocation expresses a particularly strong association. | Term | What it represents | Example | | --- | --- | --- | | Bigram | Any adjacent two-token sequence | the book | | Collocation | A characteristic strongly associated expression | heavy rain | | Language model | A distribution used to score or predict sequences | P(rain | heavy) | **Definition.** A statistical language model assigns a probability to a sequence of words, or equivalently predicts the next word from preceding context. It is useful for autocomplete, speech recognition, machine translation, spelling correction, assistive communication and ranking candidate text. High probability reflects patterns learned from its training distribution; it is not a guarantee of truth or grammatical correctness. Previous words: please turn your homework → Candidate: in / over / elephant → Language model gives each candidate a probability → Application selects or ranks candidates **Noisy-channel connection.** Suppose x is the observed noisy text or speech and y is a candidate intended message. The noisy-channel approach chooses the y that maximizes P(x | y)P(y). The channel model P(x | y) asks how y could produce x; the language model P(y) asks whether y is a plausible message. P(x) is the same for all candidate y and therefore can be omitted when ranking. $$ \hat y=\arg\max_y P(y\mid x)=\arg\max_y P(x\mid y)P(y) $$ **Question. A spelling-correction system considers “flowers” and “flours”. For the observed input, the channel likelihoods are 0.2 and 0.4, and the respective language-model probabilities in context are 0.08 and 0.01. Which candidate does the noisy-channel rule choose?** #### Given Candidate flowers: P(x|flowers)=0.2, P(flowers)=0.08. Candidate flours: P(x|flours)=0.4, P(flours)=0.01. #### To find The larger unnormalized candidate score. #### Formula and why it applies $$ \operatorname{score}(y)=P(x\mid y)P(y) $$ Both the error mechanism and the language context matter. The same denominator would normalize both candidates. #### Solution ##### Step 1. Score flowers $$ 0.2\times0.08=0.016 $$ ##### Step 2. Score flours $$ 0.4\times0.01=0.004 $$ ##### Step 3. Compare $$ 0.016>0.004 $$ **Final answer.** Choose flowers. The scores are not yet posterior probabilities. **Yaad rakho:** Har word-pair bigram hai; har frequent pair meaningful collocation nahi. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Probability chain rule and the Markov assumption Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-language-models--chain-rule-markov/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-language-models/chain-rule-markov Unit 3, pages 62–108; essential probability revision **The probability chain rule** decomposes the probability of a complete sequence into successive conditional probabilities. It is different from the derivative chain rule used in deep learning. Here we multiply probabilities, not derivatives. $$ P(w_1,w_2,w_3)=P(w_1)P(w_2\mid w_1)P(w_3\mid w_1,w_2) $$ Read left to right: probability of the first word; then probability of the second after the first; then probability of the third after both previous words. For a longer sentence, continue the same pattern. The product symbol ∏ simply means “multiply all of these terms”. $$ P(w_1,\ldots,w_m)=\prod_{i=1}^{m}P(w_i\mid w_1,\ldots,w_{i-1}) $$ **The Markov assumption** approximates long history by a limited recent context. A unigram model ignores preceding words. A bigram model keeps the previous word. A trigram model keeps the previous two words. This reduces the number of counts needed, but loses longer-range dependencies. | Model | Prediction rule | For the next word country after the prime minister of our | | --- | --- | --- | | Unigram | P(wᵢ) | P(country) | | Bigram | P(wᵢ | wᵢ₋₁) | P(country | our) | | Trigram | P(wᵢ | wᵢ₋₂,wᵢ₋₁) | P(country | of, our) | | Full chain rule | P(wᵢ | all previous words) | P(country | the, prime, minister, of, our) | **Question. For the three-word sentence “I like tea”, P(I)=0.2, P(like|I)=0.5 and P(tea|I,like)=0.4. Find its probability using the full chain rule.** #### Given Three conditional factors are given, including the full two-word history for tea. No end probability is requested. #### To find P(I, like, tea). #### Formula and why it applies $$ P(I,like,tea)=P(I)P(like\mid I)P(tea\mid I,like) $$ The chain rule supplies one factor per word in the specified sequence. #### Solution ##### Step 1. Insert the three supplied factors $$ P=0.2\times0.5\times0.4 $$ ##### Step 2. Multiply the first two $$ 0.2\times0.5=0.1 $$ ##### Step 3. Multiply by the third $$ 0.1\times0.4=0.04 $$ **Final answer.** The sentence probability is 0.04. **Question. A bigram model gives P(I|)=0.5, P(like|I)=0.6, P(tea|like)=0.25 and P(|tea)=0.8. Find the full sentence probability.** #### Given All four conditional probabilities, including the end event, are supplied. #### To find P(I like tea | ). #### Formula and why it applies $$ P=P(I\mid\langle s\rangle)P(like\mid I)P(tea\mid like)P(\langle/s\rangle\mid tea) $$ Under the bigram assumption, every factor uses one preceding token. #### Solution ##### Step 1. Insert the factors in sentence order $$ P=0.5\times0.6\times0.25\times0.8 $$ ##### Step 2. Start and second word $$ 0.5\times0.6=0.3 $$ ##### Step 3. Third word $$ 0.3\times0.25=0.075 $$ ##### Step 4. End the sentence $$ 0.075\times0.8=0.06 $$ **Final answer.** The complete sentence probability is 0.06. **Yaad rakho:** Sequence probability = saare required transitions ka product. **Typical 2-mark answer.** An n-gram language model predicts a word from its previous n−1 words using a Markov approximation. For example, a bigram model uses P(wᵢ | wᵢ₋₁). It estimates these probabilities from corpus counts and multiplies the appropriate factors to score a sentence. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Unigram estimation and complete worked examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-count-to-probability--unigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-count-to-probability/unigram-estimation Unit 3, pages 97–120 and appendix **Maximum likelihood estimation (MLE)** chooses probabilities equal to observed relative frequencies. In a unigram model, every word is predicted independently of its surrounding words. If a word occurs c times in N training tokens, its estimate is c/N. To score m specified words, multiply their m unigram probabilities. $$ P_{\mathrm{MLE}}(w)=\frac{C(w)}{N}\qquad P(w_1,\ldots,w_m)\approx\prod_{i=1}^{m}P(w_i) $$ Choose a tokenization convention before counting. Lowercasing, punctuation handling and boundary markers can change the vocabulary and counts. Never count words with one convention and form the denominator with another. A unigram model gives the same product to rearrangements of the same word multiset; that is one of its limitations. **Question. Train an unpadded, case-insensitive unigram model on “I like tea” and “I like coffee”. Find the probability of “I like tea”.** #### Given Training sentences: I like tea; I like coffee. Use lowercase words; exclude punctuation and sentence boundaries. #### To find The vocabulary, word probabilities and test-sentence probability. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P(I\ like\ tea)=P(I)P(like)P(tea) $$ The unigram approximation ignores order and uses total token count for every denominator. #### Solution ##### Step 1. Count every training token Each sentence contains three words, so N = 3+3 = 6. ##### Step 2. Count types and frequencies | Word | Count | MLE probability | | --- | --- | --- | | i | 2 | 2/6 = 1/3 | | like | 2 | 2/6 = 1/3 | | tea | 1 | 1/6 | | coffee | 1 | 1/6 | Vocabulary size V = 4. ##### Step 3. Write one factor per test word $$ P=\frac{2}{6}\times\frac{2}{6}\times\frac{1}{6} $$ ##### Step 4. Multiply the fractions $$ P=\frac{4}{216}=\frac{1}{54}\approx0.018519 $$ **Final answer.** The unigram sentence probability is approximately 0.018519. **Question. A corpus contains counts a: 4, b: 3 and c: 1. Find the unigram probability of “a b a”.** #### Given C(a)=4, C(b)=3, C(c)=1. No boundary markers. #### To find P(a b a). #### Formula and why it applies $$ P(a\,b\,a)=P(a)P(b)P(a) $$ A repeated test word contributes a probability factor at every occurrence; do not include it just once. #### Solution ##### Step 1. Find the total $$ N=4+3+1=8 $$ ##### Step 2. Estimate the required probabilities $$ P(a)=4/8=1/2,\qquad P(b)=3/8 $$ ##### Step 3. Use both occurrences of a $$ P=\frac12\times\frac38\times\frac12 $$ ##### Step 4. Simplify $$ P=\frac3{32}=0.09375 $$ **Final answer.** P(a b a) = 0.09375. **Yaad rakho:** Repeated word ka probability bhi repeat multiply hoga. **Question. An unsmoothed unigram model is trained on “red blue red”. Find the probability of “red green”.** #### Given N=3. Counts: red=2, blue=1, green=0. The test contains red and green. #### To find The unsmoothed test probability and the reason for its value. #### Formula and why it applies $$ P(red\ green)=P(red)P(green) $$ MLE assigns zero to an unseen outcome. Smoothing is needed if an unseen in-vocabulary item must receive probability. #### Solution ##### Step 1. Estimate red $$ P(red)=2/3 $$ ##### Step 2. Estimate the unseen word $$ P(green)=0/3=0 $$ ##### Step 3. Multiply $$ (2/3)\times0=0 $$ **Final answer.** Unsmoothed probability = 0. This reflects missing training evidence, not proof that the phrase is impossible. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Bigram counts: build the table before multiplying Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-count-to-probability--bigram-count-table/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-count-to-probability/bigram-count-table Unit 3, pages 97–120 and appendix A **bigram probability table** uses the previous word as its row and the next word as its column. Divide each cell count by its row's total outgoing count. With one end marker on every sentence, each ordinary training word has a continuation, so its outgoing count equals its token frequency. Do not create a bigram that crosses from one sentence into the next. $$ P(w\mid h)=\frac{C(h,w)}{\sum_{v\in V}C(h,v)}=\frac{C(h,w)}{C(h)} $$ Add boundaries to each sentence separately → List adjacent pairs → Count repeated pairs → Divide each row by its total → Follow the test sentence through the table **Question. Train a bigram model on “I like tea”, “I like coffee” and “you like tea”. Use one start and one end marker. Find the probability of “I like tea”.** #### Given Corpus: I like tea ; I like coffee ; you like tea . Preserve these tokens. #### To find All nonzero bigram estimates and the test probability. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Every sentence starts and ends independently. Four probability factors are required for the three test words plus end. #### Solution ##### Step 1. List the training pairs Sentence 1: (,I), (I,like), (like,tea), (tea,). Sentence 2: (,I), (I,like), (like,coffee), (coffee,). Sentence 3: (,you), (you,like), (like,tea), (tea,). ##### Step 2. Create the nonzero count table | History → next | Pair count | History count | Probability | | --- | --- | --- | --- | | → I | 2 | 3 | 2/3 | | → you | 1 | 3 | 1/3 | | I → like | 2 | 2 | 1 | | you → like | 1 | 1 | 1 | | like → tea | 2 | 3 | 2/3 | | like → coffee | 1 | 3 | 1/3 | | tea → | 2 | 2 | 1 | | coffee → | 1 | 1 | 1 | ##### Step 3. Write the test path $$ P=P(I\mid\langle s\rangle)P(like\mid I)P(tea\mid like)P(\langle/s\rangle\mid tea) $$ ##### Step 4. Substitute each matching row $$ P=\frac23\times1\times\frac23\times1 $$ ##### Step 5. Multiply $$ P=\frac49\approx0.444444 $$ **Final answer.** P(I like tea) = 4/9 ≈ 0.444444. **Question. Using the same three-sentence corpus given here—“I like tea”, “I like coffee”, “you like tea”—find the bigram probabilities of “you like coffee” and “tea like I”. Include boundaries.** #### Given Corpus with one start/end marker per sentence. Counts: start→you=1/3; you→like=1; like→coffee=1/3; coffee→end=1. No training sentence starts with tea. #### To find The two test probabilities. #### Formula and why it applies $$ P(\text{sentence})=\prod P(\text{next token}\mid\text{previous token}) $$ A model can assign a nonzero probability to a sentence it never saw if all of its local bigrams were seen. #### Solution ##### Step 1. Score you like coffee $$ P=\frac13\times1\times\frac13\times1=\frac19\approx0.111111 $$ ##### Step 2. Inspect tea like I $$ P(tea\mid\langle s\rangle)=0/3=0 $$ ##### Step 3. Apply the zero factor The full product is zero, regardless of the remaining factors. **Final answer.** P(you like coffee) = 1/9; P(tea like I) = 0. A new sentence is not necessarily a zero-probability sentence. **Question. Obtain the unsmoothed bigram probability of “students are from Vellore” from this corpus: “I am from Vellore”; “I am a teacher”; “students are good and are from various cities”; “students from Vellore do engineering”. Use sentence boundaries.** #### Given Four training sentences, each padded with one and . Test sequence: students are from Vellore . #### To find The five required conditional probabilities and their product. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Count all occurrences in the entire corpus. In particular, are occurs twice inside the third sentence, and Vellore is followed by end once and do once. #### Solution ##### Step 1. Count the histories | History | Occurrences with continuation | | --- | --- | | | 4 | | students | 2 | | are | 2 | | from | 3 | | Vellore | 2 | ##### Step 2. Count the required pairs | Required pair | Count | Probability | | --- | --- | --- | | students | 2 | 2/4 | | students are | 1 | 1/2 | | are from | 1 | 1/2 | | from Vellore | 2 | 2/3 | | Vellore | 1 | 1/2 | ##### Step 3. Write the complete product $$ P=\frac24\times\frac12\times\frac12\times\frac23\times\frac12 $$ ##### Step 4. Multiply progressively $$ \frac24\times\frac12=\frac14,\quad \frac14\times\frac12=\frac18 $$ $$ \frac18\times\frac23=\frac1{12},\quad \frac1{12}\times\frac12=\frac1{24} $$ **Final answer.** P = 1/24 ≈ 0.0416667. **Yaad rakho:** Same sentence mein word do baar aaye toh count bhi do. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Trigram estimates: two-word history, not two denominators Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-count-to-probability--trigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-count-to-probability/trigram-estimation Unit 3, pages 97–120 and appendix A trigram model predicts the next word from the two preceding words. The numerator is the count of a triple. The denominator is the count of its two-word history with an available next token. Do not multiply two independent bigram probabilities and call that a trigram probability. $$ P(w_i\mid w_{i-2},w_{i-1})=\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})} $$ **Question. A corpus contains 12 occurrences of “prime minister”. Eight are followed by “of”, three by “said”, and one by “visited”. Estimate the three trigram probabilities.** #### Given History h = (prime, minister). Count of this history = 12. Triple counts for of, said, visited are 8, 3, 1. #### To find The conditional distribution of the next word. #### Formula and why it applies $$ P(w\mid prime,minister)=C(prime,minister,w)/12 $$ All three candidates share one two-word history, so they share denominator 12. #### Solution ##### Step 1. Calculate of $$ P(of\mid prime,minister)=8/12=2/3 $$ ##### Step 2. Calculate said $$ P(said\mid prime,minister)=3/12=1/4 $$ ##### Step 3. Calculate visited $$ P(visited\mid prime,minister)=1/12 $$ ##### Step 4. Confirm normalization $$ 8/12+3/12+1/12=12/12=1 $$ **Final answer.** The probabilities are 2/3, 1/4 and 1/12. **Question. Train a trigram model on “I like tea”, “I like coffee” and “you like tea”. Use two start markers and one end marker. Score “I like tea”.** #### Given Three sentences. Padded first sentence is I like tea ; pad the other two in the same way. #### To find Four trigram prediction probabilities and the sentence probability. #### Formula and why it applies $$ P=\prod_i P(w_i\mid w_{i-2},w_{i-1}) $$ Each prediction uses a two-token context; the two start markers supply the first history. #### Solution ##### Step 1. Count the first-word event The history (,) occurs three times, and two are followed by I. $$ P(I\mid\langle s\rangle,\langle s\rangle)=2/3 $$ ##### Step 2. Count the second-word event (,I) occurs twice; both are followed by like. $$ P(like\mid\langle s\rangle,I)=2/2=1 $$ ##### Step 3. Count the third-word event (I,like) occurs twice; one is followed by tea. $$ P(tea\mid I,like)=1/2 $$ ##### Step 4. Count the end event (like,tea) occurs twice; both are followed by end. $$ P(\langle/s\rangle\mid like,tea)=2/2=1 $$ ##### Step 5. Multiply all four $$ P=\frac23\times1\times\frac12\times1=\frac13 $$ **Final answer.** Trigram P(I like tea) = 1/3. It differs from the bigram estimate because the context and denominator changed. **Question. A trigram model supplies P(we|,)=0.4, P(study|,we)=0.5, P(NLP|we,study)=0.8 and P(|study,NLP)=0.25. Score “we study NLP”.** #### Given Four trigram conditional factors. The end marker is included. #### To find The complete sentence probability. #### Formula and why it applies $$ P=P(we\mid s,s)P(study\mid s,we)P(NLP\mid we,study)P(end\mid study,NLP) $$ Use each supplied two-word history exactly; do not swap its word order. #### Solution ##### Step 1. Insert the values $$ P=0.4\times0.5\times0.8\times0.25 $$ ##### Step 2. Multiply the first pair $$ 0.4\times0.5=0.2 $$ ##### Step 3. Include NLP $$ 0.2\times0.8=0.16 $$ ##### Step 4. Include the end $$ 0.16\times0.25=0.04 $$ **Final answer.** P = 0.04. **Yaad rakho:** Trigram: teen-word count upar, pehle do-word count neeche. **Why larger n is not automatically better.** More context can capture local order more precisely, but each longer history occurs less often. Parameter count, storage and sparsity grow. A model trained on Shakespeare may assign poor probabilities to financial news because training and test distributions differ. Keep a held-out test set; do not evaluate only on memorized training sentences. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Class-style count-table questions and partial-table traps Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-count-to-probability--class-count-table-questions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-count-to-probability/class-count-table-questions Unit 3, pages 97–120 and appendix **Question. Train a bigram model on “I am Sam”, “Sam I am”, and “I do not like green eggs and ham”. Include sentence boundaries. Find P(I|), P(Sam|), P(am|I), P(do|I), P(Sam|am) and P(|Sam).** #### Given Three sentences; one and per sentence; preserve capital I and Sam. #### To find Six bigram estimates. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Count all matching histories, including repeated words in different sentence positions. #### Solution ##### Step 1. Count the histories START occurs 3 times; I occurs 3 times; am occurs 2 times; Sam occurs 2 times. ##### Step 2. Count the requested pairs | Pair | Pair count | History count | | --- | --- | --- | | START I | 2 | 3 | | START Sam | 1 | 3 | | I am | 2 | 3 | | I do | 1 | 3 | | am Sam | 1 | 2 | | Sam END | 1 | 2 | ##### Step 3. Divide pair by history | Requested probability | Result | | --- | --- | | P(I|START) | 2/3 | | P(Sam|START) | 1/3 | | P(am|I) | 2/3 | | P(do|I) | 1/3 | | P(Sam|am) | 1/2 | | P(END|Sam) | 1/2 | **Final answer.** The six estimates are 2/3,1/3,2/3,1/3,1/2,1/2 in the requested order. **Question. A bigram table gives P(i|)=0.25, P(want|i)=0.33, P(english|want)=0.0011, P(food|english)=0.5 and P(|food)=0.68. Score “i want english food” using these supplied rounded values.** #### Given All five transition probabilities are explicitly given. Include the end marker. #### To find The sentence probability from the supplied table. #### Formula and why it applies $$ P=P(i\mid START)P(want\mid i)P(english\mid want)P(food\mid english)P(END\mid food) $$ A table-based question uses the provided values; slight differences can arise if exact corpus fractions are used instead. #### Solution ##### Step 1. First two events $$ 0.25(0.33)=0.0825 $$ ##### Step 2. Add english $$ 0.0825(0.0011)=0.00009075 $$ ##### Step 3. Add food $$ 0.00009075(0.5)=0.000045375 $$ ##### Step 4. End the sentence $$ 0.000045375(0.68)=0.000030855 $$ **Final answer.** P≈0.000030855 using the supplied rounded probabilities. **Question. A corpus has vocabulary size 1446, C(i)=2533 and C(i,want)=827. Compute the unsmoothed and add-one-smoothed P(want|i). The displayed bigram table contains only eight vocabulary columns.** #### Given Full history count C(i)=2533; pair count 827; complete vocabulary size V=1446. The displayed table is only a subset. #### To find Both conditional probabilities. #### Formula and why it applies $$ P(want\mid i)=827/2533,\qquad P_L(want\mid i)=\frac{827+1}{2533+1446} $$ Use the supplied full history total. Summing only the eight displayed columns would wrongly ignore all other possible continuations. #### Solution ##### Step 1. Calculate the unsmoothed ratio $$ P=827/2533\approx0.326490 $$ ##### Step 2. Add one to the pair count $$ 827+1=828 $$ ##### Step 3. Add V to the history count $$ 2533+1446=3979 $$ ##### Step 4. Divide $$ P_L=828/3979\approx0.208092 $$ **Final answer.** Unsmoothed≈0.326490; smoothed≈0.208092. **Yaad rakho:** Partial table ka visible row-sum, full history total nahi hota. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Perplexity: meaning, formula and four complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-evaluation-smoothing--evaluation-perplexity/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-evaluation-smoothing/evaluation-perplexity Unit 3, pages 121–156 **Intrinsic evaluation** evaluates a model directly on a held-out test corpus, for example using perplexity. **Extrinsic evaluation** places it inside a downstream application, such as a speech recognizer or translation system, and measures the application's performance. Extrinsic testing is closer to actual use but often slower and more expensive. A lower perplexity is not a guarantee of improvement on every downstream task. **Perplexity** is the inverse probability of a test sequence, normalized by the number of prediction events. Intuitively, it is the model's effective average number of equally plausible choices. A model that always gives the correct next word probability 1/4 has perplexity 4. Lower is better when the models are tested on the same data, vocabulary, tokenization and boundary convention. $$ \operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}} $$ | Symbol | Meaning | | --- | --- | | W | The complete test sequence. | | P(W) | Product of the probabilities assigned to the actual next tokens. | | N | Number of scored prediction events under the stated convention. | | Exponent −1/N | Take the reciprocal, then the Nth root. | For calculator entry, use parentheses: P raised to (−1 ÷ N), or (1 ÷ P) raised to (1 ÷ N). Retain several digits in intermediate products. The ordinary start marker is usually given context, not a predicted event; the end marker counts if its probability is included. **Question. A model assigns probability 0.0625 to a test sequence containing four scored tokens. Compute its perplexity.** #### Given P(W)=0.0625=1/16; N=4. #### To find PP(W). #### Formula and why it applies $$ \operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N} $$ The root removes the effect of sequence length; the reciprocal makes lower surprise correspond to lower perplexity. #### Solution ##### Step 1. Take the reciprocal $$ 1/P(W)=1/0.0625=16 $$ ##### Step 2. Use the fourth root $$ \operatorname{PP}=16^{1/4} $$ ##### Step 3. Evaluate $$ 2^4=16\quad\Rightarrow\quad\operatorname{PP}=2 $$ **Final answer.** Perplexity = 2. **Question. Two language models score the same three-token test sequence. Model A gives next-token probabilities 0.5, 0.5, 0.5. Model B gives 0.5, 0.25, 0.25. Compute their perplexities and select the better intrinsic result.** #### Given Same sequence and N=3 for both models. A factors: 0.5,0.5,0.5. B factors: 0.5,0.25,0.25. #### To find Both sequence probabilities and perplexities. #### Formula and why it applies $$ P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3} $$ A fair comparison uses identical test tokens and scoring conventions. #### Solution ##### Step 1. Calculate A's probability $$ P_A=0.5\times0.5\times0.5=0.125=1/8 $$ ##### Step 2. Calculate A's perplexity $$ \operatorname{PP}_A=8^{1/3}=2 $$ ##### Step 3. Calculate B's probability $$ P_B=0.5\times0.25\times0.25=0.03125=1/32 $$ ##### Step 4. Calculate B's perplexity $$ \operatorname{PP}_B=32^{1/3}\approx3.174802 $$ ##### Step 5. Compare 2 is lower than 3.174802; A places more probability on the observed sequence. **Final answer.** Model A is better by perplexity on this test: 2 versus about 3.174802. **Question. The bigram training corpus is “This is a dog”, “This is a cat”, “I love my cat”, “This is my name”. Score “This is my cat”, including an end marker, and compute perplexity using five predicted events.** #### Given Each training sentence has one start and end marker. Test predictions: This, is, my, cat, . Thus N=5. #### To find Bigram sentence probability and perplexity. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5} $$ Start is given context. Four words plus the end marker produce five probability factors. #### Solution ##### Step 1. Count the required transitions | Transition | Pair count / history count | | --- | --- | | →This | 3/4 | | This→is | 3/3 | | is→my | 1/3 | | my→cat | 1/2 | | cat→ | 2/2 | ##### Step 2. Form the product $$ P=\frac34\times1\times\frac13\times\frac12\times1=\frac18 $$ ##### Step 3. Take the reciprocal $$ 1/P=8 $$ ##### Step 4. Take the fifth root $$ \operatorname{PP}=8^{1/5}\approx1.515717 $$ **Final answer.** P(W)=1/8 and perplexity≈1.515717 using five predicted events. **If the question explicitly states N = 6:** use that supplied convention and report PP = 81/6 ≈ 1.414214. The class slide counts both boundary markers in N. Write the chosen N beside the formula; do not silently switch between 5 and 6. Also, This occurs three times and is follows it three times, so P(is|This)=3/3, not 3/4. **Question. A uniform model assigns probability 1/10 to each actual next word in a five-token test. Find its perplexity.** #### Given Five scored tokens, each with assigned probability 0.1. N=5. #### To find The sentence probability and perplexity. #### Formula and why it applies $$ P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5} $$ Uniform equal probabilities make the effective number of choices especially clear. #### Solution ##### Step 1. Multiply the five factors $$ P(W)=0.1^5=0.00001=10^{-5} $$ ##### Step 2. Invert $$ 1/P(W)=10^5 $$ ##### Step 3. Take the fifth root $$ \operatorname{PP}=(10^5)^{1/5}=10 $$ **Final answer.** Perplexity = 10. **Yaad rakho:** Probability high → surprise low → perplexity low. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Log probabilities: replace long products with sums Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-evaluation-smoothing--log-probabilities/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-evaluation-smoothing/log-probabilities Unit 3, pages 121–156 Probabilities are at most 1, so multiplying many of them produces very small numbers. Computers can round extremely small positive products to zero: **numerical underflow**. Log probabilities convert multiplication into addition and are more stable for long sequences. Natural logs use base e; log₂ uses base 2. Choose one base consistently. $$ \ln P(W)=\sum_{i=1}^{N}\ln p_i $$ $$ P(W)=e^{\ln P(W)},\qquad \operatorname{PP}(W)=\exp\left(-\frac1N\sum_i\ln p_i\right) $$ The log of a probability below 1 is negative. A value closer to zero means a larger probability: ln(0.5) ≈ −0.693 is larger than ln(0.1) ≈ −2.303. A zero probability has log −∞; taking logs does not solve the unseen-event problem. Smoothing handles that separate problem. **Question. The next-word probabilities are 0.5, 0.25 and 0.5. Calculate the natural log probability, ordinary probability and perplexity.** #### Given p₁=0.5; p₂=0.25; p₃=0.5; N=3. #### To find ln P(W), P(W), and PP. #### Formula and why it applies $$ \ln P=\sum_i\ln p_i,\quad P=e^{\ln P},\quad\operatorname{PP}=e^{-\ln P/N} $$ Logarithms preserve products as sums and the exponent converts the result back. #### Solution ##### Step 1. Compute individual logs $$ \ln0.5\approx-0.693147,\quad\ln0.25\approx-1.386294 $$ ##### Step 2. Add all three terms $$ \ln P=-0.693147-1.386294-0.693147\approx-2.772589 $$ ##### Step 3. Recover the probability $$ P=e^{-2.772589}\approx0.0625 $$ ##### Step 4. Find average negative log probability $$ -\ln P/3\approx2.772589/3=0.924196 $$ ##### Step 5. Exponentiate $$ \operatorname{PP}=e^{0.924196}\approx2.519842 $$ **Final answer.** ln P≈−2.772589, P=0.0625, PP≈2.519842. **Question. A model has a total natural log probability −12 over six scored tokens. Calculate its perplexity.** #### Given ln P(W)=−12; N=6. #### To find PP without forming the tiny ordinary probability. #### Formula and why it applies $$ \operatorname{PP}=\exp(-\ln P(W)/N) $$ The logarithmic formula directly uses the supplied log score. #### Solution ##### Step 1. Negate the log score $$ -\ln P=12 $$ ##### Step 2. Divide by token count $$ 12/6=2 $$ ##### Step 3. Apply exponential $$ \operatorname{PP}=e^2\approx7.389056 $$ **Final answer.** Perplexity≈7.389056. **Question. A language model's average negative log₂ probability is 3 bits per token. Find its perplexity.** #### Given Cross-entropy H=3 bits/token. Log base is 2. #### To find PP. #### Formula and why it applies $$ \operatorname{PP}=2^H $$ The inverse operation of log₂ is raising 2 to that power; use e only for natural logs. #### Solution ##### Step 1. Identify the log base Bits mean base 2. ##### Step 2. Raise the base to H $$ \operatorname{PP}=2^3=8 $$ **Final answer.** Perplexity = 8. **Yaad rakho:** ln use kiya toh e se wapas; log₂ use kiya toh 2 se wapas. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Laplace smoothing: why both numerator and denominator change Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-evaluation-smoothing--laplace-basics/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-evaluation-smoothing/laplace-basics Unit 3, pages 121–156 **Smoothing** reallocates some probability mass to events that received zero or unreliable estimates from finite training data. **Laplace, or add-one, smoothing** adds one pseudo-count to every possible next outcome. If there are V possible outcomes, the total added count is V, so adding 1 only to the numerator would produce probabilities that do not sum to 1. $$ P_{\mathrm{Laplace}}(w)=\frac{C(w)+1}{N+V} $$ $$ P_{\mathrm{Laplace}}(w\mid h)=\frac{C(h,w)+1}{C(h)+V} $$ For unigram smoothing, N is the total training token count. For bigram smoothing, C(h) is the count of the particular history. In both cases V is the number of outcomes being predicted, not the number of training sentences and not the number of distinct bigrams. When end is a possible prediction, include it in V; the start marker is usually context only. Observed counts: 3, 1, 0 → Add one to every outcome: 4, 2, 1 → New total: 7 → Smoothed probabilities: 4/7, 2/7, 1/7 **Question. A fixed unigram vocabulary is {a,b,c}. The training counts are 3,1,0. Compute unsmoothed and Laplace-smoothed probabilities.** #### Given C(a)=3, C(b)=1, C(c)=0; N=4; V=3. #### To find The complete distributions. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P_L(w)=(C(w)+1)/(N+V) $$ The zero-count c is still in the fixed vocabulary, so it must receive one pseudo-count and contribute to V. #### Solution ##### Step 1. Compute unsmoothed values | Word | Unsmoothed | | --- | --- | | a | 3/4 = 0.75 | | b | 1/4 = 0.25 | | c | 0/4 = 0 | ##### Step 2. Calculate the new denominator $$ N+V=4+3=7 $$ ##### Step 3. Add one to each numerator $$ C(a)+1=4,\quad C(b)+1=2,\quad C(c)+1=1 $$ ##### Step 4. Divide | Word | Smoothed | | --- | --- | | a | 4/7 ≈ 0.571429 | | b | 2/7 ≈ 0.285714 | | c | 1/7 ≈ 0.142857 | ##### Step 5. Confirm the total $$ 4/7+2/7+1/7=1 $$ **Final answer.** Smoothed probabilities: a=4/7, b=2/7, c=1/7. Common a loses mass; unseen c gains mass. **Question. After history I, a corpus records like three times, study once and sleep zero times. The allowed next vocabulary is {like,study,sleep}. Compute all Laplace-smoothed bigram probabilities.** #### Given C(I,like)=3; C(I,study)=1; C(I,sleep)=0. History count C(I)=4. V=3. #### To find P\_L(like|I), P\_L(study|I), P\_L(sleep|I). #### Formula and why it applies $$ P_L(w\mid I)=\frac{C(I,w)+1}{C(I)+V} $$ This is smoothing within the I row; use its total 4, not the global corpus token count. #### Solution ##### Step 1. Form the common denominator $$ C(I)+V=4+3=7 $$ ##### Step 2. Calculate like $$ P_L(like\mid I)=(3+1)/7=4/7 $$ ##### Step 3. Calculate study $$ P_L(study\mid I)=(1+1)/7=2/7 $$ ##### Step 4. Calculate sleep $$ P_L(sleep\mid I)=(0+1)/7=1/7 $$ **Final answer.** The row is (4/7, 2/7, 1/7). An unseen bigram is no longer forced to zero. **Question. A fixed vocabulary has five possible next tokens. A history h is unseen in training. What is its add-one probability for each next token?** #### Given C(h)=0; C(h,w)=0 for each outcome; V=5. #### To find P\_L(w|h) for each token. #### Formula and why it applies $$ P_L(w\mid h)=\frac{0+1}{0+V} $$ All outcomes receive equal pseudo-counts because the training data gives no preference for this history. #### Solution ##### Step 1. Add the pseudo-count $$ 0+1=1 $$ ##### Step 2. Form the denominator $$ 0+5=5 $$ ##### Step 3. Divide and sum $$ P_L(w\mid h)=1/5=0.2,\qquad5(0.2)=1 $$ **Final answer.** Every outcome gets 0.2 under this fixed-vocabulary model. **Vocabulary warning.** Add-one does not automatically define probabilities for every word in the world. Unknown words require a declared vocabulary/UNK convention. If a question supplies V, use that stated V rather than inventing additional tokens. **Yaad rakho:** Add-one: upar +1, neeche +V; bigram mein neeche history ka count. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · A complete unseen-bigram question from corpus to final answer Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-evaluation-smoothing--smoothed-sentence/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-evaluation-smoothing/smoothed-sentence Unit 3, pages 121–156 **Question. Consider the corpus “I love to code in Python. I love to learn new things. I code every day.” Obtain the bigram probabilities needed for “I love Python”. Explain the zero-probability issue, then use Laplace smoothing to score it. Treat each sentence separately, preserve case, remove punctuation, and include an end token among predicted outcomes.** #### Given Three sentences padded with one start/end marker. Distinct words: I, love, to, code, in, Python, learn, new, things, every, day. With , V=12. Start is context only. #### To find Unsmoothed and smoothed P(I love Python | ). #### Formula and why it applies $$ P(w\mid h)=\frac{C(h,w)}{C(h)},\qquad P_L(w\mid h)=\frac{C(h,w)+1}{C(h)+12} $$ The test contains an unseen transition love→Python. Smoothing assigns probability to that transition and renormalizes every required history row. #### Solution ##### Step 1. Count the contexts | History | Count | | --- | --- | | | 3 | | I | 3 | | love | 2 | | Python | 1 | ##### Step 2. Count each required pair | Pair | Count | | --- | --- | | I | 3 | | I love | 2 | | love Python | 0 | | Python | 1 | ##### Step 3. Compute the unsmoothed product $$ P=\frac33\times\frac23\times\frac02\times\frac11=0 $$ Both words love and Python were seen, but their adjacent pair was not. This is an unseen bigram, not an unknown-word case. ##### Step 4. Smooth the first event $$ P_L(I\mid\langle s\rangle)=\frac{3+1}{3+12}=\frac4{15} $$ ##### Step 5. Smooth the second event $$ P_L(love\mid I)=\frac{2+1}{3+12}=\frac3{15} $$ ##### Step 6. Smooth the unseen event $$ P_L(Python\mid love)=\frac{0+1}{2+12}=\frac1{14} $$ ##### Step 7. Smooth the end event $$ P_L(\langle/s\rangle\mid Python)=\frac{1+1}{1+12}=\frac2{13} $$ ##### Step 8. Multiply the smoothed factors $$ P_L=\frac4{15}\times\frac3{15}\times\frac1{14}\times\frac2{13} $$ $$ P_L=\frac{24}{40950}=\frac4{6825}\approx0.000586081 $$ **Final answer.** Unsmoothed P=0. With the stated V=12 and boundaries, Laplace-smoothed P≈0.000586081. If an exam states a different convention—for example excluding boundary tokens from the entire calculation—recompute both V and the set of probability factors consistently. Do not combine a boundary-aware numerator with a boundary-free denominator. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Adjusted counts and relative discounting Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-evaluation-smoothing--adjusted-count-discount/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-evaluation-smoothing/adjusted-count-discount Unit 3, pages 121–156 The slides also express smoothing through an **adjusted count** c\*. Instead of writing the smoothed probability with denominator N+V, they rescale its numerator so the denominator can remain N. Algebraically these are equivalent: $$ c^*=(c+1)\frac{N}{N+V},\qquad P_L(w)=\frac{c^*}{N} $$ $$ d=\frac{c^*}{c}\quad(c>0) $$ The relative discount d compares adjusted and original counts. It is undefined when c=0 because division by zero is undefined. Add-one smoothing moves the distribution toward uniformity: frequent events are reduced, but some rare seen events may increase. It is not correct to claim that every nonzero count must decrease under Laplace smoothing. **Question. For N=20 and V=5, a word has original count c=10. Find its adjusted count, smoothed probability and relative discount.** #### Given N=20; V=5; c=10. #### To find c\*, P\_L and d. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\quad P_L=c^*/N,\quad d=c^*/c $$ These expressions rewrite add-one smoothing without changing its probability. #### Solution ##### Step 1. Form the new total $$ N+V=20+5=25 $$ ##### Step 2. Find the adjusted count $$ c^*=(10+1)\frac{20}{25}=11(0.8)=8.8 $$ ##### Step 3. Find the probability $$ P_L=8.8/20=0.44 $$ ##### Step 4. Find the relative discount $$ d=8.8/10=0.88 $$ **Final answer.** c\*=8.8, P\_L=0.44, d=0.88. The original probability 0.5 is reduced. **Question. Under N=20, V=5, a rare word occurs once. Compute its adjusted count and compare the original and smoothed probabilities.** #### Given N=20; V=5; c=1. #### To find c\*, original P and smoothed P\_L. #### Formula and why it applies $$ c^*=(c+1)N/(N+V) $$ The same smoothing rule may increase a rare event's probability. #### Solution ##### Step 1. Original probability $$ P=1/20=0.05 $$ ##### Step 2. Adjusted count $$ c^*=(1+1)(20/25)=1.6 $$ ##### Step 3. Smoothed probability $$ P_L=1.6/20=0.08 $$ **Final answer.** c\*=1.6 and P\_L=0.08, which exceeds the original 0.05. **Question. For N=20 and V=5, an in-vocabulary word has count zero. Find its adjusted count and smoothed probability. Can a relative discount be calculated?** #### Given c=0; N=20; V=5. #### To find c\*, P\_L and whether d exists. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\qquad P_L=c^*/N $$ The pseudo-count creates positive mass for the unseen vocabulary item. #### Solution ##### Step 1. Adjusted count $$ c^*=(0+1)(20/25)=0.8 $$ ##### Step 2. Smoothed probability $$ P_L=0.8/20=0.04 $$ ##### Step 3. Relative discount d would be 0.8/0. That is undefined, so do not report a finite discount. **Final answer.** c\*=0.8 and P\_L=0.04; relative discount is undefined for c=0. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Character-model perplexity: keep exact probabilities Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-evaluation-smoothing--class-perplexity-characters/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-evaluation-smoothing/class-perplexity-characters Unit 3, pages 121–156 **Question. A unigram character model gives P(A)=0.25, P(B)=0.50, P(C)=0.25. Find the perplexity of AAA and ABC. Then find both perplexities when the three characters are equally likely.** #### Given Each test string contains 3 scored characters, so N=3. Initial distribution 0.25,0.50,0.25; comparison distribution 1/3,1/3,1/3. #### To find Both sequence probabilities and perplexities under each distribution. #### Formula and why it applies $$ P(c_1c_2c_3)=P(c_1)P(c_2)P(c_3),\qquad PP=P(W)^{-1/3} $$ Characters, not words, are the model's prediction units in this question. Use exact 1/3 for uniform probabilities, rather than rounded 0.33. #### Solution ##### Step 1. Score AAA $$ P(AAA)=0.25^3=\frac1{64}=0.015625 $$ ##### Step 2. Find AAA perplexity $$ PP(AAA)=64^{1/3}=4 $$ ##### Step 3. Score ABC $$ P(ABC)=0.25(0.5)(0.25)=\frac1{32}=0.03125 $$ ##### Step 4. Find ABC perplexity $$ PP(ABC)=32^{1/3}\approx3.174802 $$ ##### Step 5. Change to an exactly uniform model $$ P(AAA)=P(ABC)=(1/3)^3=1/27 $$ ##### Step 6. Find both uniform-model perplexities $$ PP=27^{1/3}=3 $$ **Final answer.** Original model: AAA→4, ABC→3.174802. Uniform model: both→3. These values follow the stated formulas without intermediate rounding. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · N-gram interpolation: weighted evidence from several histories Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-smoothing-extensions--interpolation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-smoothing-extensions/interpolation Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Faculty topic; not an observed numerical PYQ.** Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Interpolation combines several context lengths for every prediction.** A trigram estimate is specific but may have little evidence. Bigram and unigram estimates use less context and can support unseen longer sequences. A weighted mixture combines their information. $$ P_{\mathrm{mix}}(w\mid u,v)=\lambda_3P(w\mid u,v)+\lambda_2P(w\mid v)+\lambda_1P(w) $$ $$ \lambda_1,\lambda_2,\lambda_3\ge0,\qquad\lambda_1+\lambda_2+\lambda_3=1. $$ Each component must be a normalized distribution over the same vocabulary. Then summing the mixture over all possible words gives λ₁+λ₂+λ₃=1. A zero higher-order count can be rescued by positive lower-order evidence. Unlike simple backoff, interpolation can combine all levels even when the trigram was seen. **Question. For predicting tea after I like, suppose trigram, bigram and unigram probabilities are 0, 0.2 and 0.1. Use weights λ₃=0.5, λ₂=0.3 and λ₁=0.2 to calculate the interpolated probability. Repeat for coffee with probabilities 0.4, 0.1 and 0.05. [Authored worked example]** ### Answer #### Given The three distributions use one vocabulary. Nonnegative weights sum to 1. These are supplied probabilities; no count estimation is needed. #### To find Pmix(tea | I,like) and Pmix(coffee | I,like). #### Formula and why it applies $$ P_{mix}=\lambda_3P_3+\lambda_2P_2+\lambda_1P_1 $$ We add weighted alternatives from models, rather than multiply sequential word events. #### Solution ##### Step 1. Check the weights $$ 0.5+0.3+0.2=1. $$ ##### Step 2. Calculate the three tea contributions $$ 0.5(0)=0,\qquad0.3(0.2)=0.06,\qquad0.2(0.1)=0.02. $$ ##### Step 3. Add the tea contributions $$ P_{mix}(tea)=0+0.06+0.02=0.08. $$ ##### Step 4. Calculate the coffee contributions $$ 0.5(0.4)=0.20,\qquad0.3(0.1)=0.03,\qquad0.2(0.05)=0.01. $$ ##### Step 5. Add and compare $$ P_{mix}(coffee)=0.20+0.03+0.01=0.24. $$ Coffee gets three times tea’s probability in this model. Other vocabulary outcomes carry the remaining mass; these two displayed entries are not the whole distribution. **Final answer.** Tea=0.08 and coffee=0.24. A zero trigram probability does not force a zero mixture. **Yaad rakho:** Interpolation = sab context-lengths ka weighted mix. Word-sequence product aur model-mixture sum alag operations hain. This worked lesson was added after the video plan. No matching video has been generated or reviewed. --- ## TEE · Backoff and the important score-versus-probability distinction Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u3-smoothing-extensions--stupid-backoff/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u3-smoothing-extensions/stupid-backoff Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Faculty topic; not an observed numerical PYQ.** Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Backoff uses a shorter history when the longer-history estimate lacks evidence.** A properly normalized backoff model discounts some seen-event probability and allocates the remaining mass across unseen continuations using lower-order probabilities and normalization factors. **Stupid Backoff** is a specifically named efficient scoring method in the slides’ list. It uses an observed n-gram relative frequency when available; otherwise it multiplies the shorter-history score by a fixed penalty, commonly 0.4. Its output is generally an **unnormalized score**, not a conditional probability distribution. Do not insert raw Stupid Backoff scores into standard probability-based perplexity and call the result a normalized-model perplexity. $$ S(w\mid h)=\begin{cases}C(h,w)/C(h),&C(h,w)>0\\ \alpha S(w\mid h'),&C(h,w)=0.\end{cases} $$ Here h′ drops the oldest history word; α is the backoff penalty. At the unigram base, use C(w)/N. The letter α here is a penalty, not the forward-HMM variable introduced later. **Question. A bigram scorer has complete vocabulary {a,b}. After h, C(h,a)=2 and C(h,b)=0. The unigram probabilities are P(a)=P(b)=0.5. Apply Stupid Backoff with penalty 0.4 and show whether the scores form a normalized row. [Authored worked example]** ### Answer #### Given C(h)=2. The a bigram is observed; the b bigram is unseen. Unigram probabilities are supplied. #### To find S(a|h), S(b|h), their sum, and whether they are probabilities. #### Formula and why it applies Use relative frequency for an observed pair and α times the unigram base for an unseen pair. The method does not discount a’s observed score in this example. #### Solution ##### Step 1. Score the observed pair $$ S(a\mid h)=2/2=1. $$ ##### Step 2. Back off for the unseen pair $$ S(b\mid h)=0.4(0.5)=0.2. $$ ##### Step 3. Add the complete row $$ 1+0.2=1.2\ne1. $$ ##### Step 4. Interpret the failure of normalization The scores can rank a above b, but the raw row cannot be called a probability distribution. Merely labelling the method “smoothing” does not change this. **Final answer.** Scores are 1 and 0.2; their sum is 1.2. They are useful ranking scores under this rule, not normalized probabilities. | Method | Operation | Normalization | | --- | --- | --- | | Add-one/add-k | Adjust all allowed outcome counts and the total | Normalized with a fixed vocabulary | | Interpolation | Mix normalized distributions using nonnegative weights summing to 1 | Normalized | | Normalized backoff | Discount seen events and distribute remaining mass over unseen events | Normalized when its factors are correctly computed | | Stupid Backoff | Keep seen relative frequencies; penalize shorter-history scores for unseen n-grams | Generally not normalized | Scope: Unit 3 slide 147 lists Stupid Backoff and n-gram interpolation; the class calculations emphasize Laplace. These short worked extensions explain the listed methods without claiming extra official exam weight. This worked lesson was added after the video plan. No matching video has been generated or reviewed. --- ## TEE · Parts of speech and contextual ambiguity Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-pos-tagging--word-classes-pos/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-pos-tagging/word-classes-pos Unit 4, pages 3–46 **Part-of-speech (POS) tagging** assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence. | Tag | Role | Example in context | | --- | --- | --- | | NN / NNS | Singular / plural common noun | a book / two books | | NNP | Proper noun | Mary, Mumbai | | VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads | | JJ | Adjective | the red book | | RB | Adverb | runs quickly | | DT | Determiner | the, a, this | | IN | Preposition or subordinating conjunction | on the table | | PRP | Personal pronoun | she, they | | MD | Modal auxiliary | can, will, should | **Open classes** readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. **Closed classes** contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes. Input: Book that flight → Context: an imperative request → Output: Book/VB that/DT flight/NN Compare “hand me that **book**”: book is a noun here. Similarly, “I **can** swim” has can/MD, while “the **can** is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is **POS ambiguity**. | Sentence | Tagged interpretation | Reason | | --- | --- | --- | | They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. | | The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. | | She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. | | Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. | POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning. **Yaad rakho:** Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai. ### NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule id : nlp-m2-07-pos-tags-tagger-methods title : NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule title\_confirmed : true topic\_title : POS tags, ambiguity and tagging methods mode : m2 kind : focused status : available duration : 9:06 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Tagger methods, Brill net gain and the full conditional rule; clarified tagging and imperative explanations. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Lexical, rule-based, stochastic, Brill and neural taggers Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-pos-tagging--tagger-methods/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-pos-tagging/tagger-methods Unit 4, pages 3–46 **A lexical baseline** gives each known word its most frequent training tag. It is simple and often useful, but ignores context. A **rule-based tagger** uses a lexicon of possible tags plus manually designed contextual or morphological rules to resolve ambiguity. For example, after a determiner, a noun reading of “can” may be preferred in “the can”. Rules are usually heuristics with exceptions. “A red car” shows that the word immediately after an article is not always a noun. The suffix -ed does not make “red” a past-tense verb. A robust rule system uses multiple linguistic constraints instead of treating such shortcuts as universal laws. | Method | How it chooses tags | Strength | Limitation | | --- | --- | --- | --- | | Most frequent / lexical | Most common training tag for the word | Simple baseline | Misses contextual changes and unknown words | | Rule-based | Lexicon plus handwritten rules | Readable, controllable linguistic decisions | Expert effort; rules may conflict or need updates | | Stochastic | Scores tag sequences using estimated probabilities | Uses corpus evidence and context | Sparse/unseen events; depends on training domain | | Transformation-based / Brill | Start with baseline, then apply learned correction rules | Readable learned corrections | Needs tagged training data and suitable rule templates | | Neural | Learns contextual representations and sequence predictions | Can exploit richer context | Needs training resources and data; less directly interpretable | **Transformation-based error-driven learning (Brill tagging)** combines a simple starting assignment with learned correction rules. First tag the training corpus with a baseline. Compare against the gold tags. Generate candidate correction rules from templates, such as “change MD to NN after DT”. Apply the rule that fixes the most errors net of newly introduced errors; repeat until further improvement is too small. At test time apply the learned ordered rule list. Baseline: The/DT can/MD was/VBD crushed/VBN → Context rule: change can from MD to NN after DT → Result: The/DT can/NN was/VBD crushed/VBN **Rule-based versus Brill:** a purely rule-based system depends on linguists specifying its rules; a Brill tagger learns which transformations improve a labeled corpus. A **stochastic tagger** such as an HMM finds a high-probability tag sequence using transition and emission probabilities. CRFs and neural sequence taggers are other learning approaches; they do not have to make the same HMM independence assumptions. **Exam-ready answer.** POS tagging assigns a grammatical tag to each word in context. A rule-based tagger uses lexicons and linguistic rules, whereas a stochastic tagger chooses tags through probabilities estimated from a corpus. For example, can is MD in “I can swim” but NN in “the can”; context resolves the ambiguity. ### NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule id : nlp-m2-07-pos-tags-tagger-methods title : NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule title\_confirmed : true topic\_title : POS tags, ambiguity and tagging methods mode : m2 kind : focused status : available duration : 9:06 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Tagger methods, Brill net gain and the full conditional rule; clarified tagging and imperative explanations. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Generative versus discriminative models and Bayes' rule Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-pos-tagging--generative-discriminative/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-pos-tagging/generative-discriminative Unit 4, pages 3–46 Let x be the observed words and y be the desired tag sequence. A **generative model** describes their joint distribution P(x,y), often as P(y)P(x|y). A **discriminative model** learns P(y|x) directly or learns a decision boundary that separates possible labels. Both kinds can be trained with labeled data; “generative” does not mean “always unsupervised”. $$ P(y\mid x)=\frac{P(x\mid y)P(y)}{P(x)} $$ | Quantity | Meaning | POS interpretation | | --- | --- | --- | | P(y) | Prior | How plausible is this tag or tag sequence before seeing the words? | | P(x|y) | Likelihood | How plausible are these words under that tag sequence? | | P(x) | Evidence | Total probability of the observed words across possible labels. | | P(y|x) | Posterior | Probability of the tags after seeing the words. | | Aspect | Generative | Discriminative | | --- | --- | --- | | Models | Joint P(x,y), or prior × likelihood | Conditional P(y|x) or decision boundary | | Classification examples | Naive Bayes | Logistic regression, SVM | | Sequence-labeling examples | HMM | Linear-chain CRF | | Intuition | Model how a label could generate the observation | Model which label fits the observation | **Question. A token has two possible tags, noun N and verb V. P(N)=0.6, P(V)=0.4, P(book|N)=0.2 and P(book|V)=0.5. Calculate P(N|book) and P(V|book).** #### Given Two tags are exhaustive. Priors: 0.6,0.4. Word likelihoods: 0.2,0.5. #### To find Normalized posterior probabilities. #### Formula and why it applies $$ P(t\mid w)=\frac{P(w\mid t)P(t)}{\sum_{u}P(w\mid u)P(u)} $$ Bayes' rule combines prior and likelihood. Normalization is needed to convert scores into posterior probabilities. #### Solution ##### Step 1. Calculate the noun score $$ 0.2\times0.6=0.12 $$ ##### Step 2. Calculate the verb score $$ 0.5\times0.4=0.20 $$ ##### Step 3. Find the evidence $$ P(book)=0.12+0.20=0.32 $$ ##### Step 4. Normalize the noun score $$ P(N\mid book)=0.12/0.32=0.375 $$ ##### Step 5. Normalize the verb score $$ P(V\mid book)=0.20/0.32=0.625 $$ **Final answer.** Verb is more probable in this simplified model: 0.625 versus 0.375. **Question. Two candidate tags have prior-likelihood products 0.06 and 0.14. Find their normalized probabilities.** #### Given Unnormalized scores s₁=0.06 and s₂=0.14; these are the only candidates. #### To find The two posterior probabilities. #### Formula and why it applies $$ P_i=s_i/(s_1+s_2) $$ A common denominator converts the candidate scores to a distribution summing to one. #### Solution ##### Step 1. Add scores $$ 0.06+0.14=0.20 $$ ##### Step 2. Normalize first $$ 0.06/0.20=0.3 $$ ##### Step 3. Normalize second $$ 0.14/0.20=0.7 $$ **Final answer.** Posterior probabilities are 0.3 and 0.7. **Yaad rakho:** Generative: tag se word ka model. Discriminative: word se tag ka decision. ### Bayes Rule & Models id : nlp-m2-02-bayes-generative-discriminative title : Bayes Rule & Models title\_confirmed : true topic\_title : Bayes rule and generative versus discriminative models mode : m2 kind : focused status : available duration : 10:51 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Markov chains: read a transition table and multiply a path Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm--markov-chain/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm/markov-chain Unit 4, pages 47–73 **A Markov chain** is a probabilistic model of a sequence of states. Under the first-order Markov assumption, the next state's probability depends on the current state rather than the complete previous history. The **initial probability** tells us how likely each first state is. A **transition probability** tells us how likely a move from one state to another is. $$ P(s_t\mid s_1,\ldots,s_{t-1})=P(s_t\mid s_{t-1}) $$ $$ P(s_1,\ldots,s_T)=P(s_1)\prod_{t=2}^{T}P(s_t\mid s_{t-1}) $$ For T states in a path, use one initial factor and T−1 transition factors. A row-oriented transition matrix must have each complete row sum to 1. Always read the row/column labels; some exam diagrams use a transposed arrangement. | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. **Question. For the Rain/Dry Markov chain with initial P(Rain)=0.4 and P(Dry)=0.6, and transitions Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8, find the probability of Dry, Dry, Rain, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Dry,Dry,Rain,Rain). #### Formula and why it applies $$ P(D,D,R,R)=P(D)P(D\mid D)P(R\mid D)P(R\mid R) $$ Each step conditions on the immediately preceding state. The sequence contains four states, so it needs one initial plus three transition factors. #### Solution ##### Step 1. Start in Dry $$ P(D)=0.6 $$ ##### Step 2. Move Dry to Dry $$ 0.6\times0.8=0.48 $$ ##### Step 3. Move Dry to Rain $$ 0.48\times0.2=0.096 $$ ##### Step 4. Move Rain to Rain $$ 0.096\times0.3=0.0288 $$ **Final answer.** Path probability = 0.0288. **Question. For initial probabilities Rain=0.4, Dry=0.6 and transition rows Rain=(0.3,0.7), Dry=(0.2,0.8) in column order Rain,Dry, find the probability of Rain, Dry, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Rain,Dry,Rain). #### Formula and why it applies $$ P(R,D,R)=P(R)P(D\mid R)P(R\mid D) $$ Do not replace Dry→Rain with Rain→Dry: the two directional probabilities differ. #### Solution ##### Step 1. Initial Rain $$ P(R)=0.4 $$ ##### Step 2. First transition $$ P(D\mid R)=0.7,\quad0.4(0.7)=0.28 $$ ##### Step 3. Second transition $$ P(R\mid D)=0.2,\quad0.28(0.2)=0.056 $$ **Final answer.** Path probability = 0.056. **Question. For the same fully specified chain—initial Rain=0.4, Dry=0.6; Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8—find P(Dry,Dry). Also find the conditional probability of next Dry given current Dry.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find The two-state joint probability and the one-step conditional probability. #### Formula and why it applies $$ P(D,D)=P(D)P(D\mid D) $$ The joint path includes an uncertain initial state. The conditional transition assumes the current state is already Dry. #### Solution ##### Step 1. Read the conditional transition $$ P(D\mid D)=0.8 $$ ##### Step 2. Include the initial state for the joint path $$ P(D,D)=0.6\times0.8=0.48 $$ **Final answer.** P(Dry,Dry)=0.48; P(next Dry|current Dry)=0.8. These answer different questions. **Yaad rakho:** Path: pehle start probability, phir har arrow ki probability. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Hidden states, observations, transitions and emissions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm--hmm-components/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm/hmm-components Unit 4, pages 47–73 **A Hidden Markov Model (HMM)** is a probabilistic sequence model in which the underlying state sequence is unobserved, while each state emits an observed item. For POS tagging, words are observed and their grammatical tags are hidden. The task is to infer a suitable tag sequence from the words. Hidden: Noun → Hidden: Verb → Hidden: Noun Observed: birds → Observed: eat → Observed: seeds Horizontal state-to-state links are **transitions**. The relation from each hidden state to its observed word is an **emission**. A transition P(V|N) asks how often a verb tag follows a noun tag. An emission P(eat|V) asks how often the observed word eat appears among verb-tagged tokens. | Component | Notation | Meaning | | --- | --- | --- | | State set | S={N,V,…} | Possible hidden tags. | | Observation vocabulary | O={birds,eat,seeds,…} | Visible word tokens. | | Initial distribution | πⱼ=P(t₁=j) | Probability of starting in tag j. | | Transition matrix | aᵢⱼ=P(tₖ=j|tₖ₋₁=i) | Probability of next tag j after i. | | Emission matrix | bⱼ(w)=P(w|tag j) | Probability that state j emits word w. | Two basic assumptions are used: the current hidden state depends only on the previous hidden state, and the current observation depends only on the current hidden state. A standard first-order HMM therefore does not explicitly condition an emitted word on the whole sentence. $$ P(t_{1:T},w_{1:T})=\pi_{t_1}b_{t_1}(w_1)\prod_{k=2}^{T}a_{t_{k-1},t_k}b_{t_k}(w_k) $$ Read the formula as **start × first emission × transition × emission × transition × emission…**. There are T emission factors and T−1 transitions, plus one initial factor. If the question supplies an explicit end-state probability, multiply that too. Do not invent an end factor when the model has none. **Three different questions.** (1) Scoring a specified tag path uses the joint product above. (2) Finding the probability of the observed words sums scores over all possible tag paths. (3) Finding the best tag path maximizes the path score, typically using Viterbi. A joint path score is not a normalized posterior P(tags|words). **Yaad rakho:** Transition = tag se agla tag. Emission = tag se visible word. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Estimate transition and emission probabilities from tagged data Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm--transition-emission-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm/transition-emission-estimation Unit 4, pages 47–73 $$ P(t'\mid t)=\frac{C(t,t')}{C(t)\text{ with a following tag/end event}} $$ $$ P(w\mid t)=\frac{C(w,t)}{C(t)} $$ In a transition denominator, count the previous tag's outgoing events. In an emission denominator, count all tokens bearing that tag. With explicit end transitions the totals align conveniently. Keep the two numerator types separate: tag→tag counts are not word→tag counts. **Question. A tagged corpus contains 116454 determiner tags DT, each counted with an outgoing event. DT is followed by NN 56509 times. Estimate P(NN|DT).** #### Given C(DT)=116454; C(DT,NN)=56509. #### To find The DT→NN transition probability. #### Formula and why it applies $$ P(NN\mid DT)=C(DT,NN)/C(DT) $$ The denominator is the number of previous DT contexts because DT is the condition. #### Solution ##### Step 1. Identify the transition count DT followed by NN occurs 56509 times. ##### Step 2. Identify the conditional total There are 116454 DT contexts. ##### Step 3. Divide $$ P(NN\mid DT)=\frac{56509}{116454}\approx0.485247 $$ **Final answer.** P(NN|DT)≈0.485247. **Question. The tag VBZ occurs 21627 times, and 10073 of those tokens are the word “is”. Estimate P(is|VBZ).** #### Given C(VBZ)=21627; C(is,VBZ)=10073. #### To find The emission probability of is from VBZ. #### Formula and why it applies $$ P(is\mid VBZ)=C(is,VBZ)/C(VBZ) $$ The condition is the tag VBZ; we ask which word it emits. #### Solution ##### Step 1. Choose the word-tag count The required word is is with tag VBZ: count 10073. ##### Step 2. Choose the tag total All VBZ-tagged word tokens: count 21627. ##### Step 3. Divide $$ P(is\mid VBZ)=\frac{10073}{21627}\approx0.465760 $$ **Final answer.** P(is|VBZ)≈0.465760. This is different from P(VBZ|is). **Question. Use the tagged sentences “dogs/N chase/V cats/N” and “cats/N sleep/V”. Estimate the initial tag probabilities, all nonzero transitions including END, and emissions.** #### Given Corpus: START dogs/N chase/V cats/N END; START cats/N sleep/V END. #### To find Initial distribution, transition rows and emission rows. #### Formula and why it applies $$ \pi_t=C(START,t)/2,\quad a_{tu}=C(t,u)/C(t),\quad b_t(w)=C(w,t)/C(t) $$ The corpus explicitly tells us the tags, so we can count training frequencies directly. #### Solution ##### Step 1. Count tag tokens N occurs 3 times: dogs, cats, cats. V occurs 2 times: chase, sleep. ##### Step 2. Count sentence starts Both sentences start with N. $$ \pi_N=2/2=1,\quad \pi_V=0/2=0 $$ ##### Step 3. Count transitions | Pair | Count | | --- | --- | | N→V | 2 | | N→END | 1 | | V→N | 1 | | V→END | 1 | ##### Step 4. Normalize transition rows | Current | To N | To V | To END | | --- | --- | --- | --- | | N | 0 | 2/3 | 1/3 | | V | 1/2 | 0 | 1/2 | ##### Step 5. Count and normalize emissions | Tag | dogs | cats | chase | sleep | | --- | --- | --- | --- | --- | | N | 1/3 | 2/3 | 0 | 0 | | V | 0 | 0 | 1/2 | 1/2 | **Final answer.** π=(1,0); transitions and emissions are given in the completed tables. Each complete probability row sums to 1. **Yaad rakho:** Tag→tag table aur tag→word table ko mix mat karna. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · HMM path probabilities: three complete numerical examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm--hot-cold-paths/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm/hot-cold-paths Unit 4, pages 47–73 The following model uses hidden weather states Hot (H) and Cold (C). Observations are numbers 1, 2 and 3. These numbers are observation labels, not quantities to multiply directly. Their emission probabilities come from the table. | Initial state | Probability | | --- | --- | | H | 0.8 | | C | 0.2 | | From → to | H | C | | --- | --- | --- | | H | 0.7 | 0.3 | | C | 0.4 | 0.6 | | Emission state | Observation 1 | Observation 2 | Observation 3 | | --- | --- | --- | --- | | H | 0.2 | 0.4 | 0.4 | | C | 0.5 | 0.4 | 0.1 | **Question. An HMM has π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for observations (1,2,3) are H:(0.2,0.4,0.4) and C:(0.5,0.4,0.1). Find the joint probability of hidden path H,C,H and observations 1,3,1.** #### Given Hidden path H→C→H. Observations 1→3→1. Model probabilities are specified in the question. There is no END factor. #### To find P(H,C,H; 1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HC}b_C(3)a_{CH}b_H(1) $$ There are three hidden states and observations: three emissions, two transitions, and one initial probability. #### Solution ##### Step 1. Start Hot and emit 1 $$ 0.8\times0.2=0.16 $$ ##### Step 2. Transition Hot to Cold $$ 0.16\times0.3=0.048 $$ ##### Step 3. Cold emits 3 $$ 0.048\times0.1=0.0048 $$ ##### Step 4. Transition Cold to Hot $$ 0.0048\times0.4=0.00192 $$ ##### Step 5. Hot emits the last 1 $$ 0.00192\times0.2=0.000384 $$ **Final answer.** Joint probability = 0.000384 = 3.84×10⁻⁴. **Question. For π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels (1,2,3), find the joint probability of path H,H,H and observations 1,3,1.** #### Given State path H→H→H, observations 1→3→1; no END factor. #### To find P(H,H,H;1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HH}b_H(3)a_{HH}b_H(1) $$ The state remains Hot; each transition still contributes its probability 0.7. #### Solution ##### Step 1. Initial state and first emission $$ 0.8(0.2)=0.16 $$ ##### Step 2. First self-transition $$ 0.16(0.7)=0.112 $$ ##### Step 3. Emit observation 3 $$ 0.112(0.4)=0.0448 $$ ##### Step 4. Second self-transition $$ 0.0448(0.7)=0.03136 $$ ##### Step 5. Emit final observation 1 $$ 0.03136(0.2)=0.006272 $$ **Final answer.** Joint probability = 0.006272. A self-transition is not automatically probability 1. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6) in next-state order H,C; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3, find the score of path C,C,H with observations 1,3,1.** #### Given Hidden path C→C→H. Observation sequence 1→3→1. No end-state factor. #### To find The joint path-and-observation probability. #### Formula and why it applies $$ P=\pi_C b_C(1)a_{CC}b_C(3)a_{CH}b_H(1) $$ The initial state is Cold, so use π(C), not π(H). Each emission is selected from the state at the same position. #### Solution ##### Step 1. Start Cold and emit 1 $$ 0.2(0.5)=0.1 $$ ##### Step 2. Stay Cold $$ 0.1(0.6)=0.06 $$ ##### Step 3. Emit 3 from Cold $$ 0.06(0.1)=0.006 $$ ##### Step 4. Move Cold to Hot $$ 0.006(0.4)=0.0024 $$ ##### Step 5. Emit 1 from Hot $$ 0.0024(0.2)=0.00048 $$ **Final answer.** Joint probability = 0.00048. **Yaad rakho:** Har column par: state ka emission. Columns ke beech: transition. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · POS HMM worked corpus: Mary, Jane, can, will and spot Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm--tagging-corpus-case/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm/tagging-corpus-case Unit 4, pages 47–73 The class example contains words that can be either names or ordinary verbs/modal auxiliaries. The term is **modal** (M), not “model”. To avoid guessing tags from spelling, the training corpus below supplies them explicitly and lowercases words for the emission vocabulary. | Training sentence | Gold tags | | --- | --- | | mary jane can see will | N N M V N | | spot will see mary | N M V N | | will jane spot mary | M N V N | | mary will pat spot | N M V N | There are 9 noun tokens, 4 modal tokens and 4 verb tokens. Each sentence includes a start and end transition. For nouns, the next event can be another noun, modal, verb or END. | Current → next | N | M | V | END | | --- | --- | --- | --- | --- | | START | 3/4 | 1/4 | 0 | 0 | | N | 1/9 | 3/9 | 1/9 | 4/9 | | M | 1/4 | 0 | 3/4 | 0 | | V | 4/4 | 0 | 0 | 0 | | Tag → word | mary | jane | can | see | will | spot | pat | | --- | --- | --- | --- | --- | --- | --- | --- | | N | 4/9 | 2/9 | 0 | 0 | 1/9 | 2/9 | 0 | | M | 0 | 0 | 1/4 | 0 | 3/4 | 0 | 0 | | V | 0 | 0 | 0 | 2/4 | 0 | 1/4 | 1/4 | **Question. Using the fully tagged four-sentence corpus above, calculate the joint score of “will jane spot mary” assigned tags M,N,V,N, including END.** #### Given Required factors from the tables: START→M=1/4; emit will|M=3/4; M→N=1/4; jane|N=2/9; N→V=1/9; spot|V=1/4; V→N=1; mary|N=4/9; N→END=4/9. #### To find The specified tag-path score, including words and sentence end. #### Formula and why it applies $$ P=\pi_M b_M(will)a_{MN}b_N(jane)a_{NV}b_V(spot)a_{VN}b_N(mary)a_{N,END} $$ Both transitions and emissions are needed. A tag transition alone cannot tell us whether a particular word is plausible. #### Solution ##### Step 1. Start and emit will $$ \frac14\times\frac34=\frac3{16} $$ ##### Step 2. Move to N and emit jane $$ \frac3{16}\times\frac14\times\frac29=\frac1{96} $$ ##### Step 3. Move to V and emit spot $$ \frac1{96}\times\frac19\times\frac14=\frac1{3456} $$ ##### Step 4. Move to N and emit mary $$ \frac1{3456}\times1\times\frac49=\frac1{7776} $$ ##### Step 5. End the sentence $$ \frac1{7776}\times\frac49=\frac1{17496}\approx0.0000571559 $$ **Final answer.** The joint score is 1/17496 ≈ 0.0000571559. **Question. For the same tagged corpus and tables, explain the unsmoothed score of “will can spot mary” assigned tags M,V,N,N.** #### Given The proposed second word is can with hidden tag V. In the training corpus, can occurs only with M, so b\_V(can)=0. #### To find Whether the complete path score is zero and what this means. #### Formula and why it applies $$ P(\text{path,words})=(\text{other required factors})\times b_V(can) $$ A product containing one zero factor is zero even if every other factor is positive. #### Solution ##### Step 1. Find the critical emission $$ b_V(can)=C(can,V)/C(V)=0/4=0 $$ ##### Step 2. Use the full-product consequence $$ P(\text{path,words})=0 $$ ##### Step 3. Interpret correctly This unsmoothed corpus model has no evidence for can as a verb. It does not prove that a verb usage of can is impossible in English. **Final answer.** The unsmoothed path score is zero. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Why exhaustive search grows exponentially Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm-decoding--sequence-count/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm-decoding/sequence-count Unit 4 page 73: Viterbi motivation, explained with small worked extensions If each of T observed words can receive any of N tags, there are N choices at each position. Thus the total number of complete tag sequences is N multiplied by itself T times. This counts possible assignments before grammar rules or zero probabilities eliminate any. $$ \text{Number of tag sequences}=N^T $$ **Question. A four-word sentence is tagged using three possible POS tags per word. How many complete tag sequences are possible?** #### Given N=3 possible tags per position; T=4 word positions. #### To find The number of tag sequences. #### Formula and why it applies $$ M=N^T $$ Every independent position contributes a factor of three choices. #### Solution ##### Step 1. Write one factor per position $$ M=3\times3\times3\times3 $$ ##### Step 2. Multiply $$ 3\times3=9,\quad9\times3=27,\quad27\times3=81 $$ **Final answer.** 81 possible tag sequences. **Question. A five-word sentence has two candidate tags per word. Count its tag sequences.** #### Given N=2; T=5. #### To find M. #### Formula and why it applies $$ M=N^T $$ The length is the exponent, not a multiplier. #### Solution ##### Step 1. Substitute $$ M=2^5 $$ ##### Step 2. Evaluate $$ 2\times2\times2\times2\times2=32 $$ **Final answer.** 32 possible sequences. **Question. Using 45 possible tags, compare the number of tag sequences for sentences of length 2 and 3.** #### Given N=45. First T=2, then T=3. #### To find Both counts and the growth factor. #### Formula and why it applies $$ M_2=45^2,\quad M_3=45^3 $$ Adding one position multiplies the possibilities by the number of tag choices. #### Solution ##### Step 1. Two words $$ 45^2=2025 $$ ##### Step 2. Three words $$ 45^3=2025\times45=91125 $$ ##### Step 3. Compare $$ 91125/2025=45 $$ **Final answer.** 2 words: 2025 paths. 3 words: 91125 paths. One additional word multiplies the count by 45. The slides introduce Viterbi to avoid enumerating every path. It reuses the best prefix score for each ending state. For a dense first-order HMM, it needs O(TN²) recurrence work instead of enumerating NT complete paths. **Yaad rakho:** Choices per word ki power = sentence length. ### M2 10 Viterbi Decoding id : nlp-m2-10-viterbi-decoding title : M2 10 Viterbi Decoding title\_confirmed : true topic\_title : Viterbi decoding with backpointers mode : m2 kind : focused status : available duration : 9:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Viterbi decoding: compare incoming paths, keep the best, trace back Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm-decoding--viterbi-explained/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm-decoding/viterbi-explained Unit 4 page 73: Viterbi motivation, explained with small worked extensions **The Viterbi algorithm** finds the most likely hidden-state sequence for a fixed observation sequence. At each position, it keeps the best path ending in each state. This is safe because, under the HMM assumptions, future extensions depend on that ending state, not the discarded details of the prefix. $$ \delta_1(j)=\pi_jb_j(o_1) $$ $$ \delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ $$ \psi_t(j)=\arg\max_i[\delta_{t-1}(i)a_{ij}] $$ | Symbol | Meaning in ordinary words | | --- | --- | | δₜ(j) | Score of the best prefix that ends in state j at position t. | | ψₜ(j) | Previous state that supplied that best prefix; keep it as a backpointer. | | max | Choose the larger candidate score; do not add. | | argmax | Remember which previous state won, rather than only its value. | Initialize: start × first emission → For each next state: compare incoming scores → Multiply winner by this state's emission → Keep score + winning predecessor → Choose best final state and trace arrows backward Do not choose the best state independently at each word. The best last state can come from a different best predecessor than a local guess. Backpointers recover one consistent complete path. If scores tie, either tied path is optimal under the model; state the tie. **Question. Use an HMM with π(H)=0.8, π(C)=0.2, transitions H:(0.7,0.3), C:(0.4,0.6) in H,C order, and emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3. Find the most likely hidden path for observations 1,3,1 using Viterbi.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find The best hidden-state sequence and its joint score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(o_1),\quad\delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ Viterbi keeps two best prefix scores per column, one ending H and one ending C. #### Solution ##### Step 1. Initialize at observation 1 $$ \delta_1(H)=0.8(0.2)=0.16 $$ $$ \delta_1(C)=0.2(0.5)=0.10 $$ ##### Step 2. At observation 3, calculate the H cell $$ H\to H:\ 0.16(0.7)=0.112 $$ $$ C\to H:\ 0.10(0.4)=0.040 $$ $$ \delta_2(H)=\max(0.112,0.040)(0.4)=0.0448 $$ Backpointer for this H cell: H. ##### Step 3. At observation 3, calculate the C cell $$ H\to C:\ 0.16(0.3)=0.048 $$ $$ C\to C:\ 0.10(0.6)=0.060 $$ $$ \delta_2(C)=\max(0.048,0.060)(0.1)=0.006 $$ Backpointer for this C cell: C. ##### Step 4. At final observation 1, calculate H $$ H\to H:\ 0.0448(0.7)=0.03136 $$ $$ C\to H:\ 0.006(0.4)=0.0024 $$ $$ \delta_3(H)=0.03136(0.2)=0.006272 $$ Backpointer: H. ##### Step 5. At final observation 1, calculate C $$ H\to C:\ 0.0448(0.3)=0.01344 $$ $$ C\to C:\ 0.006(0.6)=0.0036 $$ $$ \delta_3(C)=0.01344(0.5)=0.00672 $$ Backpointer: H. ##### Step 6. Select the final state 0.00672 > 0.006272, so the best final state is C. ##### Step 7. Trace back Final C came from H at position2. That H came from H at position1. Reading forward gives H,H,C. ##### Step 8. Verify the recovered complete path $$ 0.8(0.2)(0.7)(0.4)(0.3)(0.5)=0.00672 $$ **Final answer.** Best path H→H→C with joint score 0.00672. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 3,1.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best two-state hidden path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(3),\quad\delta_2(j)=b_j(1)\max_i[\delta_1(i)a_{ij}] $$ The same recurrence applies even though the observation order changed. #### Solution ##### Step 1. Initialize observation 3 $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.1)=0.02 $$ ##### Step 2. Calculate the final H $$ \max(0.32(0.7),0.02(0.4))(0.2) $$ $$ =\max(0.224,0.008)(0.2)=0.0448 $$ Winning predecessor H. ##### Step 3. Calculate the final C $$ \max(0.32(0.3),0.02(0.6))(0.5) $$ $$ =\max(0.096,0.012)(0.5)=0.048 $$ Winning predecessor H. ##### Step 4. Select and trace Final C wins with 0.048 and came from H. The path is H,C. **Final answer.** Best path H→C, score 0.048. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 2,2.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(2),\quad\delta_2(j)=b_j(2)\max_i[\delta_1(i)a_{ij}] $$ Both states emit2 with probability 0.4, but their initial and transition probabilities still differ. #### Solution ##### Step 1. Initialize $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.4)=0.08 $$ ##### Step 2. Final H $$ \max(0.32(0.7),0.08(0.4))(0.4) $$ $$ =\max(0.224,0.032)(0.4)=0.0896 $$ Predecessor H. ##### Step 3. Final C $$ \max(0.32(0.3),0.08(0.6))(0.4) $$ $$ =\max(0.096,0.048)(0.4)=0.0384 $$ Predecessor H. ##### Step 4. Select and trace The final H score 0.0896 is larger. Its predecessor is H. **Final answer.** Best path H→H with score 0.0896. **Yaad rakho:** Har destination ke liye incoming paths compare; winner ka arrow yaad. ### M2 10 Viterbi Decoding id : nlp-m2-10-viterbi-decoding title : M2 10 Viterbi Decoding title\_confirmed : true topic\_title : Viterbi decoding with backpointers mode : m2 kind : focused status : available duration : 9:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · A path score is not the probability of the observed sentence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-hmm-decoding--joint-vs-posterior/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-hmm-decoding/joint-vs-posterior Unit 4 page 73: Viterbi motivation, explained with small worked extensions **Question. An HMM has π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3. For observations 1,3, calculate the probability of the observations by summing all four two-state paths.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find P(observations 1,3), not just the best-path score. #### Formula and why it applies $$ P(o_1,o_2)=\sum_{s_1,s_2}\pi_{s_1}b_{s_1}(o_1)a_{s_1s_2}b_{s_2}(o_2) $$ The hidden path is unspecified, so its mutually exclusive alternatives must be summed. #### Solution ##### Step 1. Score H,H $$ 0.8(0.2)(0.7)(0.4)=0.0448 $$ ##### Step 2. Score H,C $$ 0.8(0.2)(0.3)(0.1)=0.0048 $$ ##### Step 3. Score C,H $$ 0.2(0.5)(0.4)(0.4)=0.016 $$ ##### Step 4. Score C,C $$ 0.2(0.5)(0.6)(0.1)=0.006 $$ ##### Step 5. Add all paths $$ P(1,3)=0.0448+0.0048+0.016+0.006=0.0716 $$ **Final answer.** Observation probability =0.0716. The best single-path score 0.0448 is a different quantity. If asked for the posterior of path H,H given observations 1,3, normalize: 0.0448/0.0716 ≈0.625698. Viterbi chooses the best path using joint scores because the observation denominator is common to every candidate. It does not compute that denominator by taking a maximum. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Chunking groups words; chinking removes words from a candidate chunk Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-chunks-grammar--chunking-chinking/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-chunks-grammar/chunking-chinking Unit 4, pages 14–19, 74–97 **Chunking**, also called shallow parsing, groups adjacent tagged words into meaningful non-overlapping units, commonly noun phrases. It usually identifies local phrase boundaries without producing the complete hierarchical structure of a sentence. **Chinking** removes specified tokens from an existing candidate chunk, splitting or trimming it. The/DT small/JJ cat/NN sat/VBD on/IN the/DT mat/NN → NP: [The small cat] → Outside: sat on → NP: [the mat] A POS tagger labels individual words. A chunker groups words into phrases. A full parser shows how phrases relate recursively to form the entire sentence. These are related tasks with different outputs. | Operation | Input | Output | | --- | --- | --- | | POS tagging | The small cat | The/DT small/JJ cat/NN | | NP chunking | Tagged tokens | [NP The small cat] | | Chinking | A broad candidate chunk | Remove verbs/prepositions, keep noun-phrase segments | | Full parsing | Sentence + grammar | A nested tree with sentence, noun phrase and verb phrase nodes | **Rule-based chunking** uses tag patterns. Statistical or learned chunkers infer boundaries from labeled examples. A useful NP rule is an optional determiner, followed by zero or more adjectives, followed by a noun. In NLTK-style notation: `NP: {
?*}`. Here ? means zero or one occurrence; \* means zero or more; + means one or more when used. The braces mean “form a chunk around the matched sequence”. **Question. Apply the NP rule optional DT, zero or more JJ, then one NN to “The/DT quick/JJ brown/JJ fox/NN jumps/VBZ over/IN the/DT lazy/JJ dog/NN”.** #### Given Tagged input is supplied. Rule: DT? JJ\* NN. #### To find All noun-phrase chunks. #### Formula and why it applies `NP → DT? JJ* NN` The rule groups a noun with its preceding determiner and adjectives; a verb or preposition does not match this NP pattern. #### Solution ##### Step 1. Start at The The matches optional DT; quick and brown both match JJ\*; fox matches NN. ##### Step 2. Form the first NP **[NP The quick brown fox]** ##### Step 3. Skip nonmatching tokens jumps/VBZ and over/IN do not satisfy the NP pattern. ##### Step 4. Match the second NP the is DT, lazy is JJ, dog is NN: **[NP the lazy dog]**. **Final answer.** [NP The quick brown fox] jumps over [NP the lazy dog]. **Question. Using NP → DT? JJ\* NN, identify the chunks in “bright/JJ stars/NN shine/VBP and/CC a/DT moon/NN glows/VBZ”. Use the supplied tags exactly.** #### Given Input tags label stars as NN for this exercise. Rule requires optional DT, zero or more JJ, then NN. #### To find The matched NP spans. #### Formula and why it applies `NP → DT? JJ* NN` The optional determiner may be absent and the adjective repetition may be empty. #### Solution ##### Step 1. Match bright stars No determiner is present, which DT? permits. bright matches JJ; stars matches NN. ##### Step 2. Inspect shine and and VBP and CC fall outside the NP rule. ##### Step 3. Match a moon a matches DT; no adjective is present; moon matches NN. **Final answer.** Chunks are [NP bright stars] and [NP a moon]. With ordinary Penn tagging, stars is usually NNS; broaden the noun pattern to include NNS when that tag is used. **Question. Begin with one candidate chunk containing “The/DT cat/NN sat/VBD on/IN the/DT mat/NN”. Chink out verbs and prepositions. Show the remaining chunks.** #### Given Initial candidate: [The cat sat on the mat]. Remove tags VBD and IN. #### To find The chunk segments after exclusion. #### Formula and why it applies Chinking removes matched tokens from the existing chunk. Removing the middle verb/preposition region divides the original candidate into two remaining contiguous spans. #### Solution ##### Step 1. Locate the tokens to remove sat is VBD and on is IN. ##### Step 2. Remove those tokens from chunk membership The remaining token groups are “The cat” and “the mat”. sat and on still exist in the sentence; they are simply outside these chunks. ##### Step 3. Write the final chunked sentence **[The cat]** sat on **[the mat]**. **Final answer.** Two chunks remain: The cat; the mat. **Yaad rakho:** Chunking = andar group karo; chinking = unwanted tokens ko group se bahar karo. ### NLP M2 — Chunking and Chinking | Reviewed rules and worked examples id : nlp-m2-11-chunking-chinking title : NLP M2 — Chunking and Chinking | Reviewed rules and worked examples title\_confirmed : true topic\_title : Chunking and chinking from POS-tagged words mode : m2 kind : focused status : available duration : 8:22 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Correct POS labels including JJ = adjective, full NP rule application and chinking boundaries. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · CFG: terminals, nonterminals, production rules and parse trees Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-chunks-grammar--cfg-foundation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-chunks-grammar/cfg-foundation Unit 4, pages 14–19, 74–97 **A context-free grammar (CFG)** specifies a language using production rules whose left side is a single nonterminal. Formally, G=(N,Σ,R,S): N is the set of nonterminals, Σ the terminal vocabulary, R the production rules, and S the start symbol. A rule can expand its nonterminal regardless of the neighboring symbols; that is the meaning of “context-free”. | Concept | Plain explanation | Example | | --- | --- | --- | | Terminal | A word at a leaf; it is not expanded further | the, girl, read | | Nonterminal | A grammatical category that can be expanded | S, NP, VP, Det, N, V | | Production | A permitted replacement | S → NP VP | | Start symbol | Category for a complete sentence | S | | Lexicon | Word-to-category rules | N → girl | paper | | Derivation | Sequence of rule applications | S ⇒ NP VP ⇒ Det N VP ⇒ … | | Parse tree | Hierarchical picture of those expansions | Root S, internal phrases, word leaves | $$ S\to NP\ VP,\quad NP\to Det\ N,\quad VP\to V\ NP $$ Read S→NP VP as “a sentence can contain a noun phrase followed by a verb phrase”. It does not mean subtraction or multiplication. A vertical bar in a grammar rule means alternatives: Det→the | a allows either the or a. This is different from the probability bar meaning “given”. S: the whole sentence → NP: the girl + VP: read the paper → NP→Det N; VP→V NP → Leaves in order: the / girl / read / the / paper **A constituent** is a group of words that acts as a grammatical unit, such as “the girl”. **Syntactic parsing** assigns a grammatical structure to an input sentence using a grammar. It can support grammar checking, information extraction, translation and later semantic analysis. Failure under one restricted grammar means that grammar could not derive the sentence; it does not necessarily prove the sentence is invalid English. **A treebank** is a corpus annotated with parse trees. It supplies training/evaluation structures for parsers and lets linguists study recurring syntactic patterns. A tagged corpus gives word tags; a treebank additionally records relationships among phrases. | Representation | Example information retained | | --- | --- | | Bag of words | Words occur, but word order/grammar is lost | | POS tags | Each word's local grammatical category | | Chunks | Short phrase boundaries | | Parse tree | Nested constituent structure of the whole sentence | **Yaad rakho:** Grammar rule ek expansion instruction hai: S ko NP aur VP mein kholo. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Ambiguity and grammar coverage Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-chunks-grammar--structural-ambiguity/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-chunks-grammar/structural-ambiguity Unit 4, pages 14–19, 74–97 **Structural ambiguity** occurs when a sentence has more than one valid parse tree. In “I saw the man with a telescope”, the prepositional phrase “with a telescope” can attach to saw (the instrument I used) or to the man (the man possessing the telescope). The words and their order stay the same, but the grouping changes. | Reading | Attachment | Meaning | | --- | --- | --- | | I [saw the man] [with a telescope] | PP modifies VP | I used a telescope to see the man. | | I saw [the man with a telescope] | PP modifies NP | The man had a telescope. | A parser may need context, probabilities or semantic constraints to choose between trees. A plain CFG describes permitted structures but does not itself assign a probability to each tree. Do not invent rule probabilities for an ordinary CFG question. Lexical ambiguity also matters: book can be a noun or verb. In “Book the flight”, the imperative analysis begins S→VP, followed by VP→V NP. A parser that initially chooses book as a noun may reach a dead end and must try another permitted analysis. **Yaad rakho:** Same words, alag grouping → alag meaning. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Recursive-descent top-down parsing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-parsing--top-down-theory/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-parsing/top-down-theory Unit 4, pages 98–116 **Top-down parsing** starts at the grammar's start symbol S and expands nonterminals until the leaves match the input words in order. **Recursive descent** commonly implements each nonterminal as a procedure that tries its productions. It builds from the root toward the leaves. Start with S → Choose an allowed S production → Expand its next nonterminal → Match the resulting terminal to the input → Continue, or backtrack if the choice fails **Algorithm.** Start with S as the current goal. Choose a production for the leftmost unexpanded nonterminal. Replace that nonterminal by the production's right side. If a terminal appears, compare it with the next input token. If it matches, consume the token; if it does not, reject that branch and try another production. Accept only when the complete input is consumed and all grammar goals are satisfied. A **leftmost derivation** always expands the leftmost remaining nonterminal first. Top-down parsing can make guesses that the actual words cannot support, so it may need backtracking. A naive recursive-descent parser can loop on left-recursive rules such as NP→NP PP unless the grammar or algorithm is changed. Predictive LL(1) parsing chooses productions using one lookahead token when the grammar permits; not every English CFG is LL(1). | Approach | Begins with | Direction | Common difficulty | | --- | --- | --- | --- | | Top-down | Start symbol S | Root to word leaves | Predictions that do not match input; left recursion | | Bottom-up | Actual input tokens | Leaves to root S | Reductions that cannot contribute to a complete sentence | **Yaad rakho:** Top-down: sentence ka structure pehle, words tak neeche aao. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Three full top-down derivations with parse trees Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-parsing--top-down-examples/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-parsing/top-down-examples Unit 4, pages 98–116 **Question. Using S→NP VP, NP→Det N, VP→V NP, Det→the, N→girl|paper and V→read, give a leftmost top-down derivation and parse tree for “the girl read the paper”.** #### Given | Rule type | Rules | | --- | --- | | Structure | S → NP VP; NP → Det N; VP → V NP | | Lexicon | Det → the; N → girl | paper; V → read | #### To find A successful derivation and tree whose leaves equal the input. #### Formula and why it applies At every stage expand the leftmost nonterminal using a supplied production. A top-down derivation begins at S; the input determines which terminal alternatives can succeed. #### Solution ##### Step 1. Begin at the sentence symbol S ##### Step 2. Expand the sentence S ⇒ NP VP ##### Step 3. Expand its first noun phrase ⇒ Det N VP ##### Step 4. Match the first determiner ⇒ the N VP ##### Step 5. Match the subject noun ⇒ the girl VP ##### Step 6. Expand the verb phrase ⇒ the girl V NP ##### Step 7. Match the verb ⇒ the girl read NP ##### Step 8. Expand the object noun phrase ⇒ the girl read Det N ##### Step 9. Match its determiner ⇒ the girl read the N ##### Step 10. Match the last noun ⇒ the girl read the paper ##### Step 11. Draw the tree Parse tree: the girl read the paperSNPDettheNgirlVPVreadNPDettheNpaper Parse tree: the girl read the paper **Final answer.** The grammar derives the complete sentence. The subject NP is “the girl”; the object NP is “the paper”. **Question. Use S→NP VP, NP→Det N|N, VP→V NP, Det→A, N→restaurant|Dosas and V→serves to parse “A restaurant serves Dosas” by a leftmost top-down derivation.** #### Given | Rule type | Rules | | --- | --- | | Structure | S → NP VP; NP → Det N | N; VP → V NP | | Lexicon | Det → A; N → restaurant | Dosas; V → serves | #### To find A derivation and parse tree. #### Formula and why it applies Expand the leftmost nonterminal; choose NP→Det N for the subject and NP→N for the object. The object has no determiner, so forcing NP→Det N there would fail. #### Solution ##### Step 1. Start and expand S S ⇒ NP VP ##### Step 2. Choose the subject NP ⇒ Det N VP ##### Step 3. Match A ⇒ A N VP ##### Step 4. Match restaurant ⇒ A restaurant VP ##### Step 5. Expand the VP ⇒ A restaurant V NP ##### Step 6. Match serves ⇒ A restaurant serves NP ##### Step 7. Choose the object NP rule ⇒ A restaurant serves N ##### Step 8. Match Dosas ⇒ A restaurant serves Dosas ##### Step 9. Draw the successful tree Parse tree: A restaurant serves DosasSNPDetANrestaurantVPVservesNPNDosas Parse tree: A restaurant serves Dosas **Final answer.** The parse succeeds with subject NP→Det N and object NP→N. **Question. Using S→NP VP|VP, NP→Det N, VP→V NP, Det→that, N→book|flight and V→book, derive the imperative “book that flight” and explain the role of book.** #### Given | Rule type | Rules | | --- | --- | | Structure | S → NP VP | VP; NP → Det N; VP → V NP | | Lexicon | Det → that; N → book | flight; V → book | #### To find The successful derivation, tree and POS of book. #### Formula and why it applies Choose a production for S that can derive the actual input; an imperative may consist of VP. The grammar allows book as both noun and verb, but this imperative needs the verb reading. #### Solution ##### Step 1. Choose the imperative sentence rule S ⇒ VP ##### Step 2. Expand the verb phrase ⇒ V NP ##### Step 3. Choose the verb book ⇒ book NP ##### Step 4. Expand the object ⇒ book Det N ##### Step 5. Match that ⇒ book that N ##### Step 6. Match flight ⇒ book that flight ##### Step 7. Draw the tree Imperative parse: book that flightSVPVbookNPDetthatNflight Imperative parse: book that flight **Final answer.** book is V; that flight is NP; S→VP derives the imperative. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Shift-reduce bottom-up parsing: stack, input and actions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-parsing--shift-reduce-theory/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-parsing/shift-reduce-theory Unit 4, pages 98–116 **Bottom-up parsing** starts from the input tokens and combines them into larger constituents until the start symbol is reached. A **shift-reduce parser** maintains a stack of recognized material and a list of remaining input tokens. | Action | Exact operation | Example | | --- | --- | --- | | Shift | Move the next input token onto the stack | Stack: the; remaining input: dog runs | | Reduce | Replace a suffix of the stack matching a rule's RHS with its LHS | Det N becomes NP using NP→Det N | | Accept | Stack is S and input is exhausted | Stack: S; input: empty | | Fail/backtrack | Chosen reductions cannot complete a parse | A noun reading of book blocks an imperative parse | Reduction uses a production in reverse: the grammar NP→Det N expands an NP top-down, but reduces Det N into NP bottom-up. The stack's **rightmost item is its top** in the traces below. Each row shows the state **after** the listed action, avoiding an off-by-one mismatch between actions and stack contents. A lexical reduction such as N→dog still counts as a reduction. Do not skip the sequence from word to category to phrase when the question asks for a complete trace. Backtracking means returning to an earlier choice and trying another applicable rule, not changing the original input. **Yaad rakho:** Shift = input se stack. Reduce = chhote pieces se bada grammatical unit. ### M2 13 — Shift-reduce parsing and backtracking id : nlp-m2-13-bottom-up-shift-reduce title : M2 13 — Shift-reduce parsing and backtracking title\_confirmed : false topic\_title : Shift-reduce parsing and backtracking mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Three full shift-reduce traces Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-parsing--shift-reduce-examples/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-parsing/shift-reduce-examples Unit 4, pages 98–116 **Question. Show a complete shift-reduce parse of “the big dog sees the cat” using S→NP VP, NP→Det N|Det Adj N, VP→V NP, Det→the, Adj→big, N→dog|cat and V→sees.** #### Given Grammar is as specified. Stack starts empty; input is the big dog sees the cat. Top of stack is on the right. #### To find The stack, remaining input and action at every step. #### Formula and why it applies Shift tokens; reduce matched right-hand sides to their left-hand nonterminals. The first noun phrase includes an adjective and must use NP→Det Adj N. #### Solution ##### Step 1. Trace each action | Step / action | Stack after action | Remaining input | | --- | --- | --- | | 0. Start | ∅ | the big dog sees the cat | | 1. Shift the | the | big dog sees the cat | | 2. Reduce Det→the | Det | big dog sees the cat | | 3. Shift big | Det big | dog sees the cat | | 4. Reduce Adj→big | Det Adj | dog sees the cat | | 5. Shift dog | Det Adj dog | sees the cat | | 6. Reduce N→dog | Det Adj N | sees the cat | | 7. Reduce NP→Det Adj N | NP | sees the cat | | 8. Shift sees | NP sees | the cat | | 9. Reduce V→sees | NP V | the cat | | 10. Shift the | NP V the | cat | | 11. Reduce Det→the | NP V Det | cat | | 12. Shift cat | NP V Det cat | ∅ | | 13. Reduce N→cat | NP V Det N | ∅ | | 14. Reduce NP→Det N | NP V NP | ∅ | | 15. Reduce VP→V NP | NP VP | ∅ | | 16. Reduce S→NP VP | S | ∅ | ##### Step 2. Check acceptance Input is empty and stack is exactly S; therefore accept. ##### Step 3. Draw the resulting structure Parse tree: the big dog sees the catSNPDettheAdjbigNdogVPVseesNPDettheNcat Parse tree: the big dog sees the cat **Final answer.** Accepted. The parser builds both noun phrases, then VP, then S. **Question. Use S→NP VP, NP→Det N, VP→V NP, Det→the|this, N→man|book and V→read to give a shift-reduce trace for “the man read this book”.** #### Given The input has five tokens. Stack is empty initially. The lexical verb read is supplied by V→read. #### To find A complete bottom-up trace. #### Formula and why it applies Reduce only when the top stack sequence matches a supplied production's right side. Both noun phrases follow Det N, and the verb phrase follows V NP. #### Solution ##### Step 1. Trace all operations | Step / action | Stack after action | Remaining input | | --- | --- | --- | | 0. Start | ∅ | the man read this book | | 1. Shift the | the | man read this book | | 2. Reduce Det→the | Det | man read this book | | 3. Shift man | Det man | read this book | | 4. Reduce N→man | Det N | read this book | | 5. Reduce NP→Det N | NP | read this book | | 6. Shift read | NP read | this book | | 7. Reduce V→read | NP V | this book | | 8. Shift this | NP V this | book | | 9. Reduce Det→this | NP V Det | book | | 10. Shift book | NP V Det book | ∅ | | 11. Reduce N→book | NP V Det N | ∅ | | 12. Reduce NP→Det N | NP V NP | ∅ | | 13. Reduce VP→V NP | NP VP | ∅ | | 14. Reduce S→NP VP | S | ∅ | ##### Step 2. Verify the final state The complete sentence has reduced to S with no input left. **Final answer.** Accepted. Subject: the man. Predicate: read this book. Object: this book. **Question. Parse “book the flight” bottom-up using S→NP VP|VP, NP→Det N, VP→V NP, Det→the, N→book|flight, V→book. Explain why choosing the noun reading first can fail and show the successful alternative.** #### Given book has two lexical categories N and V. The input is an imperative. Stack starts empty. #### To find The failed choice and a complete successful trace with backtracking. #### Formula and why it applies Try lexical reductions allowed by the grammar; on a dead end, return to the ambiguous choice and try another. Grammar ambiguity requires a search choice. The noun reading cannot supply the required imperative verb. #### Solution ##### Step 1. Show the noun-first dead end Shift book and reduce it to N. Continue shifting/reducing the flight to Det N. Reducing Det N→NP leaves stack N NP and empty input. No supplied rule reduces N NP into S, so this branch fails. ##### Step 2. Return to the lexical choice Restore the state just after shifting book, then choose V→book instead of N→book. ##### Step 3. Trace the successful branch | Step / action | Stack after action | Remaining input | | --- | --- | --- | | 0. Restored after shifting book | book | the flight | | 1. Reduce V→book | V | the flight | | 2. Shift the | V the | flight | | 3. Reduce Det→the | V Det | flight | | 4. Shift flight | V Det flight | ∅ | | 5. Reduce N→flight | V Det N | ∅ | | 6. Reduce NP→Det N | V NP | ∅ | | 7. Reduce VP→V NP | VP | ∅ | | 8. Reduce S→VP | S | ∅ | ##### Step 4. Accept the alternative The verb reading produces S and consumes all tokens. **Final answer.** The noun-first branch fails; backtracking to book/V yields the successful imperative parse. **Yaad rakho:** Wrong branch fail hua toh sentence wrong nahi; next legal rule try karo. ### M2 13 — Shift-reduce parsing and backtracking id : nlp-m2-13-bottom-up-shift-reduce title : M2 13 — Shift-reduce parsing and backtracking title\_confirmed : false topic\_title : Shift-reduce parsing and backtracking mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Entity spans, categories and context Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-ner--ner-definition/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-ner/ner-definition Unit 4, pages 117–128 **Named Entity Recognition (NER)** identifies text spans referring to entities and assigns their types, such as PERSON, ORGANIZATION and LOCATION. Many practical schemas also include DATE, TIME, MONEY or domain-specific types. The exact allowed labels depend on the application's annotation scheme. NER has two parts: **boundary detection** finds where the entity starts and ends; **classification** assigns the appropriate type. A multiword name should normally be one entity span: “New York” is a location span, not two unrelated location decisions. Text: Mary joined Google in New York on Monday. → Mary → PERSON → Google → ORGANIZATION → New York → LOCATION → Monday → DATE Not every noun is a named entity. In “a developer works at Google”, developer is an ordinary job description while Google is a named organization. A specialized HR system may define a JOB\_TITLE category, but it must state that schema explicitly. NER identifies mentions; entity linking is the separate task of connecting a mention to a unique database identity. | Task | Question it answers | Example | | --- | --- | --- | | POS | What grammatical role does this token have? | Google/NNP | | Chunking | Which adjacent words form a phrase? | [NP a skilled developer] | | NER | Which span names an entity, and of what type? | [Google] ORGANIZATION | | Entity linking | Which real entity does this mention denote? | Connect Paris to the intended city or person record | **Applications.** NER structures search results and documents, extracts organizations/people from news, routes support tickets by product or organization, supports recommendation, and extracts specialized entities from healthcare or legal text. Context is essential: “Apple released a laptop” names an organization, while “she ate an apple” describes fruit. **Yaad rakho:** NER mein do kaam: naam kahan se kahan tak, aur kis category ka. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Dictionary, rules, machine learning and deep learning Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-ner--ner-approaches/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-ner/ner-approaches Unit 4, pages 117–128 | Approach | How it works | Example | Main limitation | | --- | --- | --- | --- | | Dictionary / gazetteer | Match text against lists of known names | A city list includes Mumbai and Pune | New names, spelling variants and ambiguous names are missed | | Rule-based | Use word shapes, regular expressions and context | A title such as Dr. followed by a name-like span suggests PERSON | Manual maintenance; exceptions and domain/language dependence | | Machine learning | Learn a labeling rule from annotated examples using features | Capitalization, suffixes, POS and surrounding words help a classifier/CRF | Requires labeled data and useful features | | Deep learning | Learn contextual word/character representations and predict labels | A contextual sequence encoder labels multiword entity spans | Training/domain dependence and resource demands | A **pattern-based rule** examines the form of text, such as an email pattern or a date format. A **context-based rule** examines neighboring words: “Dr.” can support a following PERSON span, while “headquartered in” can support a following LOCATION span. Capitalization alone is insufficient because sentence-initial common nouns are also capitalized. A dictionary may find “May” as a person's name, but “in May” may refer to a month. A learned model uses surrounding evidence to disambiguate. Hybrid systems can combine dictionaries, rules and learned models; these categories are practical tools rather than mutually exclusive restrictions. **Preprocessing for NER.** Tokenize while preserving token offsets and useful capitalization. Blindly removing capital letters or stopwords before recognition can damage names such as “Bank of India”. NER output should preserve the original span so the reader can locate it in the input. **Exam-ready 3-mark answer.** NER locates named-entity spans and classifies them into categories such as person, organization and location. A dictionary-based approach matches known name lists; a rule-based approach uses patterns or context; machine-learning and deep-learning approaches learn entity decisions from labeled data and representations. For example, in “Mary works at Google in Mumbai”, Mary is PERSON, Google is ORGANIZATION and Mumbai is LOCATION. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Three complete entity-recognition examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/m2-ner--ner-worked-examples/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/m2-ner/ner-worked-examples Unit 4, pages 117–128 **Question. Identify PERSON, ORGANIZATION and LOCATION entities in “Asha Rao joined Infosys in Bengaluru.”** #### Given Sentence: Asha Rao joined Infosys in Bengaluru. Allowed entity types: PERSON, ORGANIZATION, LOCATION. #### To find Complete entity spans and labels. #### Formula and why it applies Find the span first, then assign its type from context. A multiword personal name is a single entity; the verb and preposition remain outside entities. #### Solution ##### Step 1. Identify the person span Asha Rao names the person who joined, so take both words together. ##### Step 2. Identify the organization Infosys is the organization joined. ##### Step 3. Identify the location Bengaluru names the city. ##### Step 4. Write the output | Span | Label | | --- | --- | | Asha Rao | PERSON | | Infosys | ORGANIZATION | | Bengaluru | LOCATION | **Final answer.** [Asha Rao]PERSON joined [Infosys]ORGANIZATION in [Bengaluru]LOCATION. **Question. Using PERSON, ORGANIZATION, LOCATION and DATE, identify entities in “Dr. Meera Shah visited New Delhi on 5 October 2026.” Treat professional titles as outside the PERSON span for this exercise.** #### Given Allowed types and title-boundary convention are specified. Tokens include a multiword name, city and date. #### To find Entity boundaries and types. #### Formula and why it applies Boundary rules come from the annotation schema; use the stated title convention. Dr. supplies context for a person but is excluded from that span here. #### Solution ##### Step 1. Locate the person Meera Shah is the PERSON span; Dr. is outside by instruction. ##### Step 2. Locate the location New Delhi is one LOCATION span. ##### Step 3. Locate the date 5 October 2026 is one DATE span. ##### Step 4. Check organization mentions No organization is named in this sentence. **Final answer.** Meera Shah→PERSON; New Delhi→LOCATION; 5 October 2026→DATE. **Question. Explain the entity label of Apple in “Apple opened an office in Mumbai” and apple in “Ravi ate an apple”. Use PERSON, ORGANIZATION and LOCATION only.** #### Given Two contexts contain the same surface word apart from capitalization. The schema contains no FOOD category. #### To find Contextual labels and the ordinary noun case. #### Formula and why it applies NER labels a mention in context, not every occurrence of a dictionary string identically. A company can open an office; the object eaten in the second sentence is a fruit. #### Solution ##### Step 1. Analyze the first sentence Apple refers to the company, so label it ORGANIZATION. Mumbai is LOCATION. ##### Step 2. Analyze the second sentence Ravi is PERSON. apple is an ordinary fruit noun and receives no entity label under this schema. ##### Step 3. State the dictionary limitation Blindly labeling every apple as an organization would produce a false positive. **Final answer.** First: Apple→ORGANIZATION, Mumbai→LOCATION. Second: Ravi→PERSON; apple is outside the entity types. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Prepare a grammar for CYK / CKY Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u4-forward-cky--cnf/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u4-forward-cky/cnf Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. **Faculty topic; not an observed numerical PYQ.** Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. **CYK and CKY are two names for the same dynamic-programming parsing family.** A chart stores which grammatical categories can generate each contiguous span of the sentence. Small spans are solved first and reused to build larger spans. Basic CKY uses a context-free grammar in Chomsky normal form (CNF): a production has either two nonterminals on its right, A→B C, or one terminal, A→word. Empty-string handling is a separate special case; the examples here have no empty production. | Rule | Basic CKY status | Reason | | --- | --- | --- | | S → NP VP | Allowed | Two nonterminals | | N → cat | Allowed | One terminal | | NP → Det Adj N | Convert first | Three nonterminals | | NP → N | Eliminate or handle unary closure | One nonterminal, not a lexical terminal | | VP → eats NP | Convert first | Mixed terminal and nonterminal | **Question. Convert NP→Det Adj N, NP→N, N→cat and VP→eats NP into basic CNF-compatible productions. Preserve their possible word yields. [Authored worked example]** ### Answer #### Given The symbols NP,Det,Adj,N,VP are nonterminals; cat and eats are terminals. Other needed lexical rules remain as supplied. #### To find Equivalent binary or terminal productions for these cases. #### Formula and why it applies Introduce helper categories for a long right-hand side or a terminal mixed with nonterminals. Eliminate a unary nonterminal rule by inheriting the reachable non-unit productions. #### Solution ##### Step 1. Binarize the three-part NP Introduce X: NP→Det X and X→Adj N. X is a helper, not a word; the terminal order remains Det, Adj, N. ##### Step 2. Remove the unit production NP→N and N→cat allow NP to derive cat. Add NP→cat, retain N→cat where needed, and remove NP→N. With more rules reachable through unit chains, inherit every corresponding non-unit rule, not only the first one. ##### Step 3. Replace the mixed terminal Introduce Vlex→eats and replace VP→eats NP by VP→Vlex NP. ##### Step 4. Read the converted forms NP→Det X | cat; X→Adj N; N→cat; VP→Vlex NP; Vlex→eats. All displayed productions are binary nonterminal or single-terminal rules. **Final answer.** The converted rules preserve the original word sequences while satisfying the displayed basic CKY forms. Helper nodes change tree shape; they can be removed when presenting the original grammar’s tree. General CNF conversion may also require eliminating nullable productions and useless symbols while preserving the intended language. Do not claim that simply splitting long rules handles every arbitrary CFG. An Earley parser instead supports general CFG rules using prediction, scanning and completion; it combines top-down expectations with bottom-up evidence. LR and operator-precedence parsers are other methods named in the faculty list, with grammar restrictions of their own. This worked lesson was added after the video plan. No matching video has been generated or reviewed. --- ## TEE · CYK / CKY: complete charts, backpointers and ambiguity Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u4-forward-cky--cky/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u4-forward-cky/cky Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. **Faculty topic; not an observed numerical PYQ.** Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. Number the words 1 to n. Let **T[i,j]** contain all categories that derive the span from word i through word j, inclusive. Start with one-word cells from lexical rules. For every longer span try **every split k** between i and j: if B belongs to T[i,k], C belongs to T[k+1,j], and A→B C is a rule, put A in T[i,j]. Store the rule and split as backpointers when a tree is required. $$ A\in T[i,j]\quad\text{if}\quad A\to BC,\ B\in T[i,k],\ C\in T[k+1,j],\ i\le kj); ∅ is an evaluated but empty valid span. Read the chart by increasing span length, not left-to-right across a row. ##### Step 7. Follow the stored backpointers S[1,5]→NP[1,2] VP[3,5]; NP[1,2]→Det[1,1] N[2,2]; VP[3,5]→V[3,3] NP[4,5]; NP[4,5]→Det[4,4] N[5,5]. ``` S ├── NP │ ├── Det: the │ └── N: cat └── VP ├── V: eats └── NP ├── Det: a └── N: fish ``` **Final answer.** Accepted because S∈T[1,5]. The reconstructed tree’s leaves are exactly the five input words in order. **Question. A grammar has S→A A, A→A A and A→a. Use CKY for “a a a” and explain how two parse trees are preserved. [Authored worked example]** ### Answer #### Given Three identical terminal tokens. S is the start symbol. No unit or empty rules are present. #### To find Chart categories and both successful split choices for the root. #### Formula and why it applies A chart cell is a set of possible categories. For each category keep all successful backpointers when the task requires every parse. #### Solution ##### Step 1. Initialize the diagonal Each of T[1,1], T[2,2], T[3,3] is {A}, from A→a. ##### Step 2. Fill two-word spans T[1,2] and T[2,3] each contain {A,S}: A+A can use A→A A or S→A A. ##### Step 3. Try both root splits k=1: A[1,1] + A[2,3] permits S. k=2: A[1,2] + A[3,3] also permits S. The complete cell is {A,S}, but S has two distinct backpointer alternatives. ##### Step 4. Draw both bracketed trees [S [A a] [A [A a] [A a]]] and [S [A [A a] [A a]] [A a]]. Both yield a a a; the constituent grouping differs. **Final answer.** Accepted, with two S-rooted parses. A category appearing once in a set does not mean there is only one derivation. **Question. With S→NP VP, NP→Det N, VP→V NP; Det→the|a; N→cat|fish; V→eats, test “the eats cat” using CKY. [Authored worked example]** ### Answer #### Given Tokens are the/eats/cat, with no punctuation. Use exactly the supplied grammar. #### To find Whether the grammar accepts the whole input. #### Formula and why it applies A parse exists only if the complete chart cell contains the start symbol. #### Solution ##### Step 1. Initialize the diagonal T[1,1]={Det}; T[2,2]={V}; T[3,3]={N}. ##### Step 2. Try length-two combinations Det+V matches no binary rule; V+N also matches no rule (VP requires V+NP). Both two-word cells are empty. ##### Step 3. Try the two full-span splits Each split has an empty child, so T[1,3]=∅ and S is absent. **Final answer.** Rejected by this grammar. Do not invent NP→N to force acceptance. Grammar rejection and a universal judgment about every possible English grammar are different claims. **Yaad rakho:** Diagonal par words ke tags; phir chhote spans jodo. Har split dekho. Full sentence cell mein S aaya toh parse possible. This worked lesson was added after the video plan. No matching video has been generated or reviewed. --- ## TEE · Lexeme, lemma, sense and context Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-meaning--u5-words/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-meaning/u5-words Unit 5 · PDF pages 1–22 **Semantics is the study of meaning in language.** Lexical semantics studies meanings of words and relationships between their meanings. Sentence semantics studies how word meanings and sentence structure combine. A **word form** is what appears in the sentence. A **lexeme** is the underlying vocabulary item whose grammatical forms belong together. A **lemma** is the dictionary form used to name it. A **sense** is one particular meaning. | Term | Example | What to understand | | --- | --- | --- | | Word forms | run, runs, ran, running | Different grammatical forms | | Lexeme | RUN | The shared underlying vocabulary item | | Lemma | run | Dictionary heading | | Sense | run a race / run a business | Different meanings need context | | Lexicon | A language's vocabulary resource | Stores lexical items and information about them | The four forms *run, ran, runs, running* belong to the same lexeme. They are not four unrelated lexemes. “I deposited cash in the bank” and “We sat by the bank of the river” contain the same written form but different senses. word: bank → context: cash, deposit → sense: financial institution **Syntax** asks how words are arranged. **Semantics** asks what the expression means. **Pragmatics** uses situation and speaker intention: “Can you open the window?” is grammatically a question about ability but is normally a request. Form = jo likha; lemma = dictionary naam; sense = iss sentence mein matlab. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Every lexical relation with examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-meaning--u5-relations/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-meaning/u5-relations Unit 5 · PDF pages 1–22 **A lexical relation connects words or senses by a meaning relationship.** State the relation and its direction before giving the example. | Relation | Exam definition | Examples and direction | | --- | --- | --- | | Homonymy | The same form has unrelated meanings. | bank: financial institution / river edge; bat: animal / sports implement | | Polysemy | One lexical item has multiple related senses. | head: body part / head of a department; paper: material / article | | Synonymy | Different words have identical or closely similar meanings in a specified sense. | sofa / couch; automobile / car | | Antonymy | Words have opposite meanings. | hot / cold; alive / dead; buy / sell | | Hyponymy | A more specific concept is a type of a more general one. | rose is a flower; car is a vehicle | | Hypernymy | A broader concept includes a more specific one. | flower is a hypernym of rose | | Meronymy | A concept is a part of another concept. | wheel is part of car; finger is part of hand | | Holonymy | A concept is the whole containing a part. | car is a holonym of wheel | ### How to distinguish the commonly confused pairs For homonymy versus polysemy, ask whether the senses share an identifiable meaning. A river bank and a financial bank are unrelated. A financial bank and a blood bank share a repository/storage idea, illustrating polysemy in the supplied notes. For hyponymy versus meronymy, use the sentence test: **“X is a kind of Y”** indicates hyponymy; **“X is part of Y”** indicates meronymy. A wheel is not a type of car. | Expression | Correct relation | Reason | | --- | --- | --- | | sparrow → bird | Hyponymy | A sparrow is a kind of bird | | bird → sparrow | Hypernymy | Bird is the broader category | | keyboard → computer | Meronymy | Keyboard is a component | | computer → keyboard | Holonymy | Computer is the containing whole | | mouth of a person / mouth of a river | Polysemy | Both senses share the idea of an opening; the river use is a related extension | Synonyms are not always interchangeable in every context or register. “Big mistake” and “large mistake” can sound different even though big and large overlap in meaning. Antonyms can be gradable (hot/cold), complementary (alive/dead) or relational converses (buy/sell). HYPO = specific type; HYPER = bigger category. MERO = part; HOLO = whole. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · WordNet: synsets, glosses and hierarchy Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-meaning--u5-wordnet/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-meaning/u5-wordnet Unit 5 · PDF pages 1–22 **WordNet is a lexical database that groups word senses into synonym sets called synsets and links them through semantic and lexical relations.** It is a network of meanings, not merely a list of similar spellings. A synset represents one concept. Its **lemmas** are the words that can express that concept. A **gloss** is a short definition, often accompanied by an example. An ambiguous word can occur in several synsets. vehicle (general) → car / automobile (same concept) → sports car (specific) This is a simplified teaching hierarchy, not a claim about every intermediate WordNet node. Going upward reaches hypernyms; going downward reaches hyponyms. Multiple parents can exist, so the structure need not be a single simple tree. | Operation | Meaning | Illustration | | --- | --- | --- | | Get senses of a word | Find candidate synsets | bank → financial sense, river-edge sense, … | | Get lemmas | Find synonyms within one synset | car, automobile, motorcar | | Get definition/examples | Read the gloss and uses | Choose the intended concept | | Get hypernyms | Move to broader categories | car → vehicle | | Get hyponyms | Move to narrower categories | vehicle → particular vehicle categories | | Get meronyms/holonyms | Find part/whole links | car ↔ wheel | ### WordNet versus a thesaurus A thesaurus mainly groups related expressions for vocabulary lookup. WordNet explicitly distinguishes senses and labels relations such as hypernymy, hyponymy and meronymy. This supports WSD, query expansion, lexical similarity and semantic search. ### Three uses explained 1. **Search expansion:** a query for “automobile” can retrieve relevant documents using “car”. Expansion should use the right sense. 2. **WSD:** definitions of the senses of “bank” provide words that can be compared with the sentence context. 3. **Similarity:** concepts near one another in an “is-a” hierarchy tend to be semantically related. Path-based measures depend on which senses were selected. WordNet mein spelling ka group nahi—meaning ka synset, aur meanings ke beech labelled links. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · What WSD solves and how approaches differ Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-wsd--u5-wsd-basics/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-wsd/u5-wsd-basics Unit 5 · PDF pages 23–48 **Word sense disambiguation (WSD) is the task of selecting the intended sense of an ambiguous word in its context from a set of possible senses.** The output is a sense label, not a different replacement word. Input: “He grilled the bass.” → Candidate senses: fish / low sound → Context: grilled → Output: fish sense The **target word** is the ambiguous word. The **context** is the surrounding evidence. The **sense inventory** lists candidate meanings, often using WordNet. A **feature** is a usable clue: nearby word, position, part-of-speech tag or a phrase. | Approach | What it needs | How it decides | Limitation | | --- | --- | --- | --- | | Knowledge based | Dictionary, thesaurus or semantic graph | Definition overlap, category evidence or graph scores | Short definitions and missing knowledge | | Supervised | Examples already labelled with senses | Learns context → sense mapping | Labelling cost, limited rare-sense examples | | Semi-supervised | Small labelled set plus unlabelled data | Bootstraps from reliable seed examples | Early errors can spread | | Unsupervised sense induction | Unlabelled contextual occurrences | Clusters similar contexts | Clusters are not automatically named dictionary senses | | Neural/contextual | Learned word/context representations | Predicts or compares a context representation | Needs suitable data/model and evaluation | **Bag-of-words features** record nearby words while ignoring exact positions. **Collocational features** preserve a positional phrase such as “river bank”, “bank account”, “chocolate bar” or “bar exam”. ### Why WSD matters Machine translation needs the correct target-language meaning; information retrieval should distinguish river-bank travel from banking services; text-to-speech may need different pronunciations for “bass”; information extraction should avoid connecting entities through the wrong relation. WSD: same spelling ke options mein context dekh ke sahi meaning select. ### M2 16 WSD and Lesk Scoring id : nlp-m2-16-lesk-wsd title : M2 16 WSD and Lesk Scoring title\_confirmed : true topic\_title : Word-sense disambiguation and Lesk overlap scoring mode : m2 kind : focused status : available duration : 10:17 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Lesk algorithm, from words to overlap scores Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-wsd--u5-lesk/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-wsd/u5-lesk Unit 5 · PDF pages 23–48 **Lesk is a knowledge-based WSD method that chooses senses whose dictionary descriptions have the greatest overlap with contextual evidence.** Original Lesk compares definitions of neighbouring words' senses. A common simplified version compares each target-sense gloss directly with the sentence context. State which version you use. ### Simplified Lesk procedure 1. Remove the target word from the context; tokenize and normalize the remaining content words. 2. Obtain the gloss for each candidate sense and apply the same normalization. 3. Count shared words between the context and each gloss. 4. Select the sense with maximum overlap. If scores tie, report the tie or use a specified tie-break rule. $$ \operatorname{score}(s)=|C\cap G_s|,\qquad s^*=\arg\max_s\operatorname{score}(s) $$ $$ C $$ is the set of context words, $$ G_s $$ the set of words in sense $$ s $$ 's gloss, $$ \cap $$ means common elements, $$ |\cdot| $$ means count, and $$ \arg\max $$ means “the sense giving the largest score”. This version counts unique matching words once. **Question. Use simplified Lesk to disambiguate “bank” in “The river water rose near the bank.” Use C = {river, water, rose, near}; financial gloss = {institution, money, deposit}; river gloss = {land, river, water, edge}.** #### Given The normalized context and two gloss sets are given. Count exact shared words. #### To find The score of each sense and the selected meaning. #### Formula and why it applies Use $$ \operatorname{score}(s)=|C\cap G_s| $$ because the task specifies context–gloss overlap. #### Step 1. Compare the financial sense No word occurs in both {river, water, rose, near} and {institution, money, deposit}. Overlap = {}. #### Step 2. Count the financial score $$ \operatorname{score}(F)=0 $$ . #### Step 3. Compare the river sense Shared words are {river, water}. “Rose” and “near” are not in the gloss. #### Step 4. Count and select $$ \operatorname{score}(R)=2 $$ . Since $$ 2>0 $$ , choose the river-edge sense. **Answer.** Bank means land beside a river. **Question. Use simplified Lesk for “The dog barked loudly.” Treat barked as bark. Context C = {dog, loudly}; sound gloss = {sharp, cry, dog}; tree gloss = {outer, covering, tree}; ship gloss = {sailing, ship}.** #### Given Three glosses and normalized context {dog, loudly}. #### To find Select the sense of bark. #### Formula and why it applies $$ \operatorname{score}(s)=|C\cap G_s| $$ . Lemmatization lets barked match the target lemma bark; it does not create additional overlap words. #### Step 1. Sound sense Intersection = {dog}; score = 1. #### Step 2. Tree sense Intersection = {}; score = 0. #### Step 3. Ship sense Intersection = {}; score = 0. #### Step 4. Compare The sound sense has the unique maximum: 1 > 0. **Answer.** Bark is the dog's cry. **Question. Apply definition–definition Lesk to “pine cone”. Pine senses: P1 = {evergreen, tree, needle, leaf}; P2 = {waste, sorrow, illness}. Cone senses: C1 = {solid, shape, circular}; C2 = {fruit, evergreen, tree}.** #### Given Two candidate senses for each word. Tokens are already normalized; tree and trees would be reduced to the same form if necessary. #### To find The most compatible pair of senses. #### Formula and why it applies Use $$ |G_{P_i}\cap G_{C_j}| $$ for every pair because this is original-style gloss–gloss comparison. #### Step 1. P1 with C1 No shared word → 0. #### Step 2. P1 with C2 Shared {evergreen, tree} → 2. #### Step 3. P2 with each cone sense P2–C1 has no overlap → 0; P2–C2 has no overlap → 0. #### Step 4. Choose the best pair The four scores are 0, 2, 0, 0. Maximum 2 belongs to P1–C2. **Answer.** Pine is the evergreen tree; cone is its fruit-like reproductive structure. **Limitations:** exact overlap misses synonyms, short glosses may produce zeros, and different preprocessing choices change the score. Extended Lesk may include related synset definitions or longer phrase overlaps; do not silently substitute it for a simple question. Lesk: context aur dictionary mein common words gino; sabse bada overlap jeeta. ### M2 16 WSD and Lesk Scoring id : nlp-m2-16-lesk-wsd title : M2 16 WSD and Lesk Scoring title\_confirmed : true topic\_title : Word-sense disambiguation and Lesk overlap scoring mode : m2 kind : focused status : available duration : 10:17 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Thesaurus / Walker approach, with three complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-wsd--u5-walker/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-wsd/u5-walker Unit 5 · PDF pages 23–48 **The Walker approach uses thesaurus categories as semantic evidence.** Associate each candidate sense with a category. Count or weight context words that support that category. Choose the largest total. $$ \operatorname{score}(s)=\sum_{w\in C} I(w,\operatorname{category}(s)) $$ $$ I=1 $$ if the supplied category mapping says the context word supports the sense, otherwise $$ I=0 $$ . The summation symbol means add all those contributions. Weighted questions may replace 1 with a supplied weight. Category membership must be supplied or justified; it is not guessed from word frequency. **Question. Disambiguate bank using Walker scoring. Context words are money, interest, annum, fetch. The finance category supports money, interest and annum; the location category supports none of these words.** #### Given Two categories: Finance and Location. Each supported word contributes 1. #### To find The two totals and selected bank sense. #### Formula and why it applies Sum the category indicators for each sense. #### Step 1. Make a contribution table | Word | Finance | Location | | --- | --- | --- | | money | 1 | 0 | | interest | 1 | 0 | | annum | 1 | 0 | | fetch | 0 | 0 | #### Step 2. Add Finance $$ 1+1+1+0=3 $$ . #### Step 3. Add Location $$ 0+0+0+0=0 $$ . #### Step 4. Select 3 > 0, so select the financial institution. **Answer.** Finance, with score 3 versus 0. **Question. Use Walker scoring for “The fisherman caught bass beside the lake.” Context {fisherman,caught,lake}. The Fish category contains all three clues; the Music category contains none.** #### Given Fish and Music senses, binary category membership given. #### To find The correct bass sense. #### Formula and why it applies Score = sum of context-category matches. #### Step 1. Record Fish matches fisherman → 1; caught → 1; lake → 1. #### Step 2. Add Fish score $$ 1+1+1=3 $$ . #### Step 3. Add Music score $$ 0+0+0=0 $$ . #### Step 4. Choose Fish has the larger score. **Answer.** Bass refers to a fish. **Question. Score “The player used a bat in the match.” Context {player,match}. Sport supports both words with weights 2 and 1; Animal supports neither. Use the supplied weighted Walker scheme.** #### Given Sport weights {player:2, match:1}; Animal weights {player:0, match:0}. #### To find Weighted sense scores. #### Formula and why it applies Use the sum of supplied weights, not an unweighted count, because the question explicitly gives weights. #### Step 1. Sport contribution of player Add 2. #### Step 2. Sport contribution of match Add 1. Sport total = 2 + 1 = 3. #### Step 3. Animal total 0 + 0 = 0. #### Step 4. Compare 3 > 0 selects the sporting implement. **Answer.** Bat is the sports implement. Walker: dictionary ke exact words nahi, context kis topic/category ko support karta hai woh count. ### M2 17 — Walker category scoring and graph-based WSD id : nlp-m2-17-walker-graph-wsd title : M2 17 — Walker category scoring and graph-based WSD title\_confirmed : false topic\_title : Walker category scoring and graph-based WSD mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Random-walk WSD and supervised alternatives Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-wsd--u5-graph/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-wsd/u5-graph Unit 5 · PDF pages 23–48 **Graph-based WSD represents candidate senses as vertices and semantic relationships as weighted edges.** A random walk repeatedly transfers score along these edges. Senses supported by other strongly connected context senses tend to obtain higher ranks. 1. Create a vertex for every candidate sense of every relevant word. 2. Connect semantically related senses; similarity or gloss overlap can set edge weights. 3. Normalize outgoing edge weights into transition probabilities. 4. Run a graph-ranking method such as PageRank. 5. For each target word, choose its highest-ranked candidate sense. church context → bell: instrument sense ↔ ring: sound sense → mutually supported senses receive rank One common rank update is $$ r_{t+1}(j)=\frac{1-d}{N}+d\sum_i r_t(i)P(i\rightarrow j). $$ $$ r_t(i) $$ is current score of vertex $$ i $$ , $$ P(i\to j) $$ is its outgoing probability to $$ j $$ , $$ N $$ is the number of vertices, and $$ d $$ is the probability of following a graph edge. The remaining probability $$ 1-d $$ gives a uniform restart here. A personalized variant uses a nonuniform restart. The course requires the graph method concept; numerical details depend on supplied ranking rules. **Question. A two-vertex toy sense graph has transition matrix rows A:[0.2,0.8], B:[0.4,0.6]. Starting scores are A=0.5, B=0.5. Find one random-walk update with no restart.** #### Given Rows denote source vertices, columns destination vertices. d = 1. #### To find New scores at A and B. #### Formula and why it applies Use $$ r_{t+1}(j)=\sum_i r_t(i)P(i\to j) $$ : incoming contributions are added. #### Step 1. Contribution to A from A $$ 0.5\times0.2=0.1 $$ . #### Step 2. Contribution to A from B $$ 0.5\times0.4=0.2 $$ , hence $$ r_1(A)=0.1+0.2=0.3 $$ . #### Step 3. Contribution to B from A and B $$ 0.5\times0.8=0.4 $$ ; $$ 0.5\times0.6=0.3 $$ . #### Step 4. Add B and verify total $$ r_1(B)=0.4+0.3=0.7 $$ ; $$ 0.3+0.7=1 $$ . **Answer.** A=0.3, B=0.7 after one update. This is one iteration, not a converged PageRank result. ### Other approaches explained without assuming ML knowledge **A supervised classifier** learns from examples such as “cash … bank” labelled FINANCE and “river … bank” labelled RIVER. A decision list applies ranked context rules; a decision tree branches on clues; an SVM separates feature representations; a neural model learns contextual patterns. Naive Bayes, developed next, multiplies class-specific context probabilities. **Semi-supervised bootstrapping** begins with a few labelled examples or high-confidence rules, labels suitable unlabelled cases, and retrains. **Unsupervised clustering** groups usages by context similarity without gold sense labels; an additional mapping is needed to name the discovered groups. Graph WSD: meanings ko jodo; jo meaning context ke strong meanings se supported hai uska rank badhega. ### M2 17 — Walker category scoring and graph-based WSD id : nlp-m2-17-walker-graph-wsd title : M2 17 — Walker category scoring and graph-based WSD title\_confirmed : false topic\_title : Walker category scoring and graph-based WSD mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Prior, likelihood and posterior—before the formula Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-naive-bayes--u5-bayes-meaning/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-naive-bayes/u5-bayes-meaning Unit 5 · PDF pages 49–57 **A probability is a number from 0 to 1 measuring how likely an event is under a stated model.** A class probability of $$ 3/6=0.5 $$ means 3 of 6 training examples belong to that class. **Conditional probability** means probability inside a restricted group. $$ P(\text{loan}\mid F) $$ reads “probability of the word loan given the finance sense”. The vertical bar means “given”; it is not division. | Symbol | Name | Plain meaning | | --- | --- | --- | | $$ P(F) $$ | Prior | How common finance is before reading the new context | | $$ P(C\mid F) $$ | Likelihood | How compatible the observed context is with finance | | $$ P(F\mid C) $$ | Posterior | How likely finance is after seeing the context | | $$ P(C) $$ | Evidence | Total probability of this context under all senses | $$ P(F\mid C)=\frac{P(C\mid F)P(F)}{P(C)}. $$ Bayes' theorem reverses a conditional direction using the prior. Do not swap $$ P(F\mid C) $$ and $$ P(C\mid F) $$ : one asks about the sense, the other about the evidence. The **naive conditional independence assumption** says that, once the sense is fixed, the context-word features can be multiplied as if they were independent. It is an approximation: natural-language words are not actually independent in general. Training examples → count classes → Prior per sense → Count words within each sense → Likelihood of query clues → Multiply → compare → normalize if asked Prior = pehle kitna common; likelihood = clue kitna fit; posterior = clue dekhne ke baad meaning ki chance. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Multinomial Naive Bayes: the exact solving method Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-naive-bayes--u5-bayes-steps/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-naive-bayes/u5-bayes-steps Unit 5 · PDF pages 49–57 Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. ### Always keep these counts separate Class priors use **number of examples**. Word likelihoods in this multinomial model use **number of word occurrences**. The vocabulary counts **unique words across all classes**. 1. List the class examples and calculate priors. 2. Write one shared vocabulary and count its size. 3. Count total word occurrences in each class. 4. Count each query word in each class. 5. Apply the given smoothing rule. 6. Multiply each prior by all its query likelihoods. 7. Compare scores; divide each score by their sum only if a normalized posterior is requested. $$ P(s\mid C)=\frac{\operatorname{score}(s)}{\sum_{s'}\operatorname{score}(s')}. $$ **Why smoothing is needed:** an unseen clue gives $$ 0/N_s=0 $$ ; one zero makes the whole product zero. Add-one smoothing redistributes probability across every vocabulary word and prevents these zero factors. Use it only when requested, taught for the task, or state the assumption. **Different model warning:** a Bernoulli Naive Bayes question counts presence/absence per document and may include absence factors. Do not use its denominator in a multinomial word-count question. Pehle sentences se prior; phir words se likelihood; dono counts mix mat kar. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Mam's bank numerical—every count and every multiplication Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-naive-bayes--u5-bayes-mam/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-naive-bayes/u5-bayes-mam Unit 5 · PDF pages 49–57 **Question. Classify the sense of “bank” using the training table below. The sentence is: “As the stream flowed gently past the shore, we set up camp by the bank of the river, discussing the possibility of borrowing a loan to buy a boat for future trips.” Use only the highlighted feature words stream, shore and loan, and add-one multinomial smoothing.** #### Given | Training row | Context words | Sense | | --- | --- | --- | | 1 | money, deposit, loan | Finance | | 2 | river, water, shore | Riverbank | | 3 | account, savings, investment | Finance | | 4 | boat, river, stream | Riverbank | | 5 | branch, deposit, check | Finance | | 6 | fishing, shore, river | Riverbank | Query feature set: {stream, shore, loan}. Other sentence words are not included as features in this supplied example. #### To find The most probable bank sense and normalized posteriors. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Calculate priors Finance appears in 3 of the 6 rows; Riverbank appears in 3 of 6. $$ P(F)=\frac36=\frac12,\qquad P(R)=\frac36=\frac12. $$ #### Step 2. Construct the vocabulary {money, deposit, loan, river, water, shore, account, savings, investment, boat, stream, fishing, branch, check}. Thus V=14. Repeated deposit or river is counted once in vocabulary. #### Step 3. Count word occurrences per class There are 3 finance rows × 3 words = 9 word occurrences. There are 3 river rows × 3 words = 9. Thus $$ N_F=N_R=9 $$ , and each smoothed denominator is $$ 9+14=23 $$ . #### Step 4. Count the three query words | Word | Finance count | Riverbank count | | --- | --- | --- | | stream | 0 | 1 | | shore | 0 | 2 | | loan | 1 | 0 | #### Step 5. Calculate finance likelihoods $$ P(\text{stream}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{shore}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{loan}\mid F)=\frac{1+1}{23}=\frac2{23} $$ #### Step 6. Multiply the finance score $$ \operatorname{score}(F)=\frac12\times\frac1{23}\times\frac1{23}\times\frac2{23}=\frac1{23^3}=\frac1{12167}\approx0.00008219. $$ #### Step 7. Calculate river likelihoods $$ P(\text{stream}\mid R)=\frac{1+1}{23}=\frac2{23} $$ $$ P(\text{shore}\mid R)=\frac{2+1}{23}=\frac3{23} $$ $$ P(\text{loan}\mid R)=\frac{0+1}{23}=\frac1{23} $$ #### Step 8. Multiply the river score $$ \operatorname{score}(R)=\frac12\times\frac2{23}\times\frac3{23}\times\frac1{23}=\frac3{12167}\approx0.00024657. $$ #### Step 9. Compare scores 3/12167 is larger than 1/12167. Riverbank wins even though “loan” is financial evidence: the combined clues matter. #### Step 10. Normalize if a posterior is requested $$ P(F\mid C)=\frac{1/12167}{4/12167}=\frac14=0.25 $$ $$ P(R\mid C)=\frac{3/12167}{4/12167}=\frac34=0.75. $$ **Answer.** Bank is the riverbank sense. Scores are 1/12167 and 3/12167; normalized model posteriors are 0.25 and 0.75. **Without smoothing:** stream and shore are absent in Finance; loan is absent in Riverbank. Both unsmoothed scores become zero, so there is no valid winning class from those products. Also, unnormalized scores should not be labelled as final posterior probabilities. Zero word? Add-one: count+1 upar, total words+vocabulary neeche. Last mein score bada wala sense. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · More WSD numericals: given likelihoods, unequal priors and repeated clues Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-naive-bayes--u5-bayes-practice/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-naive-bayes/u5-bayes-practice Unit 5 · PDF pages 49–57 **Question. For bass, let P(Fish)=0.6 and P(Music)=0.4. Given Fish likelihoods P(caught|Fish)=0.5, P(lake|Fish)=0.4 and Music likelihoods P(caught|Music)=0.05, P(lake|Music)=0.1, classify context {caught,lake} and normalize.** #### Given Priors and likelihoods are provided directly; no smoothing or training-count estimation is requested. #### To find The winning sense and its posterior. #### Formula and why it applies $$ \operatorname{score}(s)=P(s)P(\text{caught}\mid s)P(\text{lake}\mid s) $$ , because the two features are treated as conditionally independent. #### Step 1. Fish multiplication $$ 0.6\times0.5=0.3 $$ , then $$ 0.3\times0.4=0.12 $$ . #### Step 2. Music multiplication $$ 0.4\times0.05=0.02 $$ , then $$ 0.02\times0.1=0.002 $$ . #### Step 3. Total evidence $$ 0.12+0.002=0.122 $$ . #### Step 4. Normalize $$ P(Fish\mid C)=0.12/0.122\approx0.983607 $$ $$ P(Music\mid C)=0.002/0.122\approx0.016393. $$ **Answer.** Fish, with model posterior approximately 0.983607. **Question. Disambiguate bank from the repeated context tokens {loan, loan, river}. Priors are Finance=2/3 and River=1/3. Vocabulary V=4. Finance has N=8 total tokens, loan count 3 and river count 0; River has N=4, loan count 0 and river count 2. Apply add-one smoothing.** #### Given Multinomial counts are supplied. The two appearances of loan must both contribute. #### To find Scores and normalized Finance posterior. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Finance denominator $$ 8+4=12 $$ . $$ P(loan\mid F)=(3+1)/12=1/3 $$ ; $$ P(river\mid F)=(0+1)/12=1/12 $$ . #### Step 2. River denominator $$ 4+4=8 $$ . $$ P(loan\mid R)=(0+1)/8=1/8 $$ ; $$ P(river\mid R)=(2+1)/8=3/8 $$ . #### Step 3. Finance product $$ \operatorname{score}(F)=\frac23\times\frac13\times\frac13\times\frac1{12}=\frac1{162}\approx0.00617284. $$ #### Step 4. River product $$ \operatorname{score}(R)=\frac13\times\frac18\times\frac18\times\frac38=\frac1{512}\approx0.001953125. $$ #### Step 5. Compare and normalize $$ P(F\mid C)=\frac{1/162}{1/162+1/512}=\frac{512}{674}=\frac{256}{337}\approx0.759644. $$ **Answer.** Finance wins. Repeated loan contributes a squared likelihood; do not reduce the query to a unique-word set. ### Answer-writing order in an exam Write the class priors, vocabulary size, class token totals, a likelihood table, both score products and the final class. Keep fractions until the comparison where possible. If the question only asks classification, normalization is optional; if it asks posterior probabilities, normalization is required. Repeated word aaye toh factor bhi repeat. Sirf word-set bana diya toh multinomial answer badal jayega. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Distributional hypothesis and a co-occurrence matrix Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-vectors--u5-distribution/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-vectors/u5-distribution TEE · Unit 5 PDF pages 58–118 **The distributional hypothesis states that words occurring in similar contexts tend to have similar meanings or semantic behaviour.** “Cat” and “dog” often occur near pet, feed and animal. Their context patterns can therefore be represented and compared numerically. A **vector** is an ordered list of numbers. Each position must have a fixed meaning. For context order [pet, food, road], cat=[4,3,0] means four co-occurrences with pet, three with food, none with road. A **matrix** stacks vectors as rows in a table. A **word–context co-occurrence matrix** records how often a target word occurs with each context word under a specified context definition. A window may be a sentence, document, or a fixed number of neighbouring tokens. State its width, direction and whether targets count themselves. Choose target words → Choose context/window → Count co-occurrences → Store each row as a vector → Compare vectors **Question. Build the symmetric co-occurrence matrix for “cats eat fish” using a window of one token on either side, no self-counts and vocabulary [cats,eat,fish].** #### Given One sentence, one occurrence of each token. Left and right adjacent tokens are contexts. #### To find A 3×3 target-by-context count matrix. #### Formula and why it applies $$ X_{ij} $$ counts the number of times context word $$ j $$ occurs within the chosen window of target $$ i $$ . #### Step 1. Target cats Only eat is adjacent. Row cats = [0,1,0]. #### Step 2. Target eat cats is left and fish is right. Row eat = [1,0,1]. #### Step 3. Target fish Only eat is adjacent. Row fish = [0,1,0]. #### Step 4. Assemble the matrix | Target / context | cats | eat | fish | | --- | --- | --- | --- | | cats | 0 | 1 | 0 | | eat | 1 | 0 | 1 | | fish | 0 | 1 | 0 | **Answer.** Rows are [0,1,0], [1,0,1], [0,1,0]. The symmetry follows from the symmetric window. **Question. Build a target–context matrix for sentences “cats eat fish” and “dogs eat fish”. Target words are cats and dogs; context columns [eat,fish]; use the entire sentence excluding the target.** #### Given Each sentence contains one target. Count context token occurrences. #### To find Two context vectors. #### Formula and why it applies Count each context word in every sentence containing the target. #### Step 1. Count cats contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 2. Count dogs contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 3. Compare Both targets have identical observed context patterns in this very small corpus. **Answer.** cats=[1,1], dogs=[1,1]. This corpus supplies no context evidence to distinguish them. **Question. For “a b a”, use a right-only window of one token. Build the matrix with rows/columns [a,b].** #### Given Count directed target→next-word events, not both directions. #### To find The co-occurrence matrix. #### Formula and why it applies $$ X_{ij}=C(\text{target }i\text{ immediately followed by }j) $$ . #### Step 1. First target a Next token is b → add 1 to X(a,b). #### Step 2. Target b Next token is a → add 1 to X(b,a). #### Step 3. Final a No following token → no addition. #### Step 4. Write matrix | Target / next | a | b | | --- | --- | --- | | a | 0 | 1 | | b | 1 | 0 | **Answer.** a=[0,1], b=[1,0]. A different context window would produce a different matrix. Word ka meaning: uske aas-paas kaunse words kitni baar aaye, uski number-list. ### TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors id : tee-09 title : TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors title\_confirmed : false topic\_title : Distributional semantics, co-occurrence matrices and sentence vectors mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Sparse, dense, distributional and distributed Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-vectors--u5-representation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-vectors/u5-representation TEE · Unit 5 PDF pages 58–118 **A sparse representation** has many zero entries and often one coordinate per vocabulary item. BoW and TF–IDF are common sparse document representations. **A dense embedding** has comparatively few learned real-valued coordinates; one word's properties are spread across many coordinates. | Representation | What coordinates mean | Strength | Limitation | | --- | --- | --- | --- | | One-hot | One identity position per vocabulary word | Unambiguous ID | Different words are orthogonal; no learned similarity | | Count-based context vector | Context-word co-occurrence counts | Transparent corpus evidence | Large, sparse; frequent words may dominate | | TF–IDF document vector | Weighted vocabulary terms | Useful retrieval weighting | Word order and contextual meaning are limited | | Dense word embedding | Learned latent features | Compact semantic/syntactic relationships | Coordinates are not generally human-labelled features | **Distributional** describes learning from patterns of linguistic context. **Distributed** describes representing an item across multiple coordinates rather than a single active identity bit. These are related but different ideas. A one-hot ID alone does not learn distributional similarity. ### Model design choices Choose contexts (document, sentence, fixed window), weighting (raw counts, normalization, TF–IDF-like weighting or association scores), dimensionality reduction or learned embeddings, and a similarity measure. Cosine compares direction and is less affected by length than a raw dot product; Euclidean distance also reflects magnitude. Sparse = bahut zero. Dense = chhoti learned number-list. Similar context se useful similarity seekhte hain. ### TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors id : tee-09 title : TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors title\_confirmed : false topic\_title : Distributional semantics, co-occurrence matrices and sentence vectors mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Word2Vec: CBOW and skip-gram from training pairs Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-vectors--u5-word2vec/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-vectors/u5-word2vec TEE · Unit 5 PDF pages 58–118 **Word2Vec learns dense word embeddings by predicting words from nearby context.** Its two common architectures are CBOW and skip-gram. A vocabulary ID or one-hot vector is the input representation; the learned embedding table converts that identity into a short trainable vector. | Architecture | Input | Prediction | Memory cue | | --- | --- | --- | --- | | CBOW | Surrounding context words | Centre/target word | Context → centre | | Skip-gram | Centre/target word | Surrounding words | Centre → context | CBOW: the + sat → Average context embeddings → Predict: cat Skip-gram: cat → Embedding of cat → Predict: the and sat For vocabulary size $$ V $$ and embedding dimension $$ d $$ , an input one-hot row is $$ 1\times V $$ ; the input embedding matrix is $$ V\times d $$ ; the hidden row is $$ 1\times d $$ ; the output matrix is $$ d\times V $$ ; output scores are $$ 1\times V $$ . Multiplying a one-hot row by the embedding matrix simply selects one row. $$ P(w_o\mid w_i)=\frac{\exp(u_{w_o}^{T}v_{w_i})}{\sum_{w\in V}\exp(u_w^{T}v_{w_i})}. $$ $$ v_{w_i} $$ is the input word embedding, $$ u_w $$ an output-context embedding, and the dot product is a compatibility score. Softmax turns all scores into probabilities that sum to one. Training increases the probability of observed context words. Negative sampling or hierarchical softmax can reduce computation in larger vocabularies; use the algorithm specified by the question. **Question. For the sentence “the cat sat on mat”, generate CBOW and skip-gram training data for centre word sat with a window of one token on each side.** #### Given Tokens indexed 1 to 5: the, cat, sat, on, mat. Centre = sat. #### To find The CBOW example and the skip-gram pairs. #### Formula and why it applies CBOW predicts the centre from its neighbours; skip-gram predicts each neighbour from the centre. #### Step 1. Identify neighbours Left neighbour = cat; right neighbour = on. “the” and “mat” are outside this window. #### Step 2. CBOW example Input [cat,on] → target sat. #### Step 3. Skip-gram examples (sat,cat) and (sat,on), with first item as input and second as predicted context. **Answer.** CBOW: [cat,on]→sat. Skip-gram: sat→cat and sat→on. **Question. The embeddings of context words cat and on are [1,2] and [3,0]. Compute the CBOW hidden vector using mean aggregation. Output vectors for [sat,ran] are [1,0] and [0,1]. Compute softmax probabilities.** #### Given Context embeddings [1,2], [3,0]; mean aggregation; two candidate output vectors. #### To find The hidden vector, logits and probabilities. #### Formula and why it applies Mean context vector $$ h=(v_1+v_2)/2 $$ . Logit $$ z_w=u_w^Th $$ . Softmax $$ P(w)=e^{z_w}/\sum_j e^{z_j} $$ . #### Step 1. Add coordinates $$ [1+3,\,2+0]=[4,2] $$ . #### Step 2. Divide by number of words $$ h=[4/2,2/2]=[2,1] $$ . #### Step 3. Compute logits $$ z_{sat}=1(2)+0(1)=2 $$ . $$ z_{ran}=0(2)+1(1)=1 $$ . #### Step 4. Exponentials $$ e^2\approx7.389056 $$ , $$ e^1\approx2.718282 $$ ; sum $$ \approx10.107338 $$ . #### Step 5. Divide $$ P(sat)\approx0.731059 $$ , $$ P(ran)\approx0.268941 $$ . **Answer.** h=[2,1]; sat is more probable (0.731059). **Question. A skip-gram centre embedding is v=[1,−1]. Output embeddings for [eat,sleep,drive] are [1,1], [0,1], [1,0]. Calculate the predicted context probabilities.** #### Given One centre vector and three output vectors; full softmax over these three words. #### To find Three logits and three probabilities. #### Formula and why it applies $$ z_j=u_j^Tv $$ , then $$ P(j)=e^{z_j}/\sum_k e^{z_k} $$ . #### Step 1. Eat dot product $$ 1(1)+1(-1)=0 $$ . #### Step 2. Sleep and drive dot products $$ z_{sleep}=0(1)+1(-1)=-1 $$ . $$ z_{drive}=1(1)+0(-1)=1 $$ . #### Step 3. Exponentiate and sum $$ e^0=1 $$ , $$ e^{-1}\approx0.367879 $$ , $$ e^1\approx2.718282 $$ . Sum $$ \approx4.086161 $$ . #### Step 4. Normalize $$ P(eat)\approx0.244728 $$ ; $$ P(sleep)\approx0.090031 $$ ; $$ P(drive)\approx0.665241 $$ . **Answer.** drive has the highest probability in this toy learned model. **Limitations:** a standard static Word2Vec vector merges a word's senses; it has limited representation of subword morphology and out-of-vocabulary words; its local prediction objective does not directly fit a full global co-occurrence table. Similarity is learned from data and is not a guarantee of factual or causal understanding. CBOW = neighbours se missing centre. Skip-gram = centre se neighbours. ### TEE 10 — Word2Vec: CBOW and skip-gram, training pairs and softmax numericals id : tee-10 title : TEE 10 — Word2Vec: CBOW and skip-gram, training pairs and softmax numericals title\_confirmed : false topic\_title : Word2Vec: CBOW and skip-gram, training pairs and softmax numericals mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Word analogies and sentence embeddings Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-vectors--u5-sentence/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-vectors/u5-sentence TEE · Unit 5 PDF pages 58–118 **A sentence embedding** represents an entire sentence by one fixed-length vector. A basic method averages word embeddings. More advanced encoders learn a sentence representation from context and task objectives. $$ v_{\text{sentence}}=\frac1n\sum_{i=1}^{n}v_{w_i}. $$ This means add corresponding coordinates and divide each coordinate by the number of words. Averaging ignores order: “dog bites man” and “man bites dog” have the same average if their words are identical. **Question. Average the word vectors [1,0], [0,2] and [2,1] to represent a three-word sentence.** #### Given Three two-dimensional vectors with equal weights. #### To find The two-dimensional sentence vector. #### Formula and why it applies Use coordinate-wise mean $$ v=(v_1+v_2+v_3)/3 $$ . #### Step 1. First coordinate $$ 1+0+2=3 $$ ; $$ 3/3=1 $$ . #### Step 2. Second coordinate $$ 0+2+1=3 $$ ; $$ 3/3=1 $$ . **Answer.** Sentence vector = [1,1]. **Question. Toy word vectors are king=[3,2], man=[1,0], woman=[1,1]. Find king−man+woman and compare to queen=[3,3].** #### Given All vectors use the same two-coordinate space. #### To find The analogy vector and its distance from queen. #### Formula and why it applies Vector arithmetic is coordinate by coordinate. Euclidean distance is the square root of the sum of squared coordinate differences. #### Step 1. Subtract man $$ [3-1,2-0]=[2,2] $$ . #### Step 2. Add woman $$ [2+1,2+1]=[3,3] $$ . #### Step 3. Compare with queen $$ \sqrt{(3-3)^2+(3-3)^2}=0 $$ . **Answer.** The toy analogy exactly equals queen. Real learned embeddings only approximate some analogies. **Contextual encoders** can assign “bank” different representations in different sentences. BERT and RoBERTa produce contextual token representations; obtaining strong sentence similarity vectors requires an appropriate pooling/training method, not simply assuming every raw token vector is a sentence embedding. Universal Sentence Encoder is designed to encode sentence-level meaning. Skip-thought predicts surrounding sentences; InferSent learns from natural-language-inference supervision. Sentence average: har column add karo, words ki count se divide. Word order ka loss yaad rakh. ### TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors id : tee-09 title : TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors title\_confirmed : false topic\_title : Distributional semantics, co-occurrence matrices and sentence vectors mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Why GloVe uses global statistics Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-glove--u5-glove-theory/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-glove/u5-glove-theory TEE · Unit 5 PDF pages 119–130 **GloVe (Global Vectors) learns word embeddings by fitting weighted relationships in a global word–context co-occurrence matrix.** Word2Vec uses local prediction examples; GloVe uses aggregate counts directly in its objective. $$ X_i=\sum_k X_{ik},\qquad P_{ik}=\frac{X_{ik}}{X_i}. $$ $$ X_{ik} $$ is how often context word $$ k $$ appears around target $$ i $$ . $$ X_i $$ is the total of row $$ i $$ . $$ P_{ik} $$ is the resulting context probability. The ratio $$ P(k\mid i)/P(k\mid j) $$ helps identify how a probe context differs between two words. For targets ice and steam, a probe such as solid should favour ice; gas should favour steam; a probe shared similarly by both gives a ratio near one. These are data tendencies, not universal numerical identities. **Question. For context solid, ice has count 12 out of row total 20; steam has count 2 out of row total 40. Compute P(solid|ice), P(solid|steam) and their ratio.** #### Given X(ice,solid)=12, X(ice)=20; X(steam,solid)=2, X(steam)=40. #### To find Two conditional probabilities and the ice/steam ratio. #### Formula and why it applies $$ P(k\mid i)=X_{ik}/X_i $$ , and $$ R=P(k\mid ice)/P(k\mid steam) $$ . #### Step 1. Ice probability $$ 12/20=0.6 $$ . #### Step 2. Steam probability $$ 2/40=0.05 $$ . #### Step 3. Ratio $$ 0.6/0.05=12 $$ . **Answer.** The probe solid is 12 times as probable in ice's context under these counts. **Question. For probe water, ice and steam both have row total 50. Counts are 10 and 10. Calculate the ratio.** #### Given Both probabilities use a denominator of 50. #### To find The probability ratio. #### Formula and why it applies Use row-normalized counts before taking the ratio. #### Step 1. Ice probability $$ 10/50=0.2 $$ . #### Step 2. Steam probability $$ 10/50=0.2 $$ . #### Step 3. Divide $$ 0.2/0.2=1 $$ . **Answer.** Ratio 1: this probe does not distinguish the targets in this toy data. **Question. For probe gas, counts are 1/100 around ice and 20/100 around steam. Compute the ice-to-steam ratio.** #### Given Conditional counts have the same row denominator. #### To find The ratio and interpretation. #### Formula and why it applies $$ R=P(gas\mid ice)/P(gas\mid steam) $$ . #### Step 1. Ice probability $$ 1/100=0.01 $$ . #### Step 2. Steam probability $$ 20/100=0.2 $$ . #### Step 3. Ratio $$ 0.01/0.2=0.05 $$ . **Answer.** Ratio 0.05, so gas is much more strongly associated with steam. Ratio >1: pehla word favoured. <1: doosra. ≈1: probe dono mein similar. ### GloVe: global counts, probability ratios and weighted loss id : tee-11 title : GloVe: global counts, probability ratios and weighted loss title\_confirmed : false topic\_title : GloVe: global counts, probability ratios and weighted loss mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · The GloVe objective and numerical interpretation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u5-glove--u5-glove-loss/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u5-glove/u5-glove-loss TEE · Unit 5 PDF pages 119–130 $$ J=\sum_{i,j:X_{ij}>0}f(X_{ij})\left(w_i^T\widetilde w_j+b_i+\widetilde b_j-\ln X_{ij}\right)^2. $$ **What each piece means:** $$ w_i $$ is a target vector; $$ \widetilde w_j $$ is a separate context vector; the dot product estimates their compatibility; biases adjust word-level baselines. The desired score is the natural log of the observed count. Squaring penalizes both positive and negative errors. Weighting prevents unreliable rare pairs or extremely frequent pairs from dominating inappropriately. $$ f(x)=\begin{cases}(x/x_{\max})^\alpha,&0 youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Reference, pragmatics and discourse Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u6--reference/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u6/reference TEE only · Unit 6 · syllabus-based supplement; faculty slide pack not supplied **Pragmatic analysis** interprets an utterance using its situation and the speaker’s intended meaning. **Discourse analysis** studies how utterances connect across a text or conversation. Syntax and literal word meanings alone cannot settle every interpretation. | Text | Literal content | Contextual interpretation | | --- | --- | --- | | “Can you close the window?” | A question about ability. | Usually a polite request. | | “Meera bought a laptop. It was expensive.” | Two sentences. | “It” connects the second sentence to the laptop. | | “Meet me here tomorrow.” | An instruction. | Me, here and tomorrow depend on speaker, place and date. | ### Reference, mention and antecedent A **referring expression** points to an entity; a **mention** is its occurrence in the text. Its **referent** is the entity being discussed. An **antecedent** is a preceding expression used to interpret a later referring expression. Reference resolution identifies the intended referent. Coreference resolution groups mentions that refer to the same entity. “Meera”name mention → Person entitysame referent → “she” / “the engineer”other mentions | Concept | Definition | Example | | --- | --- | --- | | Anaphora | Reference back to something introduced earlier. | “Riya arrived. She sat down.” She → Riya. | | Cataphora | Reference forward to a later expression. | “Before she spoke, Riya checked the microphone.” she → Riya. | | Exophora | Reference to the situation outside the text. | “Put it there”, said while pointing. | | Coreference | Two or more mentions refer to the same entity. | “Dr Rao”, “the lecturer”, “she”. | | Pronoun resolution | Find the referent of a pronoun. | Resolve “it” in “The laptop stopped because it overheated.” | Coreference is broader than pronoun resolution: repeated names and descriptions also form a chain. Anaphora describes direction/dependence; coreference describes identity. In “I bought a car. The engine is noisy”, engine is related to the car but is not the same entity. **Yaad rakho:** Reference = kiski baat? Coreference = alag mentions, same entity. Pronoun resolution = he/she/it kis se linked? ### Cohesion, coherence and discourse markers **Cohesion** is the explicit linguistic linking of a text through reference, repeated or related words, conjunctions and other devices. In “Rita bought a book. She read it”, she and it connect the second sentence to Rita and the book. **Coherence** is the conceptual unity that makes a discourse meaningful. “The road flooded, so the match was cancelled” presents a plausible cause and consequence. A reader relates the propositions using context and knowledge. **Discourse markers** signal relationships or organize speech: however signals contrast; therefore signals a result; first and next signal sequence; for example introduces an illustration. These concepts differ: “I like apples. Therefore the moon is square” has a linking word but no sensible inference. A connector alone does not guarantee coherence. **Yaad rakho:** Cohesion = words se link; coherence = meaning se link; marker = relation ka signal. ### Reference and coreference: mentions, antecedents and pronoun resolution id : tee-12 title : Reference and coreference: mentions, antecedents and pronoun resolution title\_confirmed : false topic\_title : Reference and coreference: mentions, antecedents and pronoun resolution mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Pronoun resolution: methods and examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u6--resolution/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u6/resolution TEE only · Unit 6 · syllabus-based supplement; faculty slide pack not supplied ### A practical resolution procedure First detect the pronoun and candidate noun phrases. Eliminate candidates inconsistent with person, number, grammatical binding or relevant semantic constraints. Rank the surviving candidates using syntactic prominence, recency and discourse context. Preserve ambiguity if the available evidence does not decide. | Approach | How it works | Strength | Limitation | | --- | --- | --- | --- | | Rule based / syntactic | Use parse trees, agreement and explicit search order; Hobbs is a classic example. | Can explain each decision without labelled training data. | Parsing errors and world knowledge gaps cause errors. | | Salience based | Score candidates using recency, subjecthood and other discourse features; select the strongest compatible candidate. | Captures the tendency to refer again to prominent entities. | Feature weights and competing candidates can mislead. | | Supervised statistical | Learn candidate/pronoun compatibility from labelled examples. | Combines many features from data. | Needs representative annotated examples. | | Neural / contextual | Encode the surrounding text and score mention pairs or antecedents. | Can learn contextual patterns and long-range dependencies. | Needs data/compute and can still make confident mistakes. | ### Worked analysis 1 · number and semantics Resolve “they” in: “Riya put the books on the shelf because they were heavy.” **Given** Candidates: Riya (singular person), books (plural objects), shelf (singular object); pronoun they is plural here. **To find** The antecedent and a justified resolution procedure. ### Formula and why it applies Use number agreement first, then semantic compatibility. There is no multiplication formula in this linguistic task. ### Solution #### Step 1. List candidates Riya, the books, the shelf. #### Step 2. Apply number agreement Riya and shelf are singular. Books is plural, matching they. #### Step 3. Check meaning Books can be heavy; the causal explanation fits carrying/placing books. “They” refers to “the books”. ### Worked analysis 2 · a complete coreference chain Identify the coreference chain in: “Dr Sen entered. The surgeon greeted Ravi. She opened her notebook.” **Given** Mentions: Dr Sen, the surgeon, Ravi, she, her, her notebook. Assume the surgeon is Dr Sen and she/her refers to that surgeon. **To find** The person chain, and a mention that must not be placed in it. ### Formula and why it applies Group expressions by identity of referent; a possessive expression and the owned object are different entities. ### Solution #### Step 1. Link the name and description Dr Sen and the surgeon denote the same person under the given context. #### Step 2. Link the pronouns She and her refer to the surgeon. #### Step 3. Keep the notebook separate “Her notebook” is an object owned by the person; it is not the person herself. Person chain: {Dr Sen, the surgeon, She, her}. “Her notebook” is a separate object mention. ### Worked analysis 3 · honest ambiguity Discuss the pronoun in: “Asha called Nisha after she finished work.” **Given** Both named candidates are singular; the sentence gives no further context. **To find** Whether one referent is logically guaranteed. ### Formula and why it applies Agreement filters incompatible candidates but cannot always select between equally compatible candidates. ### Solution #### Step 1. Check agreement Both Asha and Nisha are compatible with she. #### Step 2. Check the event Either person could have finished work. #### Step 3. State the limitation A heuristic may prefer a candidate, but grammar and this sentence alone do not prove the intended referent. Ambiguous without further context. State any heuristic and parse assumption before giving its preferred candidate. ### Reference and coreference: mentions, antecedents and pronoun resolution id : tee-12 title : Reference and coreference: mentions, antecedents and pronoun resolution title\_confirmed : false topic\_title : Reference and coreference: mentions, antecedents and pronoun resolution mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Hobbs’ algorithm, step by step Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u6--hobbs/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u6/hobbs TEE only · Unit 6 · syllabus-based supplement; faculty slide pack not supplied **Hobbs’ algorithm is a syntax-based procedure that searches constituent parse trees in a fixed order for a compatible noun-phrase antecedent of a pronoun.** The tree determines which candidates are proposed first. Agreement and other relevant constraints determine whether a proposed candidate can be accepted. It is not simply the rule “choose the closest noun”. A nearby noun may have the wrong number, may be syntactically ineligible, or may be visited after a structurally more prominent noun. The algorithm produces a preference under the supplied parse; ambiguous language can still require context. ### The tree vocabulary you need | Term | Meaning | Example | | --- | --- | --- | | NP | Noun phrase: a phrase referring to an entity | the student; Anita; they | | S | A sentence/clause node | the node containing a subject and predicate | | VP | Verb phrase | told Ravi that she should revise | | X | The NP or S node currently being searched | An algorithm label, not a word | | Path p | The branch travelled upward from the pronoun/previous X | Used to distinguish left and right subtrees | | N-bar (N′) | The noun-centred part of an NP in a grammar using that label | The nominal-child test depends on the given tree | | Breadth-first | Visit shallower nodes before deeper ones | Level 1 before level 2 | | Left-to-right | Visit equally deep nodes in their written/tree order | Earlier left branch before right branch | Locate pronoun NP → Climb to NP/S → Propose candidates in tree order → Check compatibility → Accept or continue searching ### The complete search procedure #### Step 1. Start at the pronoun Locate the NP immediately dominating the pronoun. “Dominating” means containing it as a descendant in the tree. #### Step 2. Climb to the first NP or S Move upward to the first NP or S above that starting NP. Call this node X. Keep the upward path p. #### Step 3. Search the left branches initially Below X, traverse branches to the left of p breadth-first and left-to-right. At this initial search, propose an NP only if another NP or S lies between that candidate and X. Test proposed candidates for the required agreement and constraints. #### Step 4. If X is the highest sentence node, inspect earlier sentences Search previous sentence trees from most recent to oldest. Within each, use breadth-first, left-to-right order to propose NPs. Accept the first eligible compatible candidate, or report that no candidate was found in the available context. #### Step 5. Otherwise climb again Move from X to the next NP or S above it. Record this new X and the path followed during the climb. #### Step 6. Apply the higher-NP test If the new X is an NP, propose X when the path did not pass through its nominal N-bar child. Use the given parse labels; do not invent an N-bar branch missing from the question's structure. #### Step 7. Search its left branches Traverse branches below the new X to the left of the path, breadth-first and left-to-right, proposing NPs. The special intervening-NP/S restriction belongs to the initial search in Step 3. #### Step 8. If X is S, search the permitted right branches Search branches to the right of the path breadth-first and left-to-right. Do not descend below an encountered NP or S. Propose eligible NP candidates there. #### Step 9. Repeat or stop Return to the highest-S check and continue the climb/search procedure until a compatible candidate is accepted or the available trees are exhausted. Record the candidate order rather than only writing an unexplained name. ### Why the first left search has an extra condition In a simple clause such as “John saw him”, the subject NP John is directly under the clause S. There is no extra NP or S between John and that S. The initial-search restriction therefore prevents casually selecting John as the ordinary local antecedent of him. Compare the reflexive “John saw himself”, which is a different grammatical case; do not treat every pronoun type as interchangeable. ### Worked tree 1 · move to the previous sentence **Question. Apply Hobbs’ search to “The student arrived. She smiled.” Assume the student is compatible with she.** #### Given Previous tree: [S [NP The student] [VP arrived]]. Current tree: [S [NP She] [VP smiled]]. #### To find The selected antecedent and the search path. #### Method and why it applies Use the supplied constituent trees. The current sentence has no earlier candidate, so the highest-S rule activates previous-sentence search. ### Solution #### Step 1. Draw both sentence structures ``` Previous sentence Current sentence S S ├── NP: The student ├── NP: She ← start └── VP: arrived └── VP: smiled ``` #### Step 2. Locate and climb Start at NP She. Its first higher NP/S is the current S. Set X to that S. #### Step 3. Search left of the path She is the leftmost subject. No NP lies to its left in the current sentence. #### Step 4. Use the highest-S rule This X is the highest S. Inspect the immediately preceding sentence tree. #### Step 5. Propose its first NP Breadth-first, left-to-right finds NP The student. #### Step 6. Check compatibility and accept The question stipulates that the student agrees with she. Accept this candidate. **Answer.** She refers to the student under the supplied assumptions. ### Worked tree 2 · reject an incompatible subject **Question. Resolve they in “Riya spoke to the students. They laughed.” Use a previous-sentence search and interpret they as plural.** #### Given Previous tree has subject NP Riya, then VP spoke [PP to [NP the students]]. Current subject is NP They. #### To find The candidate order and accepted antecedent. #### Method and why it applies Breadth-first search favours a shallow subject, but agreement must still be applied; a first candidate is not automatically a valid answer. ### Solution #### Step 1. Draw the useful part of the previous tree ``` S ├── NP: Riya ← candidate 1 └── VP ├── V: spoke └── PP ├── P: to └── NP: the students ← candidate 2 ``` #### Step 2. Search the current sentence From NP They climb to its S; there is no earlier NP in “They laughed”. This is the highest S, so move to the previous tree. #### Step 3. Visit Riya first Riya is the shallower leftmost NP. Under the stated plural interpretation, this singular candidate fails number agreement. #### Step 4. Visit the students Continue breadth-first. The next relevant NP is the students. #### Step 5. Apply number agreement Students is plural and is an eligible candidate. It matches plural they. #### Step 6. State the resolution Accept the students. The method did not simply accept the first noun it saw. **Answer.** They → the students. Candidate order: Riya rejected; the students accepted. ### Worked tree 3 · climb out of an embedded clause **Question. Resolve she in “Anita told Ravi that she should revise.” Assume Anita is compatible with she and Ravi is not. Use the displayed parse.** #### Given The pronoun is the subject of an embedded S inside the main VP. #### To find The preferred antecedent found by the climb and left-branch search. #### Method and why it applies The embedded clause has no left candidate. Climbing to the main S exposes its subject Anita as the first suitable higher-level candidate. ### Solution #### Step 1. Show the clause structure ``` S (main) ├── NP: Anita ← accepted candidate └── VP ├── V: told ├── NP: Ravi └── SBAR: that └── S (embedded) ← first X ├── NP: she ← start └── VP: should revise ``` #### Step 2. Start inside the embedded clause From NP she climb to the embedded S. There is no candidate to the left of she within this embedded S. #### Step 3. Climb to the next NP/S The embedded S is not the highest S. Move upward through SBAR and VP to the main S. These intermediate nodes are not NP/S search nodes. #### Step 4. Search left of the upward path At the main S, the subject NP Anita is a shallow left candidate. The object Ravi is deeper within the VP. Breadth-first order proposes Anita first. #### Step 5. Check the stated compatibility Anita is compatible with she by the question's assumption. Accept it; no need to choose a later incompatible candidate. #### Step 6. Give the result and its limit This is the algorithm's result under the displayed parse and agreement assumptions. It does not prove the intention of every speaker using an otherwise ambiguous sentence. **Answer.** She → Anita under the stated tree and compatibility information. ### Advantages, limitations and a complete exam answer | Advantages | Limitations | | --- | --- | | Explicit candidate order makes the decision traceable. | Wrong parse structure can produce the wrong search order. | | Requires no labelled coreference-training corpus for its basic rules. | World knowledge and discourse intention may override syntactic preference. | | Uses structure rather than only surface distance. | Some ambiguous, long-distance or unusual constructions need additional treatment. | For a longer answer, write the definition, inputs, tree notation, ordered algorithm, one labelled tree with a candidate table, final antecedent, and limitations. In a worked question show separately: candidate → why proposed → agreement result → accept/reject. Do not equate the selected candidate with a logically guaranteed interpretation when the sentence is ambiguous. **Yaad rakho:** Tree order candidate deta hai; agreement usko allow/reject karta hai. Nearest noun automatically answer nahi. ### Hobbs algorithm: parse-tree traversal with three complete traces id : tee-13 title : Hobbs algorithm: parse-tree traversal with three complete traces title\_confirmed : false topic\_title : Hobbs algorithm: parse-tree traversal with three complete traces mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Text classification and its full pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--classification/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/classification TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied **Text classification** assigns a document to one or more predefined categories. Spam detection usually uses two classes; news classification may use several topics. In single-label multiclass classification, choose one of several categories. In multilabel classification, several labels may apply to the same document. Define labels → Prepare labelled examples → Train representation + model → Evaluate on unseen text → Predict new labels ### An exam-ready pipeline | Stage | What you do | Why it matters | | --- | --- | --- | | Task definition | Decide what counts as spam or which news topics exist. | Unclear labels create inconsistent training data. | | Data collection and annotation | Collect representative examples and assign correct labels. | The model learns from these examples. | | Split data | Separate training, validation and test data. | Test examples must not influence training choices. | | Preprocess | Tokenize and normalise where suitable; retain meaningful negation. | Reduce irrelevant variation without deleting signal. | | Represent | Use BoW, TF-IDF or learned embeddings. | Models need numerical features. | | Train | Fit Naive Bayes, logistic regression or another appropriate model. | Learn how features predict categories. | | Evaluate | Measure relevant precision, recall and F1 on held-out examples. | Accuracy alone may hide poor minority-class performance. | | Deploy and monitor | Apply the same preprocessing and monitor changing language. | Incoming data can differ from old examples. | Naive Bayes is especially useful for hand calculations. With a bag-of-words representation, its class score is the prior multiplied by the conditional probability of each token; repeated tokens repeat the factor. The “naive” assumption is conditional independence of features given the class, not that words are independent in real language. $$ s(c)=P(c)\prod_{w\in V}P(w\mid c)^{C(w,d)} $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ C(w,d) counts word w in document d. C(w,c) counts it in training documents belonging to class c. Nc is the total token count for class c. Add-one smoothing adds one to every vocabulary count and therefore adds |V| to the denominator. This is the same arithmetic taught for WSD, with document classes instead of senses. ### Worked classification · two-token message A spam classifier has vocabulary {win, prize, meeting}. Priors are P(spam)=P(ham)=0.5. Spam token counts are [3,2,0]; ham counts are [0,0,5]. With add-one smoothing, classify “win prize”. **Given** |V|=3; N(spam)=5; N(ham)=5; message has win once and prize once. **To find** Class scores and predicted label. ### Formula and why it applies $$ s(c)=P(c)P(\text{win}\mid c)P(\text{prize}\mid c) $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ The model is multinomial Naive Bayes; multiply the two token likelihoods and the class prior. ### Solution #### Step 1. Compute the smoothed denominators $$ 5+3=8 $$ #### Step 2. Compute spam likelihoods $$ P(\text{win}\mid S)=\frac{3+1}{8}=\frac12 $$ $$ P(\text{prize}\mid S)=\frac{2+1}{8}=\frac38 $$ #### Step 3. Compute ham likelihoods $$ P(\text{win}\mid H)=P(\text{prize}\mid H)=\frac{0+1}{8}=\frac18 $$ #### Step 4. Multiply the spam score $$ s(S)=\frac12\times\frac12\times\frac38=\frac{3}{32}=0.09375 $$ #### Step 5. Multiply the ham score $$ s(H)=\frac12\times\frac18\times\frac18=\frac{1}{128}=0.0078125 $$ #### Step 6. Compare 0.09375 > 0.0078125. These are unnormalised scores; the label is chosen by the larger score. Predicted label: spam. **Yaad rakho:** Class prior × har word ki class-likelihood; sabse bada score wins. ### Text classification, sentiment and evaluation numericals id : tee-14 title : Text classification, sentiment and evaluation numericals title\_confirmed : false topic\_title : Text classification, sentiment and evaluation numericals mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Sentiment analysis: meaning, methods, example Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--sentiment/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/sentiment TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied **Sentiment analysis** identifies expressed opinion or attitude, often as positive, negative or neutral. Document-level analysis labels a complete text; sentence-level analysis labels a sentence; aspect-level analysis attaches sentiment to a specific feature or entity. | Review | Useful interpretation | | --- | --- | | “The display is excellent, but the battery is poor.” | Display: positive; battery: negative. One overall label loses this distinction. | | “The service is not good.” | Negative: negation changes good. Removing not can reverse the intended meaning. | | “Great, another two-hour delay.” | Likely sarcasm: literal positive word alone is insufficient. | ### Approaches **Lexicon based:** look up sentiment scores, combine them, and apply stated handling for negation/intensifiers. It is interpretable but fragile with context and sarcasm. **Supervised:** learn a mapping from labelled text features to sentiment labels. **Neural/contextual:** learn or use contextual representations to capture word interactions, then fine-tune a sentiment classifier. None guarantees correct interpretation of every review. ### A specified lexicon calculation Use good = +2, excellent = +3 and slow = −2. A “not” immediately before a sentiment word reverses its score. Score “The support was not good but the delivery was excellent and slow.” **Given** Three sentiment words; not modifies good only; sum word scores under the stated rule. **To find** Total sentiment score and label, taking positive totals as positive. ### Formula and why it applies $$ S=\sum_i s_i $$ Apply the given negation rule before summing. This is a simplified specified model, not a universal linguistic rule. ### Solution #### Step 1. Identify sentiment words good, excellent and slow. #### Step 2. Apply negation $$ s(\text{not good})=-2 $$ #### Step 3. Add the contributions $$ S=-2+3-2=-1 $$ #### Step 4. Map score to label The sum is negative. Total score = −1; negative under this rule. ### Text classification, sentiment and evaluation numericals id : tee-14 title : Text classification, sentiment and evaluation numericals title\_confirmed : false topic\_title : Text classification, sentiment and evaluation numericals mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Precision, recall, F1 and accuracy Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--evaluation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/evaluation TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied Evaluation compares predicted results with known answers. A **true positive** is a correctly predicted positive; a **false positive** is a negative item predicted positive; a **false negative** is a missed positive; a **true negative** is a correctly rejected negative. $$ \text{Precision}=\frac{TP}{TP+FP} $$ $$ \text{Recall}=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN} $$ $$ \text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} $$ Precision asks how many predicted positives were correct. Recall asks how many actual positives were found. F1 balances them by a harmonic mean. If a denominator is zero, state the convention rather than dividing by zero. ### Example 1 · evaluate a classifier A binary text classifier gives TP=6, FP=2, FN=4, TN=8. Compute precision, recall, F1 and accuracy. **Given** TP=6; FP=2; FN=4; TN=8. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{6}{6+2}=0.750000 $$ #### Step 2. Recall $$ R=\frac{6}{6+4}=0.600000 $$ #### Step 3. F1 $$ F_1=\frac{12}{12+2+4}=0.666667 $$ #### Step 4. Accuracy $$ A=\frac{6+8}{6+2+4+8}=0.700000 $$ Precision 0.7500; recall 0.6000; F1 0.6667; accuracy 0.7000. ### Example 2 · evaluate a classifier A binary text classifier gives TP=9, FP=3, FN=1, TN=7. Compute precision, recall, F1 and accuracy. **Given** TP=9; FP=3; FN=1; TN=7. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{9}{9+3}=0.750000 $$ #### Step 2. Recall $$ R=\frac{9}{9+1}=0.900000 $$ #### Step 3. F1 $$ F_1=\frac{18}{18+3+1}=0.818182 $$ #### Step 4. Accuracy $$ A=\frac{9+7}{9+3+1+7}=0.800000 $$ Precision 0.7500; recall 0.9000; F1 0.8182; accuracy 0.8000. ### Example 3 · evaluate a classifier A binary text classifier gives TP=4, FP=0, FN=4, TN=12. Compute precision, recall, F1 and accuracy. **Given** TP=4; FP=0; FN=4; TN=12. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{4}{4+0}=1.000000 $$ #### Step 2. Recall $$ R=\frac{4}{4+4}=0.500000 $$ #### Step 3. F1 $$ F_1=\frac{8}{8+0+4}=0.666667 $$ #### Step 4. Accuracy $$ A=\frac{4+12}{4+0+4+12}=0.800000 $$ Precision 1.0000; recall 0.5000; F1 0.6667; accuracy 0.8000. ### Text classification, sentiment and evaluation numericals id : tee-14 title : Text classification, sentiment and evaluation numericals title\_confirmed : false topic\_title : Text classification, sentiment and evaluation numericals mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Spelling correction and edit distance Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--spelling/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/spelling TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied **Spelling correction** proposes the intended word or sentence when the observed text contains an error. A non-word error produces an invalid vocabulary item (“recieve”); a real-word error produces an existing word in the wrong context (“I will meat you”). Dictionary lookup can detect many non-words but cannot reliably detect real-word errors. Detect suspicious text → Generate candidates → Score edit + context fit → Choose correction ### Minimum edit distance Levenshtein distance is the minimum total cost of insertions, deletions and substitutions needed to transform one string into another. In the examples below, each edit costs 1; matching characters cost 0. Transposition is not a single Levenshtein operation. State different costs explicitly if the question gives them. $$ D(i,j)=\min\begin{cases}D(i-1,j)+1\\D(i,j-1)+1\\D(i-1,j-1)+[a_i\ne b_j]\end{cases} $$ Rows represent prefixes of the source; columns represent prefixes of the target. D(i,j) is the best cost for transforming the first i source characters to the first j target characters. Initialise D(i,0)=i and D(0,j)=j. The three choices are deletion, insertion and diagonal match/substitution. ### Example 1 · cat → cut Calculate the unit-cost Levenshtein distance from “cat” to “cut”. Show the dynamic-programming table. **Given** Source=cat; target=cut; insertion=deletion=substitution=1; match=0. **To find** Minimum edit cost. ### Formula and why it applies $$ D(i,j)=\min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+c) $$ c = 0 for matching characters, 1 otherwise. Compare all three possible ways to reach each cell. ### Solution #### Step 1. Initialise empty prefixes First row counts insertions; first column counts deletions. #### Step 2. Fill row 1: source character “c” $$ D(1,1)=\min(1+1,\ 1+1,\ 0+0)=0 $$ $$ D(1,2)=\min(2+1,\ 0+1,\ 1+1)=1 $$ $$ D(1,3)=\min(3+1,\ 1+1,\ 2+1)=2 $$ #### Step 3. Fill row 2: source character “a” $$ D(2,1)=\min(0+1,\ 2+1,\ 1+1)=1 $$ $$ D(2,2)=\min(1+1,\ 1+1,\ 0+1)=1 $$ $$ D(2,3)=\min(2+1,\ 1+1,\ 1+1)=2 $$ #### Step 4. Fill row 3: source character “t” $$ D(3,1)=\min(1+1,\ 3+1,\ 2+1)=2 $$ $$ D(3,2)=\min(1+1,\ 2+1,\ 1+1)=2 $$ $$ D(3,3)=\min(2+1,\ 2+1,\ 1+0)=1 $$ #### Step 5. Read the completed matrix | Source / target | ∅ | c | u | t | | --- | --- | --- | --- | --- | | ∅ | 0 | 1 | 2 | 3 | | c | 1 | 0 | 1 | 2 | | a | 2 | 1 | 1 | 2 | | t | 3 | 2 | 2 | 1 | Minimum edit distance = 1. ### Example 2 · at → cat Calculate the unit-cost Levenshtein distance from “at” to “cat”. Show the dynamic-programming table. **Given** Source=at; target=cat; insertion=deletion=substitution=1; match=0. **To find** Minimum edit cost. ### Formula and why it applies $$ D(i,j)=\min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+c) $$ c = 0 for matching characters, 1 otherwise. Compare all three possible ways to reach each cell. ### Solution #### Step 1. Initialise empty prefixes First row counts insertions; first column counts deletions. #### Step 2. Fill row 1: source character “a” $$ D(1,1)=\min(1+1,\ 1+1,\ 0+1)=1 $$ $$ D(1,2)=\min(2+1,\ 1+1,\ 1+0)=1 $$ $$ D(1,3)=\min(3+1,\ 1+1,\ 2+1)=2 $$ #### Step 3. Fill row 2: source character “t” $$ D(2,1)=\min(1+1,\ 2+1,\ 1+1)=2 $$ $$ D(2,2)=\min(1+1,\ 2+1,\ 1+1)=2 $$ $$ D(2,3)=\min(2+1,\ 2+1,\ 1+0)=1 $$ #### Step 4. Read the completed matrix | Source / target | ∅ | c | a | t | | --- | --- | --- | --- | --- | | ∅ | 0 | 1 | 2 | 3 | | a | 1 | 1 | 1 | 2 | | t | 2 | 2 | 2 | 1 | Minimum edit distance = 1. ### Example 3 · cats → cat Calculate the unit-cost Levenshtein distance from “cats” to “cat”. Show the dynamic-programming table. **Given** Source=cats; target=cat; insertion=deletion=substitution=1; match=0. **To find** Minimum edit cost. ### Formula and why it applies $$ D(i,j)=\min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+c) $$ c = 0 for matching characters, 1 otherwise. Compare all three possible ways to reach each cell. ### Solution #### Step 1. Initialise empty prefixes First row counts insertions; first column counts deletions. #### Step 2. Fill row 1: source character “c” $$ D(1,1)=\min(1+1,\ 1+1,\ 0+0)=0 $$ $$ D(1,2)=\min(2+1,\ 0+1,\ 1+1)=1 $$ $$ D(1,3)=\min(3+1,\ 1+1,\ 2+1)=2 $$ #### Step 3. Fill row 2: source character “a” $$ D(2,1)=\min(0+1,\ 2+1,\ 1+1)=1 $$ $$ D(2,2)=\min(1+1,\ 1+1,\ 0+0)=0 $$ $$ D(2,3)=\min(2+1,\ 0+1,\ 1+1)=1 $$ #### Step 4. Fill row 3: source character “t” $$ D(3,1)=\min(1+1,\ 3+1,\ 2+1)=2 $$ $$ D(3,2)=\min(0+1,\ 2+1,\ 1+1)=1 $$ $$ D(3,3)=\min(1+1,\ 1+1,\ 0+0)=0 $$ #### Step 5. Fill row 4: source character “s” $$ D(4,1)=\min(2+1,\ 4+1,\ 3+1)=3 $$ $$ D(4,2)=\min(1+1,\ 3+1,\ 2+1)=2 $$ $$ D(4,3)=\min(0+1,\ 2+1,\ 1+1)=1 $$ #### Step 6. Read the completed matrix | Source / target | ∅ | c | a | t | | --- | --- | --- | --- | --- | | ∅ | 0 | 1 | 2 | 3 | | c | 1 | 0 | 1 | 2 | | a | 2 | 1 | 0 | 1 | | t | 3 | 2 | 1 | 0 | | s | 4 | 3 | 2 | 1 | Minimum edit distance = 1. ### Noisy-channel correction $$ \hat w=\arg\max_w P(x\mid w)P(w) $$ x is the observed misspelling and w a candidate intended word. P(x | w) is the error-model likelihood; P(w) is the language-model prior. In context, replace the isolated-word prior with an appropriate contextual probability. A frequent word with a plausible typo may beat a rarer candidate, but context can reverse the ranking. ### Candidate ranking For observed “teh”, candidate “the” has prior 0.05 and error likelihood 0.02. Candidate “ten” has prior 0.01 and likelihood 0.01. Select the intended word. **Given** Two candidates and their prior/error-model likelihoods are supplied. **To find** Largest noisy-channel score. ### Formula and why it applies $$ s(w)=P(x\mid w)P(w) $$ Multiply likelihood by prior; the same evidence normaliser is irrelevant to ranking. ### Solution #### Step 1. Score the $$ s(\text{the})=0.02\times0.05=0.001 $$ #### Step 2. Score ten $$ s(\text{ten})=0.01\times0.01=0.0001 $$ #### Step 3. Compare 0.001 is ten times 0.0001. Choose “the”. ### TEE 15 — Spelling correction: full edit-distance tables and noisy-channel ranking id : tee-15 title : TEE 15 — Spelling correction: full edit-distance tables and noisy-channel ranking title\_confirmed : false topic\_title : Spelling correction: full edit-distance tables and noisy-channel ranking mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Machine translation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--translation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/translation TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied **Machine translation (MT) automatically converts a source-language text into a target-language text while aiming to preserve its meaning and produce natural, grammatical output.** The source is the language received; the target is the language generated. Translating words independently is insufficient because languages differ in order, agreement, idioms and word senses. ### What a translation must preserve | Requirement | What to check | Example | | --- | --- | --- | | Participants | Who performed the action and who received it | Asha helped Riya must not become Riya helped Asha | | Event and polarity | What happened, and whether it did not happen | Do not remove not | | Time/aspect | When and how the event unfolds | is working differs from worked | | Entities and quantities | Names, dates and numbers | ₹500 must not become ₹5,000 | | Intended sense | Meaning selected by context | Financial bank versus riverbank | | Target grammar | Natural order and agreement | Hindi object-before-verb often becomes English verb-before-object | Source text → Tokenize and analyse → Resolve structure + sense → Choose target words/structure → Generate target sentence → Check adequacy and fluency ### The three major approaches | Approach | Procedure | Advantages | Limitations | | --- | --- | --- | --- | | Rule-based MT | Analyse source words/grammar → transfer structure or meaning → generate target language using dictionaries and rules. | Explicit rules and terminology can be inspected. | Rule writing is expensive; exceptions and language pairs require maintenance. | | Statistical MT | Learn translation correspondences from parallel text; combine translation adequacy with a target-language model; search candidate translations. | Learns from examples and can use phrase correspondences. | Sparse data, word-order differences and rare phrases cause errors. | | Neural MT | Encode source context; a decoder generates target tokens conditioned on that context, often using attention. | Learns contextual representations and long-range interactions. | Needs suitable data/compute; can omit, distort or generate unsupported content. | **Parallel corpus:** aligned source and target examples expressing corresponding content. A monolingual corpus contains text in only one language. Parallel alignment does not mean every source word maps to exactly one target word: translation may insert, omit or reorder function words to preserve meaning. In a simple noisy-channel view of statistical MT, let f be the observed source sentence and e a candidate target sentence. The decoder compares translation compatibility and target-language fluency: $$ \hat e=\arg\max_e P(f\mid e)P(e). $$ P(f|e) is the translation-model likelihood; P(e) is the target-language-model prior. The common P(f) denominator is omitted for ranking. A modern MT system may use a different objective; this formula explains the stated statistical model rather than every translation system. ### Worked application 1 · translation requires reordering **Question. Translate the simple Hindi sentence “मैं पानी पीता हूँ” into natural English and explain the processing stages.** #### Given Intended subject is I; object is water; predicate expresses drinking in a habitual/simple-present reading. #### To find A natural translation with preserved roles and the stage-by-stage explanation. #### Method and why it applies Identify roles before generating the target order. Use natural English subject–verb–object order for this sentence rather than copying the Hindi order. ### Solution #### Step 1. Identify the source components | Source component | Function | Meaning | | --- | --- | --- | | मैं | Subject | I | | पानी | Object | water | | पीता हूँ | Predicate | drink, habitual/simple-present reading | #### Step 2. Build the meaning relation The speaker performs the action drink; water is what is drunk. The subject and object must stay attached to these roles. #### Step 3. Choose target lexical forms मैं → I; पानी → water; the predicate is naturally realized as drink with subject I in this reading. #### Step 4. Arrange target syntax English order here is subject I + verb drink + object water. #### Step 5. Check adequacy The participant, action and object are retained. No new object, negation or time expression has been invented. #### Step 6. Check fluency “I drink water” is natural English. A literal “I water drink” retains words but fails the required target order. **Answer.** A suitable translation is “I drink water.” The processing preserves meaning while changing structure. ### Worked application 2 · one word, two meanings **Question. Explain how an MT system should distinguish bank in “She deposited money in the bank” and “They sat on the bank of the river” before translation.** #### Given Two English source sentences contain the same written word bank. #### To find The intended sense in each and suitable Hindi phrase-level renderings. #### Method and why it applies Apply word-sense disambiguation using nearby context before choosing the target term. Identical spelling does not force identical translation. ### Solution #### Step 1. Collect the first sentence's clues Deposited and money strongly support the financial-institution sense. #### Step 2. Select its target expression Use बैंक for the financial institution; retain the money-deposit event in the complete translated sentence. #### Step 3. Collect the second sentence's clues River, sat and the location phrase support land beside a river. #### Step 4. Select its target expression Use नदी का किनारा / नदी के किनारे as appropriate to the target sentence. A financial-bank translation would be wrong here. #### Step 5. Generate grammatical context A suitable rendering of the second sentence is “वे नदी के किनारे बैठे।” Grammar adjusts the phrase to its role; do not paste an isolated dictionary form without inflection. #### Step 6. State why a dictionary alone fails A bilingual entry offers candidate words. Context and syntax select and inflect the candidate expressing the intended sense. **Answer.** Financial bank → बैंक; riverbank → नदी का किनारा, with grammatical realization such as नदी के किनारे in the sentence. ### Worked numerical 3 · compare two statistical translations **Question. A statistical MT system compares candidate A with P(f|A)=0.12 and P(A)=0.03, and candidate B with P(f|B)=0.08 and P(B)=0.06. Select the candidate under the noisy-channel product model.** #### Given Two candidate translations; likelihoods and target-language priors are provided. Ignore all other candidates for this ranking exercise. #### To find Each score and the selected translation. #### Method and why it applies Multiply translation likelihood by target-language prior because this specified model values both compatibility and fluency. These products are comparison scores, not normalized posterior probabilities. ### Solution #### Step 1. Write the scoring rule $$ s(e)=P(f\mid e)P(e). $$ #### Step 2. Compute candidate A $$ s(A)=0.12\times0.03=0.0036. $$ #### Step 3. Compute candidate B $$ s(B)=0.08\times0.06=0.0048. $$ #### Step 4. Compare the products 0.0048 > 0.0036. B wins even though A had a larger translation likelihood; B's target-language prior compensates. #### Step 5. Interpret the result The model prefers B under the supplied numbers. The score alone does not establish that B is semantically correct; a poor model can rank a bad translation highly. **Answer.** Select B, with score 0.0048 versus A's 0.0036. ### Evaluation and common failure cases **Adequacy** asks whether the original information and intended meaning were preserved. **Fluency** asks whether the target text is natural and grammatical. A fluent sentence can still mistranslate a negation or number, so assess both. Automatic reference-overlap metrics provide a useful signal, but several valid translations can use different words. | Failure | Why it occurs | What the system needs | | --- | --- | --- | | Word-for-word order | Source and target syntax differ | Structural modelling/reordering | | Wrong bank/bat sense | Multiple senses share a written form | Context-sensitive lexical choice | | Idiomatic phrase translated literally | Meaning is not the sum of isolated words | Phrase/context knowledge | | Rare name or specialist term corrupted | Weak vocabulary/domain coverage | Appropriate data and terminology handling | | Missing negation or number | Omission or generation error | Content-preservation checks | | Low-resource language pair | Few representative parallel examples | Suitable data collection/adaptation | For a 10-mark design explanation, connect the definition, pipeline, three approaches, worked source-to-target example, data requirements, evaluation and limitations. Merely listing “rule based, statistical, neural” does not explain how a translation is produced. **Yaad rakho:** MT: pehle kaun-kya-kisko samjho; phir target language ke words aur order mein wahi meaning banao. ### Machine translation: meaning preservation and worked processing id : tee-16 title : Machine translation: meaning preservation and worked processing title\_confirmed : false topic\_title : Machine translation: meaning preservation and worked processing mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Question answering Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--qa/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/qa TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied **Question answering (QA) returns an answer to a natural-language question using available evidence or knowledge.** Information retrieval usually returns ranked documents; QA must identify the requested information and deliver the answer. A relevant document is useful evidence but is not automatically the answer. ### Question types and answer types | Question | Expected answer type | Required operation | | --- | --- | --- | | Who founded the company? | Person or organisation | Find the founder relation | | When did the museum open? | Date/time | Locate the opening date | | Where is the campus? | Location | Locate the campus-location relation | | How many books were issued? | Quantity | Read or compute the stated count | | Why was the train cancelled? | Reason/explanation | Find the causal evidence | | Is the office open on Sunday? | Yes/no with evidence | Check the relevant schedule statement | **Closed-domain QA** operates within a restricted area such as a university handbook. **Open-domain QA** addresses a wider range of subjects and usually needs broad retrieval. **Extractive QA** selects a span from a passage. **Generative QA** constructs answer text; it still must preserve the evidence and avoid unsupported details. Question analysis → Evidence retrieval → Passage/record understanding → Answer selection or composition → Support check → Present answer ### Explain each module | Module | What happens | Why it matters | | --- | --- | --- | | Question analysis | Identify topic, intent, constraints and answer type. | When asks for a time, not a place. | | Retrieval | Find a relevant passage or structured fact. | Reduces the search space. | | Evidence understanding | Parse relations, resolve references and ambiguous words. | The correct date must belong to the correct event. | | Answer formation | Extract a span or compose a short supported response. | Turns a document into an actual answer. | | Verification | Check whether every claimed detail follows from evidence. | Prevents filling gaps by guessing. | | Presentation | Return a concise answer with suitable context. | Avoids dumping unrelated text. | ### Worked application 1 · extract the correct fact **Question. Given “The City Museum opened in 1998 in Pune. It was renovated in 2016”, answer “When did the museum open?” Show the QA stages.** #### Given A two-sentence passage. Dates 1998 and 2016 refer to different events. #### To find The opening date, not the renovation date. #### Method and why it applies Use extractive QA: identify the requested event and expected date type, then choose the span linked to that event. ### Solution #### Step 1. Analyse the question When requires a date/time answer. The event is opening; the entity is the City Museum. #### Step 2. Retrieve the relevant sentence The first sentence mentions the museum and its opening. The second sentence discusses renovation. #### Step 3. Identify event–date relations | Event | Date | | --- | --- | | opened | 1998 | | renovated | 2016 | #### Step 4. Select the answer span 1998 is attached to opened, so select the four-character year span 1998. #### Step 5. Reject the distractor 2016 is also a date, but it answers a different question: when was the museum renovated? #### Step 6. Present the answer “The museum opened in 1998.” **Answer.** 1998. Matching only the answer type date would be insufficient without matching the event. ### Worked application 2 · resolve a pronoun before answering **Question. Given “Mira joined Orion Labs in 2022. She became its research lead in 2024”, answer “Who became Orion Labs' research lead, and when?”** #### Given The passage contains the name Mira, pronoun She, organisation Orion Labs and possessive its. Assume the normal reading that She refers to Mira and its to Orion Labs. #### To find The person and the date of becoming research lead. #### Method and why it applies Combine reference resolution with relation extraction. The question asks for two fields, both tied to the promotion/role event. ### Solution #### Step 1. Identify requested fields Who → person; when → date. The relation is became research lead of Orion Labs. #### Step 2. Resolve mentions | Mention | Referent | | --- | --- | | She | Mira | | its | Orion Labs | #### Step 3. Read the target event After resolution, the second sentence states that Mira became Orion Labs' research lead in 2024. #### Step 4. Distinguish the dates 2022 belongs to joining the organisation. 2024 belongs to becoming research lead. #### Step 5. Compose a supported answer “Mira became Orion Labs' research lead in 2024.” #### Step 6. Verify completeness The answer supplies both requested fields and does not add an unsupported department, qualification or reason. **Answer.** Mira, in 2024. Reference resolution connects the second sentence to the named person and organisation. ### Worked application 3 · detect insufficient evidence **Question. A handbook excerpt states “The library opens at 8 a.m. on weekdays.” A user asks “At what time does it close on Sunday?” Explain the correct QA response.** #### Given Only the opening time for weekdays is supplied. No closing time or Sunday schedule is present. #### To find Whether an answer is supported and how to handle the missing information. #### Method and why it applies Compare every question constraint against the evidence. Topic overlap with library is not sufficient: time type and day must also match. ### Solution #### Step 1. Extract the question constraints | Constraint | Requested | | --- | --- | | Entity | Library | | Event | Closing | | Day | Sunday | | Answer type | Time | #### Step 2. Extract the supplied fact Entity=library; event=opening; day=weekdays; time=8 a.m. #### Step 3. Identify mismatches Opening is not closing. Weekdays do not supply a Sunday schedule. #### Step 4. Avoid the unsupported substitution Do not answer 8 a.m. just because it is the only time in the passage. #### Step 5. Return an evidence-limited response “The excerpt does not state the library's Sunday closing time.” A system with further retrieval can search a relevant schedule. #### Step 6. State the benefit The system avoids converting a relevant-looking number into an incorrect answer. **Answer.** No Sunday closing time is supported by the excerpt. Report insufficient evidence or retrieve the missing schedule. ### Advantages, limits and evaluation | Design choice | Benefit | Limitation | | --- | --- | --- | | Extractive answer | The returned wording comes directly from evidence. | The answer may require combining sentences or may not be present as one span. | | Generative answer | Can combine supported facts into a readable response. | Can introduce unsupported information unless carefully checked. | | Closed-domain collection | Controlled terminology and relevant resources. | Cannot answer facts outside its knowledge source. | | Broad retrieval | Wider question coverage. | Irrelevant, outdated or conflicting sources can be retrieved. | Evaluate **exact match** when a canonical answer string is expected, **token overlap/F1** when wording may vary, and **factual correctness and completeness** when answers are composed. An answer can be grammatically fluent but wrong. Questions with no answer in the evidence should be included in evaluation so the system's restraint is also tested. For a longer exam answer, define QA, distinguish it from retrieval, explain answer types and system variants, draw the pipeline, work through a passage including distractors, and discuss ambiguity, missing evidence and evaluation. A list of chatbot names is not a QA design. **Yaad rakho:** Question ka entity + event + constraint match karo. Passage mein koi bhi number mil gaya matlab answer nahi. ### Question answering: identify, retrieve and justify the answer id : tee-17 title : Question answering: identify, retrieve and justify the answer title\_confirmed : false topic\_title : Question answering: identify, retrieve and justify the answer mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Information extraction Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--extraction/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/extraction TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied **Information extraction (IE) converts unstructured language into structured information such as named entities, relations and events.** A structured output has explicit fields or records that can be stored, searched or analysed. IE is broader than NER: finding two names does not yet determine how those entities are related. ### The three levels of extracted information | Level | What it identifies | Example | | --- | --- | --- | | Entity extraction / NER | A span and its entity type | Asha → person; Orbit Labs → organisation | | Relation extraction | A labelled connection between entities | works-for(Asha, Orbit Labs) | | Event extraction | An occurrence with roles and attributes | Joining: employee=Asha, employer=Orbit Labs, date=2025 | An **entity** is a participant such as a person or organisation. A **relation** links participants. An **event** describes something that happened, may happen or did not happen. An **event role** names a participant's function: buyer, seller, employee, employer, location or time. These labels must follow the chosen extraction schema. Raw text → Sentence/token processing → Entity spans + types → Reference resolution → Relations and event roles → Normalize and store records ### An extraction pipeline explained | Step | Operation | Reason | | --- | --- | --- | | Specify schema | Choose fields and allowed relations/events. | A system needs to know what records it is building. | | Preprocess | Segment sentences and tokenize without losing useful structure. | Locate expressions and boundaries. | | Recognise entities | Find spans and assign types. | Names can contain several words. | | Resolve references | Link pronouns/descriptions to the right entities. | Two mentions may denote one participant. | | Extract links/events | Use patterns, syntax or learned models. | Co-occurrence alone does not identify a relationship. | | Handle qualifiers | Preserve negation, time and uncertainty. | Not hired differs from hired. | | Normalize | Standardize dates, names or quantities where justified. | Equivalent forms should be comparable. | | Store/evaluate | Save records and compare them with annotated answers. | Wrong links matter even if entity names are correct. | ### Worked application 1 · turn a sentence into a record **Question. Extract entities and a joining event from “Asha joined Orbit Labs in Mumbai on 3 May 2025.” Use fields person, organisation, location, event and date.** #### Given One affirmative sentence; Orbit Labs is an organisation and Mumbai is a location. #### To find Entity spans/types and one structured joining record. #### Method and why it applies Find the complete entity spans, then assign their roles using the verb joined and its grammatical context. ### Solution #### Step 1. Mark entity spans | Span | Type | | --- | --- | | Asha | Person | | Orbit Labs | Organisation | | Mumbai | Location | | 3 May 2025 | Date | #### Step 2. Identify the event trigger Joined signals a joining event. The subject Asha is the joining person; Orbit Labs is the joined organisation. #### Step 3. Attach contextual fields In Mumbai supplies a location associated with the stated joining event. On 3 May 2025 supplies its date. Do not infer the company's permanent headquarters from this sentence alone. #### Step 4. Normalize the date The unambiguous date can be stored as 2025-05-03 while preserving its original text. #### Step 5. Build the record | Field | Value | | --- | --- | | event | joining | | person | Asha | | organisation | Orbit Labs | | location | Mumbai | | date | 2025-05-03 | #### Step 6. State the information limit The sentence supports a joining event on that date. By itself it does not prove that Asha still works there indefinitely. **Answer.** A joining record for Asha and Orbit Labs at the stated location/date, with complete entity spans preserved. ### Worked application 2 · direction of a relation matters **Question. From “Nova acquired Luma for ₹50 crore in 2025”, extract an acquisition event with buyer, acquired organisation, amount and year.** #### Given Nova and Luma are organisation names. The active verb is acquired; no seller is named. #### To find A correctly directed acquisition record. #### Method and why it applies Use grammatical roles to identify buyer and acquired organisation. Merely returning the unordered pair {Nova,Luma} loses the event direction. ### Solution #### Step 1. Recognise participant entities Nova is the subject organisation; Luma is the object organisation. #### Step 2. Locate the trigger Acquired identifies an acquisition event. #### Step 3. Assign roles | Role | Entity | | --- | --- | | Buyer/acquirer | Nova | | Acquired organisation | Luma | #### Step 4. Extract amount and time Amount text=₹50 crore; year=2025. Currency and magnitude belong together; do not drop crore or replace it by lakh. #### Step 5. Create the record | Field | Value | | --- | --- | | event | acquisition | | buyer | Nova | | acquired\_organisation | Luma | | amount | INR 50 crore | | year | 2025 | | seller | Not stated | #### Step 6. Reject an unsupported inference The sentence does not name the former owner/seller. Luma is the acquired organisation, not automatically the seller. **Answer.** Acquisition(buyer=Nova, acquired organisation=Luma, amount=INR 50 crore, year=2025). Seller remains unspecified. ### Worked application 3 · reference, negation and time **Question. Extract employment-related events from “Ravi joined Vega in 2021. He left the company in 2024. He did not join Orion.” Keep event type, person, organisation, date and polarity.** #### Given He refers to Ravi; the company refers to Vega. The last sentence explicitly negates joining Orion. #### To find A structured event list without claiming an unsupported current employer. #### Method and why it applies Resolve references, identify each event separately and preserve whether it is affirmed or negated. ### Solution #### Step 1. Resolve the repeated mentions | Mention | Referent | | --- | --- | | He in sentence 2 | Ravi | | the company | Vega | | He in sentence 3 | Ravi | #### Step 2. Extract the first event Joining; person=Ravi; organisation=Vega; date=2021; polarity=affirmed. #### Step 3. Extract the second event Leaving; person=Ravi; organisation=Vega; date=2024; polarity=affirmed. #### Step 4. Extract the third statement Joining; person=Ravi; organisation=Orion; date=not stated; polarity=negated. This must not create an affirmative employment relation with Orion. #### Step 5. Write the event table | Event | Person | Organisation | Date | Polarity | | --- | --- | --- | --- | --- | | joining | Ravi | Vega | 2021 | Affirmed | | leaving | Ravi | Vega | 2024 | Affirmed | | joining | Ravi | Orion | Not stated | Negated | #### Step 6. Interpret current status cautiously The text says that Ravi left Vega and did not join Orion. It supplies no other current employer. Do not fill that field by guessing. **Answer.** Three qualified event statements, including a negated Orion joining. No current employer can be established from this text. ### Approaches and their trade-offs | Approach | Mechanism | Advantage | Limitation | | --- | --- | --- | --- | | Dictionary/rule based | Match known names, trigger words and patterns such as person joined organisation. | Transparent and useful in a narrow domain. | New names, paraphrases and complex syntax can escape patterns. | | Supervised statistical | Learn entity/relation/event labels from annotated examples and features. | Combines several cues consistently. | Requires representative labels and careful feature design. | | Neural/contextual | Use contextual representations to classify spans and links or generate a structured record. | Can capture paraphrases and broader context. | May still invent or misassign fields; schema and factual checks remain necessary. | Applications include extracting company events from news, conditions/treatments from clinical text, skills from resumes and attributes from product descriptions. Extraction quality is measured against annotated records: an entity match can require the correct span and type; a relation match additionally requires the correct participants and direction. Precision measures how many extracted records are correct; recall measures how many gold records were found. For a 10-mark answer, begin with the definition and structured output, distinguish NER/relation/event extraction, draw and explain the pipeline, work through a record with roles and qualifiers, compare approaches, then state evaluation and limitations. “Extract keywords” alone is too narrow: keywords do not necessarily represent a factual relation. **Yaad rakho:** NER naam pakadta hai; relation kaun-kisse jodta hai; event kya hua batata hai. Negation aur date mat girao. ### Information extraction and retrieval: records, indexes and precision/recall id : tee-18 title : Information extraction and retrieval: records, indexes and precision/recall title\_confirmed : false topic\_title : Information extraction and retrieval: records, indexes and precision/recall mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Information retrieval Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/read/u7--retrieval/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/read/u7/retrieval TEE only · Unit 7 · current syllabus, including spelling correction; faculty slide pack not supplied **Information retrieval (IR)** locates and ranks items relevant to an information need, usually expressed as a query. The collection may contain documents, product descriptions or scientific papers. Retrieval estimates relevance; it does not itself establish that every retrieved claim is true. Documents → cleaning → Build index → Query → same representation → Match & rank → Return relevant results ### The inverted index An inverted index maps a term to the documents containing it, often with positions or counts. For D1 “cats drink milk” and D2 “dogs drink water”, the posting list for drink is {D1,D2}, while milk maps only to D1. This avoids rescanning every document for every query. | Term | Posting list | | --- | --- | | cats | D1 | | drink | D1, D2 | | milk | D1 | | dogs | D2 | | water | D2 | ### Matching and ranking Boolean retrieval combines term membership using AND, OR and NOT. Vector-space retrieval represents documents and queries as weighted vectors, for example with TF-IDF, and compares them with cosine similarity. Learned retrieval can use dense embeddings for meaning-based matching. The representation used at query time must be compatible with the document index. ### Search engines and recommendation A search engine receives an explicit query and ranks matching indexed pages. A recommender may derive an implicit query from a user’s liked items or profile and retrieve similar candidates. Both require candidate retrieval and ranking, but the source of the information need differs. ### Evaluate the results For retrieval, TP is the number of retrieved relevant documents, FP retrieved irrelevant documents, and FN relevant documents not retrieved. Precision and recall use the same formulas as classification. Ranking measures also consider the positions of relevant items; a relevant result at rank 1 is more useful than one far below. ### Retrieval evaluation A search system retrieves 8 documents; 6 are relevant. There are 10 relevant documents in the collection. Find precision, recall and F1. **Given** Retrieved=8; relevant retrieved=6; all relevant=10. **To find** P, R and F1. ### Formula and why it applies $$ P=\frac{\text{relevant retrieved}}{\text{retrieved}},\quad R=\frac{\text{relevant retrieved}}{\text{all relevant}} $$ $$ F_1=\frac{2PR}{P+R} $$ The denominator is 8 for precision and 10 for recall because they ask different questions. ### Solution #### Step 1. Identify the confusion counts TP=6; FP=8−6=2; FN=10−6=4. #### Step 2. Precision $$ P=\frac68=0.75 $$ #### Step 3. Recall $$ R=\frac6{10}=0.6 $$ #### Step 4. F1 $$ F_1=\frac{2(0.75)(0.6)}{0.75+0.6}=\frac{0.9}{1.35}=\frac23\approx0.666667 $$ Precision=0.75, recall=0.60, F1≈0.666667. **Yaad rakho:** Precision: jo laaye, unmein kitne sahi? Recall: jo sahi the, unmein kitne laaye? ### Information extraction and retrieval: records, indexes and precision/recall id : tee-18 title : Information extraction and retrieval: records, indexes and precision/recall title\_confirmed : false topic\_title : Information extraction and retrieval: records, indexes and precision/recall mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Core formula sheet Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/formula-index--all/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/formula-index/all M2 · formulas with meaning · full examples follow Read each formula with its meaning. The remaining lessons in this tab contain the complete worked examples for these methods. Use the same vocabulary, boundary, case and token-count conventions throughout one solution. ### N-gram count $$ \max(T-n+1,0) $$ Unpadded sequence of T tokens. Count only valid contiguous windows. ### Unigram MLE $$ P(w)=C(w)/N $$ N is the training-token total under the stated boundary convention. ### Bigram MLE $$ P(w_i\mid w_{i-1})=C(w_{i-1},w_i)/C(w_{i-1}) $$ Denominator = history count. ### Trigram MLE $$ P(w_i\mid w_{i-2},w_{i-1})=C(w_{i-2},w_{i-1},w_i)/C(w_{i-2},w_{i-1}) $$ Denominator = two-word history count. ### Add-one / Laplace $$ P(w\mid h)=\frac{C(hw)+1}{C(h)+|V|} $$ Add one to every possible vocabulary continuation. ### Sentence probability $$ P(W)\approx\prod_iP(w_i\mid w_{i-1}) $$ Multiply in order; include an end factor if specified. ### Perplexity $$ PP=P(W)^{-1/N} $$ N is the number of predicted tokens/events used by the question. ### Emission $$ P(w\mid t)=C(w,t)/C(t) $$ Word given tag; count tokens with that tag. ### Transition $$ P(t_j\mid t_i)=C(t_i\to t_j)/\sum_k C(t_i\to t_k) $$ Next tag given previous tag; count outgoing transitions. ### HMM joint path $$ P(W,T)=\pi_{t_1}b_{t_1}(w_1)\prod_{i=2}^{n}a_{t_{i-1},t_i}b_{t_i}(w_i) $$ Multiply initial, emission and transition factors; add end probability if used. ### Tag sequences $$ K^T $$ K tags, T word positions; count before constraints. ### Viterbi $$ \delta_i(t)=b_t(w_i)\max_s[\delta_{i-1}(s)a_{s,t}] $$ Choose the largest predecessor score; store a backpointer. ### Lesk $$ score(s)=|context\cap gloss(s)| $$ Use the specified normalisation and overlap convention. ### Naive Bayes class prior $$ P(c)=D_c/D $$ D counts labelled examples; this is not the word-token total. ### Smoothed NB likelihood $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ N\_c counts training tokens in class c. ### NB decision $$ \hat c=\arg\max_c P(c)\prod_iP(w_i\mid c) $$ Select label with largest score; normalise only for posterior values. **Yaad rakho:** Condition jis par hai, denominator usi ka; sequence mein multiply; options mein maximum. --- ## TEE · Unigrams, bigrams and trigrams: construct them correctly Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-counts--n-grams-construction/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-counts/n-grams-construction Formulas explained through full worked examples **Definition.** An n-gram is a contiguous sequence of n items. For word n-grams, the items are consecutive word tokens. A unigram contains one word; a bigram contains two; a trigram contains three. Character n-grams use characters instead, so always identify the unit being counted. Tokens: we | study | natural | language → Move a window one token at a time → Bigrams: we study / study natural / natural language For m tokens without padding, the number of n-gram *positions* is max(0, m−n+1). Repeated positions may contain the same n-gram type. A bag-of-n-grams representation counts each n-gram type and loses the overall sentence order beyond the local order inside each n-gram. $$ \text{Number of unpadded n-gram positions}=\max(0,m-n+1) $$ **Question. List all word unigrams, bigrams and trigrams for “the prime minister of our country”. Do not add boundary tokens.** #### Given Six tokens: the, prime, minister, of, our, country. n = 1, 2 and 3. #### To find All three lists and their counts. #### Formula and why it applies $$ \#\text{positions}=m-n+1 $$ Each window must contain consecutive words and move one position. No wraparound or invented final word is allowed. #### Solution ##### Step 1. Take one word at a time Unigrams: (the), (prime), (minister), (of), (our), (country). Count = 6−1+1 = 6. ##### Step 2. Take two consecutive words Bigrams: (the, prime), (prime, minister), (minister, of), (of, our), (our, country). Count = 6−2+1 = 5. ##### Step 3. Take three consecutive words Trigrams: (the, prime, minister), (prime, minister, of), (minister, of, our), (of, our, country). Count = 6−3+1 = 4. **Final answer.** 6 unigrams, 5 bigrams and 4 trigrams. **Question. Construct bigrams for “I like tea” using one start and one end marker.** #### Given Original tokens: I, like, tea. Add and once each. #### To find The boundary-aware bigram sequence. #### Formula and why it applies $$ \#\text{bigrams}=(m+2)-1=m+1 $$ The start context predicts the first word, and the last word predicts that the sentence ends. #### Solution ##### Step 1. Write the padded sequence I like tea ##### Step 2. Move a two-token window (, I), (I, like), (like, tea), (tea, ). ##### Step 3. Count the windows $$ 3+1=4 $$ **Final answer.** There are four boundary-aware bigrams. **Question. Construct trigrams for “I like tea” using two start markers and one end marker.** #### Given Original sentence has three words. For a trigram model each predicted token needs two prior context tokens. #### To find All trigram prediction events. #### Formula and why it applies $$ P(w_i\mid w_{i-2},w_{i-1}) $$ Two start markers provide a complete context even for the first word. #### Solution ##### Step 1. Pad with two starts I like tea ##### Step 2. Slide a three-token window (, , I), (, I, like), (I, like, tea), (like, tea, ). ##### Step 3. Count predicted events The model predicts I, like, tea and the end marker: four predictions. **Final answer.** Four trigrams. Start markers are context; they are not ordinary predicted words. **Question. Give the bag-of-bigrams counts for “a b a b” without padding.** #### Given Four tokens in their given order. #### To find Bigram positions, distinct bigram types and their counts. #### Formula and why it applies $$ \#\text{positions}=4-2+1=3 $$ A bag stores frequencies; repeated bigrams must not be discarded. #### Solution ##### Step 1. Enumerate all positions (a,b), (b,a), (a,b). ##### Step 2. Group identical pairs | Bigram type | Count | | --- | --- | | a b | 2 | | b a | 1 | **Final answer.** Three bigram positions but two distinct bigram types. **Yaad rakho:** n-gram mein n words; context mein n−1 words. --- ## TEE · Unigram estimation and complete worked examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-counts--unigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-counts/unigram-estimation Formulas explained through full worked examples **Maximum likelihood estimation (MLE)** chooses probabilities equal to observed relative frequencies. In a unigram model, every word is predicted independently of its surrounding words. If a word occurs c times in N training tokens, its estimate is c/N. To score m specified words, multiply their m unigram probabilities. $$ P_{\mathrm{MLE}}(w)=\frac{C(w)}{N}\qquad P(w_1,\ldots,w_m)\approx\prod_{i=1}^{m}P(w_i) $$ Choose a tokenization convention before counting. Lowercasing, punctuation handling and boundary markers can change the vocabulary and counts. Never count words with one convention and form the denominator with another. A unigram model gives the same product to rearrangements of the same word multiset; that is one of its limitations. **Question. Train an unpadded, case-insensitive unigram model on “I like tea” and “I like coffee”. Find the probability of “I like tea”.** #### Given Training sentences: I like tea; I like coffee. Use lowercase words; exclude punctuation and sentence boundaries. #### To find The vocabulary, word probabilities and test-sentence probability. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P(I\ like\ tea)=P(I)P(like)P(tea) $$ The unigram approximation ignores order and uses total token count for every denominator. #### Solution ##### Step 1. Count every training token Each sentence contains three words, so N = 3+3 = 6. ##### Step 2. Count types and frequencies | Word | Count | MLE probability | | --- | --- | --- | | i | 2 | 2/6 = 1/3 | | like | 2 | 2/6 = 1/3 | | tea | 1 | 1/6 | | coffee | 1 | 1/6 | Vocabulary size V = 4. ##### Step 3. Write one factor per test word $$ P=\frac{2}{6}\times\frac{2}{6}\times\frac{1}{6} $$ ##### Step 4. Multiply the fractions $$ P=\frac{4}{216}=\frac{1}{54}\approx0.018519 $$ **Final answer.** The unigram sentence probability is approximately 0.018519. **Question. A corpus contains counts a: 4, b: 3 and c: 1. Find the unigram probability of “a b a”.** #### Given C(a)=4, C(b)=3, C(c)=1. No boundary markers. #### To find P(a b a). #### Formula and why it applies $$ P(a\,b\,a)=P(a)P(b)P(a) $$ A repeated test word contributes a probability factor at every occurrence; do not include it just once. #### Solution ##### Step 1. Find the total $$ N=4+3+1=8 $$ ##### Step 2. Estimate the required probabilities $$ P(a)=4/8=1/2,\qquad P(b)=3/8 $$ ##### Step 3. Use both occurrences of a $$ P=\frac12\times\frac38\times\frac12 $$ ##### Step 4. Simplify $$ P=\frac3{32}=0.09375 $$ **Final answer.** P(a b a) = 0.09375. **Yaad rakho:** Repeated word ka probability bhi repeat multiply hoga. **Question. An unsmoothed unigram model is trained on “red blue red”. Find the probability of “red green”.** #### Given N=3. Counts: red=2, blue=1, green=0. The test contains red and green. #### To find The unsmoothed test probability and the reason for its value. #### Formula and why it applies $$ P(red\ green)=P(red)P(green) $$ MLE assigns zero to an unseen outcome. Smoothing is needed if an unseen in-vocabulary item must receive probability. #### Solution ##### Step 1. Estimate red $$ P(red)=2/3 $$ ##### Step 2. Estimate the unseen word $$ P(green)=0/3=0 $$ ##### Step 3. Multiply $$ (2/3)\times0=0 $$ **Final answer.** Unsmoothed probability = 0. This reflects missing training evidence, not proof that the phrase is impossible. --- ## TEE · Bigram counts: build the table before multiplying Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-counts--bigram-count-table/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-counts/bigram-count-table Formulas explained through full worked examples A **bigram probability table** uses the previous word as its row and the next word as its column. Divide each cell count by its row's total outgoing count. With one end marker on every sentence, each ordinary training word has a continuation, so its outgoing count equals its token frequency. Do not create a bigram that crosses from one sentence into the next. $$ P(w\mid h)=\frac{C(h,w)}{\sum_{v\in V}C(h,v)}=\frac{C(h,w)}{C(h)} $$ Add boundaries to each sentence separately → List adjacent pairs → Count repeated pairs → Divide each row by its total → Follow the test sentence through the table **Question. Train a bigram model on “I like tea”, “I like coffee” and “you like tea”. Use one start and one end marker. Find the probability of “I like tea”.** #### Given Corpus: I like tea ; I like coffee ; you like tea . Preserve these tokens. #### To find All nonzero bigram estimates and the test probability. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Every sentence starts and ends independently. Four probability factors are required for the three test words plus end. #### Solution ##### Step 1. List the training pairs Sentence 1: (,I), (I,like), (like,tea), (tea,). Sentence 2: (,I), (I,like), (like,coffee), (coffee,). Sentence 3: (,you), (you,like), (like,tea), (tea,). ##### Step 2. Create the nonzero count table | History → next | Pair count | History count | Probability | | --- | --- | --- | --- | | → I | 2 | 3 | 2/3 | | → you | 1 | 3 | 1/3 | | I → like | 2 | 2 | 1 | | you → like | 1 | 1 | 1 | | like → tea | 2 | 3 | 2/3 | | like → coffee | 1 | 3 | 1/3 | | tea → | 2 | 2 | 1 | | coffee → | 1 | 1 | 1 | ##### Step 3. Write the test path $$ P=P(I\mid\langle s\rangle)P(like\mid I)P(tea\mid like)P(\langle/s\rangle\mid tea) $$ ##### Step 4. Substitute each matching row $$ P=\frac23\times1\times\frac23\times1 $$ ##### Step 5. Multiply $$ P=\frac49\approx0.444444 $$ **Final answer.** P(I like tea) = 4/9 ≈ 0.444444. **Question. Using the same three-sentence corpus given here—“I like tea”, “I like coffee”, “you like tea”—find the bigram probabilities of “you like coffee” and “tea like I”. Include boundaries.** #### Given Corpus with one start/end marker per sentence. Counts: start→you=1/3; you→like=1; like→coffee=1/3; coffee→end=1. No training sentence starts with tea. #### To find The two test probabilities. #### Formula and why it applies $$ P(\text{sentence})=\prod P(\text{next token}\mid\text{previous token}) $$ A model can assign a nonzero probability to a sentence it never saw if all of its local bigrams were seen. #### Solution ##### Step 1. Score you like coffee $$ P=\frac13\times1\times\frac13\times1=\frac19\approx0.111111 $$ ##### Step 2. Inspect tea like I $$ P(tea\mid\langle s\rangle)=0/3=0 $$ ##### Step 3. Apply the zero factor The full product is zero, regardless of the remaining factors. **Final answer.** P(you like coffee) = 1/9; P(tea like I) = 0. A new sentence is not necessarily a zero-probability sentence. **Question. Obtain the unsmoothed bigram probability of “students are from Vellore” from this corpus: “I am from Vellore”; “I am a teacher”; “students are good and are from various cities”; “students from Vellore do engineering”. Use sentence boundaries.** #### Given Four training sentences, each padded with one and . Test sequence: students are from Vellore . #### To find The five required conditional probabilities and their product. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h) $$ Count all occurrences in the entire corpus. In particular, are occurs twice inside the third sentence, and Vellore is followed by end once and do once. #### Solution ##### Step 1. Count the histories | History | Occurrences with continuation | | --- | --- | | | 4 | | students | 2 | | are | 2 | | from | 3 | | Vellore | 2 | ##### Step 2. Count the required pairs | Required pair | Count | Probability | | --- | --- | --- | | students | 2 | 2/4 | | students are | 1 | 1/2 | | are from | 1 | 1/2 | | from Vellore | 2 | 2/3 | | Vellore | 1 | 1/2 | ##### Step 3. Write the complete product $$ P=\frac24\times\frac12\times\frac12\times\frac23\times\frac12 $$ ##### Step 4. Multiply progressively $$ \frac24\times\frac12=\frac14,\quad \frac14\times\frac12=\frac18 $$ $$ \frac18\times\frac23=\frac1{12},\quad \frac1{12}\times\frac12=\frac1{24} $$ **Final answer.** P = 1/24 ≈ 0.0416667. **Yaad rakho:** Same sentence mein word do baar aaye toh count bhi do. --- ## TEE · Trigram estimates: two-word history, not two denominators Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-counts--trigram-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-counts/trigram-estimation Formulas explained through full worked examples A trigram model predicts the next word from the two preceding words. The numerator is the count of a triple. The denominator is the count of its two-word history with an available next token. Do not multiply two independent bigram probabilities and call that a trigram probability. $$ P(w_i\mid w_{i-2},w_{i-1})=\frac{C(w_{i-2},w_{i-1},w_i)}{C(w_{i-2},w_{i-1})} $$ **Question. A corpus contains 12 occurrences of “prime minister”. Eight are followed by “of”, three by “said”, and one by “visited”. Estimate the three trigram probabilities.** #### Given History h = (prime, minister). Count of this history = 12. Triple counts for of, said, visited are 8, 3, 1. #### To find The conditional distribution of the next word. #### Formula and why it applies $$ P(w\mid prime,minister)=C(prime,minister,w)/12 $$ All three candidates share one two-word history, so they share denominator 12. #### Solution ##### Step 1. Calculate of $$ P(of\mid prime,minister)=8/12=2/3 $$ ##### Step 2. Calculate said $$ P(said\mid prime,minister)=3/12=1/4 $$ ##### Step 3. Calculate visited $$ P(visited\mid prime,minister)=1/12 $$ ##### Step 4. Confirm normalization $$ 8/12+3/12+1/12=12/12=1 $$ **Final answer.** The probabilities are 2/3, 1/4 and 1/12. **Question. Train a trigram model on “I like tea”, “I like coffee” and “you like tea”. Use two start markers and one end marker. Score “I like tea”.** #### Given Three sentences. Padded first sentence is I like tea ; pad the other two in the same way. #### To find Four trigram prediction probabilities and the sentence probability. #### Formula and why it applies $$ P=\prod_i P(w_i\mid w_{i-2},w_{i-1}) $$ Each prediction uses a two-token context; the two start markers supply the first history. #### Solution ##### Step 1. Count the first-word event The history (,) occurs three times, and two are followed by I. $$ P(I\mid\langle s\rangle,\langle s\rangle)=2/3 $$ ##### Step 2. Count the second-word event (,I) occurs twice; both are followed by like. $$ P(like\mid\langle s\rangle,I)=2/2=1 $$ ##### Step 3. Count the third-word event (I,like) occurs twice; one is followed by tea. $$ P(tea\mid I,like)=1/2 $$ ##### Step 4. Count the end event (like,tea) occurs twice; both are followed by end. $$ P(\langle/s\rangle\mid like,tea)=2/2=1 $$ ##### Step 5. Multiply all four $$ P=\frac23\times1\times\frac12\times1=\frac13 $$ **Final answer.** Trigram P(I like tea) = 1/3. It differs from the bigram estimate because the context and denominator changed. **Question. A trigram model supplies P(we|,)=0.4, P(study|,we)=0.5, P(NLP|we,study)=0.8 and P(|study,NLP)=0.25. Score “we study NLP”.** #### Given Four trigram conditional factors. The end marker is included. #### To find The complete sentence probability. #### Formula and why it applies $$ P=P(we\mid s,s)P(study\mid s,we)P(NLP\mid we,study)P(end\mid study,NLP) $$ Use each supplied two-word history exactly; do not swap its word order. #### Solution ##### Step 1. Insert the values $$ P=0.4\times0.5\times0.8\times0.25 $$ ##### Step 2. Multiply the first pair $$ 0.4\times0.5=0.2 $$ ##### Step 3. Include NLP $$ 0.2\times0.8=0.16 $$ ##### Step 4. Include the end $$ 0.16\times0.25=0.04 $$ **Final answer.** P = 0.04. **Yaad rakho:** Trigram: teen-word count upar, pehle do-word count neeche. **Why larger n is not automatically better.** More context can capture local order more precisely, but each longer history occurs less often. Parameter count, storage and sparsity grow. A model trained on Shakespeare may assign poor probabilities to financial news because training and test distributions differ. Keep a held-out test set; do not evaluate only on memorized training sentences. --- ## TEE · Laplace smoothing: why both numerator and denominator change Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-smoothing--laplace-basics/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-smoothing/laplace-basics Formulas explained through full worked examples **Smoothing** reallocates some probability mass to events that received zero or unreliable estimates from finite training data. **Laplace, or add-one, smoothing** adds one pseudo-count to every possible next outcome. If there are V possible outcomes, the total added count is V, so adding 1 only to the numerator would produce probabilities that do not sum to 1. $$ P_{\mathrm{Laplace}}(w)=\frac{C(w)+1}{N+V} $$ $$ P_{\mathrm{Laplace}}(w\mid h)=\frac{C(h,w)+1}{C(h)+V} $$ For unigram smoothing, N is the total training token count. For bigram smoothing, C(h) is the count of the particular history. In both cases V is the number of outcomes being predicted, not the number of training sentences and not the number of distinct bigrams. When end is a possible prediction, include it in V; the start marker is usually context only. Observed counts: 3, 1, 0 → Add one to every outcome: 4, 2, 1 → New total: 7 → Smoothed probabilities: 4/7, 2/7, 1/7 **Question. A fixed unigram vocabulary is {a,b,c}. The training counts are 3,1,0. Compute unsmoothed and Laplace-smoothed probabilities.** #### Given C(a)=3, C(b)=1, C(c)=0; N=4; V=3. #### To find The complete distributions. #### Formula and why it applies $$ P(w)=C(w)/N,\qquad P_L(w)=(C(w)+1)/(N+V) $$ The zero-count c is still in the fixed vocabulary, so it must receive one pseudo-count and contribute to V. #### Solution ##### Step 1. Compute unsmoothed values | Word | Unsmoothed | | --- | --- | | a | 3/4 = 0.75 | | b | 1/4 = 0.25 | | c | 0/4 = 0 | ##### Step 2. Calculate the new denominator $$ N+V=4+3=7 $$ ##### Step 3. Add one to each numerator $$ C(a)+1=4,\quad C(b)+1=2,\quad C(c)+1=1 $$ ##### Step 4. Divide | Word | Smoothed | | --- | --- | | a | 4/7 ≈ 0.571429 | | b | 2/7 ≈ 0.285714 | | c | 1/7 ≈ 0.142857 | ##### Step 5. Confirm the total $$ 4/7+2/7+1/7=1 $$ **Final answer.** Smoothed probabilities: a=4/7, b=2/7, c=1/7. Common a loses mass; unseen c gains mass. **Question. After history I, a corpus records like three times, study once and sleep zero times. The allowed next vocabulary is {like,study,sleep}. Compute all Laplace-smoothed bigram probabilities.** #### Given C(I,like)=3; C(I,study)=1; C(I,sleep)=0. History count C(I)=4. V=3. #### To find P\_L(like|I), P\_L(study|I), P\_L(sleep|I). #### Formula and why it applies $$ P_L(w\mid I)=\frac{C(I,w)+1}{C(I)+V} $$ This is smoothing within the I row; use its total 4, not the global corpus token count. #### Solution ##### Step 1. Form the common denominator $$ C(I)+V=4+3=7 $$ ##### Step 2. Calculate like $$ P_L(like\mid I)=(3+1)/7=4/7 $$ ##### Step 3. Calculate study $$ P_L(study\mid I)=(1+1)/7=2/7 $$ ##### Step 4. Calculate sleep $$ P_L(sleep\mid I)=(0+1)/7=1/7 $$ **Final answer.** The row is (4/7, 2/7, 1/7). An unseen bigram is no longer forced to zero. **Question. A fixed vocabulary has five possible next tokens. A history h is unseen in training. What is its add-one probability for each next token?** #### Given C(h)=0; C(h,w)=0 for each outcome; V=5. #### To find P\_L(w|h) for each token. #### Formula and why it applies $$ P_L(w\mid h)=\frac{0+1}{0+V} $$ All outcomes receive equal pseudo-counts because the training data gives no preference for this history. #### Solution ##### Step 1. Add the pseudo-count $$ 0+1=1 $$ ##### Step 2. Form the denominator $$ 0+5=5 $$ ##### Step 3. Divide and sum $$ P_L(w\mid h)=1/5=0.2,\qquad5(0.2)=1 $$ **Final answer.** Every outcome gets 0.2 under this fixed-vocabulary model. **Vocabulary warning.** Add-one does not automatically define probabilities for every word in the world. Unknown words require a declared vocabulary/UNK convention. If a question supplies V, use that stated V rather than inventing additional tokens. **Yaad rakho:** Add-one: upar +1, neeche +V; bigram mein neeche history ka count. --- ## TEE · A complete unseen-bigram question from corpus to final answer Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-smoothing--smoothed-sentence/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-smoothing/smoothed-sentence Formulas explained through full worked examples **Question. Consider the corpus “I love to code in Python. I love to learn new things. I code every day.” Obtain the bigram probabilities needed for “I love Python”. Explain the zero-probability issue, then use Laplace smoothing to score it. Treat each sentence separately, preserve case, remove punctuation, and include an end token among predicted outcomes.** #### Given Three sentences padded with one start/end marker. Distinct words: I, love, to, code, in, Python, learn, new, things, every, day. With , V=12. Start is context only. #### To find Unsmoothed and smoothed P(I love Python | ). #### Formula and why it applies $$ P(w\mid h)=\frac{C(h,w)}{C(h)},\qquad P_L(w\mid h)=\frac{C(h,w)+1}{C(h)+12} $$ The test contains an unseen transition love→Python. Smoothing assigns probability to that transition and renormalizes every required history row. #### Solution ##### Step 1. Count the contexts | History | Count | | --- | --- | | | 3 | | I | 3 | | love | 2 | | Python | 1 | ##### Step 2. Count each required pair | Pair | Count | | --- | --- | | I | 3 | | I love | 2 | | love Python | 0 | | Python | 1 | ##### Step 3. Compute the unsmoothed product $$ P=\frac33\times\frac23\times\frac02\times\frac11=0 $$ Both words love and Python were seen, but their adjacent pair was not. This is an unseen bigram, not an unknown-word case. ##### Step 4. Smooth the first event $$ P_L(I\mid\langle s\rangle)=\frac{3+1}{3+12}=\frac4{15} $$ ##### Step 5. Smooth the second event $$ P_L(love\mid I)=\frac{2+1}{3+12}=\frac3{15} $$ ##### Step 6. Smooth the unseen event $$ P_L(Python\mid love)=\frac{0+1}{2+12}=\frac1{14} $$ ##### Step 7. Smooth the end event $$ P_L(\langle/s\rangle\mid Python)=\frac{1+1}{1+12}=\frac2{13} $$ ##### Step 8. Multiply the smoothed factors $$ P_L=\frac4{15}\times\frac3{15}\times\frac1{14}\times\frac2{13} $$ $$ P_L=\frac{24}{40950}=\frac4{6825}\approx0.000586081 $$ **Final answer.** Unsmoothed P=0. With the stated V=12 and boundaries, Laplace-smoothed P≈0.000586081. If an exam states a different convention—for example excluding boundary tokens from the entire calculation—recompute both V and the set of probability factors consistently. Do not combine a boundary-aware numerator with a boundary-free denominator. --- ## TEE · Adjusted counts and relative discounting Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-smoothing--adjusted-count-discount/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-smoothing/adjusted-count-discount Formulas explained through full worked examples The slides also express smoothing through an **adjusted count** c\*. Instead of writing the smoothed probability with denominator N+V, they rescale its numerator so the denominator can remain N. Algebraically these are equivalent: $$ c^*=(c+1)\frac{N}{N+V},\qquad P_L(w)=\frac{c^*}{N} $$ $$ d=\frac{c^*}{c}\quad(c>0) $$ The relative discount d compares adjusted and original counts. It is undefined when c=0 because division by zero is undefined. Add-one smoothing moves the distribution toward uniformity: frequent events are reduced, but some rare seen events may increase. It is not correct to claim that every nonzero count must decrease under Laplace smoothing. **Question. For N=20 and V=5, a word has original count c=10. Find its adjusted count, smoothed probability and relative discount.** #### Given N=20; V=5; c=10. #### To find c\*, P\_L and d. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\quad P_L=c^*/N,\quad d=c^*/c $$ These expressions rewrite add-one smoothing without changing its probability. #### Solution ##### Step 1. Form the new total $$ N+V=20+5=25 $$ ##### Step 2. Find the adjusted count $$ c^*=(10+1)\frac{20}{25}=11(0.8)=8.8 $$ ##### Step 3. Find the probability $$ P_L=8.8/20=0.44 $$ ##### Step 4. Find the relative discount $$ d=8.8/10=0.88 $$ **Final answer.** c\*=8.8, P\_L=0.44, d=0.88. The original probability 0.5 is reduced. **Question. Under N=20, V=5, a rare word occurs once. Compute its adjusted count and compare the original and smoothed probabilities.** #### Given N=20; V=5; c=1. #### To find c\*, original P and smoothed P\_L. #### Formula and why it applies $$ c^*=(c+1)N/(N+V) $$ The same smoothing rule may increase a rare event's probability. #### Solution ##### Step 1. Original probability $$ P=1/20=0.05 $$ ##### Step 2. Adjusted count $$ c^*=(1+1)(20/25)=1.6 $$ ##### Step 3. Smoothed probability $$ P_L=1.6/20=0.08 $$ **Final answer.** c\*=1.6 and P\_L=0.08, which exceeds the original 0.05. **Question. For N=20 and V=5, an in-vocabulary word has count zero. Find its adjusted count and smoothed probability. Can a relative discount be calculated?** #### Given c=0; N=20; V=5. #### To find c\*, P\_L and whether d exists. #### Formula and why it applies $$ c^*=(c+1)N/(N+V),\qquad P_L=c^*/N $$ The pseudo-count creates positive mass for the unseen vocabulary item. #### Solution ##### Step 1. Adjusted count $$ c^*=(0+1)(20/25)=0.8 $$ ##### Step 2. Smoothed probability $$ P_L=0.8/20=0.04 $$ ##### Step 3. Relative discount d would be 0.8/0. That is undefined, so do not report a finite discount. **Final answer.** c\*=0.8 and P\_L=0.04; relative discount is undefined for c=0. --- ## TEE · Perplexity: meaning, formula and four complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-smoothing--evaluation-perplexity/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-smoothing/evaluation-perplexity Formulas explained through full worked examples **Intrinsic evaluation** evaluates a model directly on a held-out test corpus, for example using perplexity. **Extrinsic evaluation** places it inside a downstream application, such as a speech recognizer or translation system, and measures the application's performance. Extrinsic testing is closer to actual use but often slower and more expensive. A lower perplexity is not a guarantee of improvement on every downstream task. **Perplexity** is the inverse probability of a test sequence, normalized by the number of prediction events. Intuitively, it is the model's effective average number of equally plausible choices. A model that always gives the correct next word probability 1/4 has perplexity 4. Lower is better when the models are tested on the same data, vocabulary, tokenization and boundary convention. $$ \operatorname{PP}(W)=P(W)^{-1/N}=\sqrt[N]{\frac1{P(W)}} $$ | Symbol | Meaning | | --- | --- | | W | The complete test sequence. | | P(W) | Product of the probabilities assigned to the actual next tokens. | | N | Number of scored prediction events under the stated convention. | | Exponent −1/N | Take the reciprocal, then the Nth root. | For calculator entry, use parentheses: P raised to (−1 ÷ N), or (1 ÷ P) raised to (1 ÷ N). Retain several digits in intermediate products. The ordinary start marker is usually given context, not a predicted event; the end marker counts if its probability is included. **Question. A model assigns probability 0.0625 to a test sequence containing four scored tokens. Compute its perplexity.** #### Given P(W)=0.0625=1/16; N=4. #### To find PP(W). #### Formula and why it applies $$ \operatorname{PP}(W)=\left(\frac1{P(W)}\right)^{1/N} $$ The root removes the effect of sequence length; the reciprocal makes lower surprise correspond to lower perplexity. #### Solution ##### Step 1. Take the reciprocal $$ 1/P(W)=1/0.0625=16 $$ ##### Step 2. Use the fourth root $$ \operatorname{PP}=16^{1/4} $$ ##### Step 3. Evaluate $$ 2^4=16\quad\Rightarrow\quad\operatorname{PP}=2 $$ **Final answer.** Perplexity = 2. **Question. Two language models score the same three-token test sequence. Model A gives next-token probabilities 0.5, 0.5, 0.5. Model B gives 0.5, 0.25, 0.25. Compute their perplexities and select the better intrinsic result.** #### Given Same sequence and N=3 for both models. A factors: 0.5,0.5,0.5. B factors: 0.5,0.25,0.25. #### To find Both sequence probabilities and perplexities. #### Formula and why it applies $$ P(W)=\prod_i p_i,\qquad\operatorname{PP}=P(W)^{-1/3} $$ A fair comparison uses identical test tokens and scoring conventions. #### Solution ##### Step 1. Calculate A's probability $$ P_A=0.5\times0.5\times0.5=0.125=1/8 $$ ##### Step 2. Calculate A's perplexity $$ \operatorname{PP}_A=8^{1/3}=2 $$ ##### Step 3. Calculate B's probability $$ P_B=0.5\times0.25\times0.25=0.03125=1/32 $$ ##### Step 4. Calculate B's perplexity $$ \operatorname{PP}_B=32^{1/3}\approx3.174802 $$ ##### Step 5. Compare 2 is lower than 3.174802; A places more probability on the observed sequence. **Final answer.** Model A is better by perplexity on this test: 2 versus about 3.174802. **Question. The bigram training corpus is “This is a dog”, “This is a cat”, “I love my cat”, “This is my name”. Score “This is my cat”, including an end marker, and compute perplexity using five predicted events.** #### Given Each training sentence has one start and end marker. Test predictions: This, is, my, cat, . Thus N=5. #### To find Bigram sentence probability and perplexity. #### Formula and why it applies $$ P(w\mid h)=C(h,w)/C(h),\qquad\operatorname{PP}=P(W)^{-1/5} $$ Start is given context. Four words plus the end marker produce five probability factors. #### Solution ##### Step 1. Count the required transitions | Transition | Pair count / history count | | --- | --- | | →This | 3/4 | | This→is | 3/3 | | is→my | 1/3 | | my→cat | 1/2 | | cat→ | 2/2 | ##### Step 2. Form the product $$ P=\frac34\times1\times\frac13\times\frac12\times1=\frac18 $$ ##### Step 3. Take the reciprocal $$ 1/P=8 $$ ##### Step 4. Take the fifth root $$ \operatorname{PP}=8^{1/5}\approx1.515717 $$ **Final answer.** P(W)=1/8 and perplexity≈1.515717 using five predicted events. **If the question explicitly states N = 6:** use that supplied convention and report PP = 81/6 ≈ 1.414214. The class slide counts both boundary markers in N. Write the chosen N beside the formula; do not silently switch between 5 and 6. Also, This occurs three times and is follows it three times, so P(is|This)=3/3, not 3/4. **Question. A uniform model assigns probability 1/10 to each actual next word in a five-token test. Find its perplexity.** #### Given Five scored tokens, each with assigned probability 0.1. N=5. #### To find The sentence probability and perplexity. #### Formula and why it applies $$ P(W)=(1/10)^5,\qquad \operatorname{PP}=P(W)^{-1/5} $$ Uniform equal probabilities make the effective number of choices especially clear. #### Solution ##### Step 1. Multiply the five factors $$ P(W)=0.1^5=0.00001=10^{-5} $$ ##### Step 2. Invert $$ 1/P(W)=10^5 $$ ##### Step 3. Take the fifth root $$ \operatorname{PP}=(10^5)^{1/5}=10 $$ **Final answer.** Perplexity = 10. **Yaad rakho:** Probability high → surprise low → perplexity low. --- ## TEE · Log probabilities: replace long products with sums Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-smoothing--log-probabilities/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-smoothing/log-probabilities Formulas explained through full worked examples Probabilities are at most 1, so multiplying many of them produces very small numbers. Computers can round extremely small positive products to zero: **numerical underflow**. Log probabilities convert multiplication into addition and are more stable for long sequences. Natural logs use base e; log₂ uses base 2. Choose one base consistently. $$ \ln P(W)=\sum_{i=1}^{N}\ln p_i $$ $$ P(W)=e^{\ln P(W)},\qquad \operatorname{PP}(W)=\exp\left(-\frac1N\sum_i\ln p_i\right) $$ The log of a probability below 1 is negative. A value closer to zero means a larger probability: ln(0.5) ≈ −0.693 is larger than ln(0.1) ≈ −2.303. A zero probability has log −∞; taking logs does not solve the unseen-event problem. Smoothing handles that separate problem. **Question. The next-word probabilities are 0.5, 0.25 and 0.5. Calculate the natural log probability, ordinary probability and perplexity.** #### Given p₁=0.5; p₂=0.25; p₃=0.5; N=3. #### To find ln P(W), P(W), and PP. #### Formula and why it applies $$ \ln P=\sum_i\ln p_i,\quad P=e^{\ln P},\quad\operatorname{PP}=e^{-\ln P/N} $$ Logarithms preserve products as sums and the exponent converts the result back. #### Solution ##### Step 1. Compute individual logs $$ \ln0.5\approx-0.693147,\quad\ln0.25\approx-1.386294 $$ ##### Step 2. Add all three terms $$ \ln P=-0.693147-1.386294-0.693147\approx-2.772589 $$ ##### Step 3. Recover the probability $$ P=e^{-2.772589}\approx0.0625 $$ ##### Step 4. Find average negative log probability $$ -\ln P/3\approx2.772589/3=0.924196 $$ ##### Step 5. Exponentiate $$ \operatorname{PP}=e^{0.924196}\approx2.519842 $$ **Final answer.** ln P≈−2.772589, P=0.0625, PP≈2.519842. **Question. A model has a total natural log probability −12 over six scored tokens. Calculate its perplexity.** #### Given ln P(W)=−12; N=6. #### To find PP without forming the tiny ordinary probability. #### Formula and why it applies $$ \operatorname{PP}=\exp(-\ln P(W)/N) $$ The logarithmic formula directly uses the supplied log score. #### Solution ##### Step 1. Negate the log score $$ -\ln P=12 $$ ##### Step 2. Divide by token count $$ 12/6=2 $$ ##### Step 3. Apply exponential $$ \operatorname{PP}=e^2\approx7.389056 $$ **Final answer.** Perplexity≈7.389056. **Question. A language model's average negative log₂ probability is 3 bits per token. Find its perplexity.** #### Given Cross-entropy H=3 bits/token. Log base is 2. #### To find PP. #### Formula and why it applies $$ \operatorname{PP}=2^H $$ The inverse operation of log₂ is raising 2 to that power; use e only for natural logs. #### Solution ##### Step 1. Identify the log base Bits mean base 2. ##### Step 2. Raise the base to H $$ \operatorname{PP}=2^3=8 $$ **Final answer.** Perplexity = 8. **Yaad rakho:** ln use kiya toh e se wapas; log₂ use kiya toh 2 se wapas. --- ## TEE · Markov chains: read a transition table and multiply a path Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-hmm--markov-chain/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-hmm/markov-chain Formulas explained through full worked examples **A Markov chain** is a probabilistic model of a sequence of states. Under the first-order Markov assumption, the next state's probability depends on the current state rather than the complete previous history. The **initial probability** tells us how likely each first state is. A **transition probability** tells us how likely a move from one state to another is. $$ P(s_t\mid s_1,\ldots,s_{t-1})=P(s_t\mid s_{t-1}) $$ $$ P(s_1,\ldots,s_T)=P(s_1)\prod_{t=2}^{T}P(s_t\mid s_{t-1}) $$ For T states in a path, use one initial factor and T−1 transition factors. A row-oriented transition matrix must have each complete row sum to 1. Always read the row/column labels; some exam diagrams use a transposed arrangement. | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. **Question. For the Rain/Dry Markov chain with initial P(Rain)=0.4 and P(Dry)=0.6, and transitions Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8, find the probability of Dry, Dry, Rain, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Dry,Dry,Rain,Rain). #### Formula and why it applies $$ P(D,D,R,R)=P(D)P(D\mid D)P(R\mid D)P(R\mid R) $$ Each step conditions on the immediately preceding state. The sequence contains four states, so it needs one initial plus three transition factors. #### Solution ##### Step 1. Start in Dry $$ P(D)=0.6 $$ ##### Step 2. Move Dry to Dry $$ 0.6\times0.8=0.48 $$ ##### Step 3. Move Dry to Rain $$ 0.48\times0.2=0.096 $$ ##### Step 4. Move Rain to Rain $$ 0.096\times0.3=0.0288 $$ **Final answer.** Path probability = 0.0288. **Question. For initial probabilities Rain=0.4, Dry=0.6 and transition rows Rain=(0.3,0.7), Dry=(0.2,0.8) in column order Rain,Dry, find the probability of Rain, Dry, Rain.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find P(Rain,Dry,Rain). #### Formula and why it applies $$ P(R,D,R)=P(R)P(D\mid R)P(R\mid D) $$ Do not replace Dry→Rain with Rain→Dry: the two directional probabilities differ. #### Solution ##### Step 1. Initial Rain $$ P(R)=0.4 $$ ##### Step 2. First transition $$ P(D\mid R)=0.7,\quad0.4(0.7)=0.28 $$ ##### Step 3. Second transition $$ P(R\mid D)=0.2,\quad0.28(0.2)=0.056 $$ **Final answer.** Path probability = 0.056. **Question. For the same fully specified chain—initial Rain=0.4, Dry=0.6; Rain→Rain=0.3, Rain→Dry=0.7, Dry→Rain=0.2, Dry→Dry=0.8—find P(Dry,Dry). Also find the conditional probability of next Dry given current Dry.** #### Given | Current → next | Rain | Dry | | --- | --- | --- | | Rain | 0.3 | 0.7 | | Dry | 0.2 | 0.8 | Initial probabilities: P(Rain)=0.4, P(Dry)=0.6. Rows describe the current state; columns describe the next state. #### To find The two-state joint probability and the one-step conditional probability. #### Formula and why it applies $$ P(D,D)=P(D)P(D\mid D) $$ The joint path includes an uncertain initial state. The conditional transition assumes the current state is already Dry. #### Solution ##### Step 1. Read the conditional transition $$ P(D\mid D)=0.8 $$ ##### Step 2. Include the initial state for the joint path $$ P(D,D)=0.6\times0.8=0.48 $$ **Final answer.** P(Dry,Dry)=0.48; P(next Dry|current Dry)=0.8. These answer different questions. **Yaad rakho:** Path: pehle start probability, phir har arrow ki probability. --- ## TEE · Estimate transition and emission probabilities from tagged data Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-hmm--transition-emission-estimation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-hmm/transition-emission-estimation Formulas explained through full worked examples $$ P(t'\mid t)=\frac{C(t,t')}{C(t)\text{ with a following tag/end event}} $$ $$ P(w\mid t)=\frac{C(w,t)}{C(t)} $$ In a transition denominator, count the previous tag's outgoing events. In an emission denominator, count all tokens bearing that tag. With explicit end transitions the totals align conveniently. Keep the two numerator types separate: tag→tag counts are not word→tag counts. **Question. A tagged corpus contains 116454 determiner tags DT, each counted with an outgoing event. DT is followed by NN 56509 times. Estimate P(NN|DT).** #### Given C(DT)=116454; C(DT,NN)=56509. #### To find The DT→NN transition probability. #### Formula and why it applies $$ P(NN\mid DT)=C(DT,NN)/C(DT) $$ The denominator is the number of previous DT contexts because DT is the condition. #### Solution ##### Step 1. Identify the transition count DT followed by NN occurs 56509 times. ##### Step 2. Identify the conditional total There are 116454 DT contexts. ##### Step 3. Divide $$ P(NN\mid DT)=\frac{56509}{116454}\approx0.485247 $$ **Final answer.** P(NN|DT)≈0.485247. **Question. The tag VBZ occurs 21627 times, and 10073 of those tokens are the word “is”. Estimate P(is|VBZ).** #### Given C(VBZ)=21627; C(is,VBZ)=10073. #### To find The emission probability of is from VBZ. #### Formula and why it applies $$ P(is\mid VBZ)=C(is,VBZ)/C(VBZ) $$ The condition is the tag VBZ; we ask which word it emits. #### Solution ##### Step 1. Choose the word-tag count The required word is is with tag VBZ: count 10073. ##### Step 2. Choose the tag total All VBZ-tagged word tokens: count 21627. ##### Step 3. Divide $$ P(is\mid VBZ)=\frac{10073}{21627}\approx0.465760 $$ **Final answer.** P(is|VBZ)≈0.465760. This is different from P(VBZ|is). **Question. Use the tagged sentences “dogs/N chase/V cats/N” and “cats/N sleep/V”. Estimate the initial tag probabilities, all nonzero transitions including END, and emissions.** #### Given Corpus: START dogs/N chase/V cats/N END; START cats/N sleep/V END. #### To find Initial distribution, transition rows and emission rows. #### Formula and why it applies $$ \pi_t=C(START,t)/2,\quad a_{tu}=C(t,u)/C(t),\quad b_t(w)=C(w,t)/C(t) $$ The corpus explicitly tells us the tags, so we can count training frequencies directly. #### Solution ##### Step 1. Count tag tokens N occurs 3 times: dogs, cats, cats. V occurs 2 times: chase, sleep. ##### Step 2. Count sentence starts Both sentences start with N. $$ \pi_N=2/2=1,\quad \pi_V=0/2=0 $$ ##### Step 3. Count transitions | Pair | Count | | --- | --- | | N→V | 2 | | N→END | 1 | | V→N | 1 | | V→END | 1 | ##### Step 4. Normalize transition rows | Current | To N | To V | To END | | --- | --- | --- | --- | | N | 0 | 2/3 | 1/3 | | V | 1/2 | 0 | 1/2 | ##### Step 5. Count and normalize emissions | Tag | dogs | cats | chase | sleep | | --- | --- | --- | --- | --- | | N | 1/3 | 2/3 | 0 | 0 | | V | 0 | 0 | 1/2 | 1/2 | **Final answer.** π=(1,0); transitions and emissions are given in the completed tables. Each complete probability row sums to 1. **Yaad rakho:** Tag→tag table aur tag→word table ko mix mat karna. --- ## TEE · HMM path probabilities: three complete numerical examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-hmm--hot-cold-paths/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-hmm/hot-cold-paths Formulas explained through full worked examples The following model uses hidden weather states Hot (H) and Cold (C). Observations are numbers 1, 2 and 3. These numbers are observation labels, not quantities to multiply directly. Their emission probabilities come from the table. | Initial state | Probability | | --- | --- | | H | 0.8 | | C | 0.2 | | From → to | H | C | | --- | --- | --- | | H | 0.7 | 0.3 | | C | 0.4 | 0.6 | | Emission state | Observation 1 | Observation 2 | Observation 3 | | --- | --- | --- | --- | | H | 0.2 | 0.4 | 0.4 | | C | 0.5 | 0.4 | 0.1 | **Question. An HMM has π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for observations (1,2,3) are H:(0.2,0.4,0.4) and C:(0.5,0.4,0.1). Find the joint probability of hidden path H,C,H and observations 1,3,1.** #### Given Hidden path H→C→H. Observations 1→3→1. Model probabilities are specified in the question. There is no END factor. #### To find P(H,C,H; 1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HC}b_C(3)a_{CH}b_H(1) $$ There are three hidden states and observations: three emissions, two transitions, and one initial probability. #### Solution ##### Step 1. Start Hot and emit 1 $$ 0.8\times0.2=0.16 $$ ##### Step 2. Transition Hot to Cold $$ 0.16\times0.3=0.048 $$ ##### Step 3. Cold emits 3 $$ 0.048\times0.1=0.0048 $$ ##### Step 4. Transition Cold to Hot $$ 0.0048\times0.4=0.00192 $$ ##### Step 5. Hot emits the last 1 $$ 0.00192\times0.2=0.000384 $$ **Final answer.** Joint probability = 0.000384 = 3.84×10⁻⁴. **Question. For π(H)=0.8, π(C)=0.2; H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels (1,2,3), find the joint probability of path H,H,H and observations 1,3,1.** #### Given State path H→H→H, observations 1→3→1; no END factor. #### To find P(H,H,H;1,3,1). #### Formula and why it applies $$ P=\pi_H b_H(1)a_{HH}b_H(3)a_{HH}b_H(1) $$ The state remains Hot; each transition still contributes its probability 0.7. #### Solution ##### Step 1. Initial state and first emission $$ 0.8(0.2)=0.16 $$ ##### Step 2. First self-transition $$ 0.16(0.7)=0.112 $$ ##### Step 3. Emit observation 3 $$ 0.112(0.4)=0.0448 $$ ##### Step 4. Second self-transition $$ 0.0448(0.7)=0.03136 $$ ##### Step 5. Emit final observation 1 $$ 0.03136(0.2)=0.006272 $$ **Final answer.** Joint probability = 0.006272. A self-transition is not automatically probability 1. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6) in next-state order H,C; emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3, find the score of path C,C,H with observations 1,3,1.** #### Given Hidden path C→C→H. Observation sequence 1→3→1. No end-state factor. #### To find The joint path-and-observation probability. #### Formula and why it applies $$ P=\pi_C b_C(1)a_{CC}b_C(3)a_{CH}b_H(1) $$ The initial state is Cold, so use π(C), not π(H). Each emission is selected from the state at the same position. #### Solution ##### Step 1. Start Cold and emit 1 $$ 0.2(0.5)=0.1 $$ ##### Step 2. Stay Cold $$ 0.1(0.6)=0.06 $$ ##### Step 3. Emit 3 from Cold $$ 0.06(0.1)=0.006 $$ ##### Step 4. Move Cold to Hot $$ 0.006(0.4)=0.0024 $$ ##### Step 5. Emit 1 from Hot $$ 0.0024(0.2)=0.00048 $$ **Final answer.** Joint probability = 0.00048. **Yaad rakho:** Har column par: state ka emission. Columns ke beech: transition. --- ## TEE · POS HMM worked corpus: Mary, Jane, can, will and spot Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-hmm--tagging-corpus-case/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-hmm/tagging-corpus-case Formulas explained through full worked examples The class example contains words that can be either names or ordinary verbs/modal auxiliaries. The term is **modal** (M), not “model”. To avoid guessing tags from spelling, the training corpus below supplies them explicitly and lowercases words for the emission vocabulary. | Training sentence | Gold tags | | --- | --- | | mary jane can see will | N N M V N | | spot will see mary | N M V N | | will jane spot mary | M N V N | | mary will pat spot | N M V N | There are 9 noun tokens, 4 modal tokens and 4 verb tokens. Each sentence includes a start and end transition. For nouns, the next event can be another noun, modal, verb or END. | Current → next | N | M | V | END | | --- | --- | --- | --- | --- | | START | 3/4 | 1/4 | 0 | 0 | | N | 1/9 | 3/9 | 1/9 | 4/9 | | M | 1/4 | 0 | 3/4 | 0 | | V | 4/4 | 0 | 0 | 0 | | Tag → word | mary | jane | can | see | will | spot | pat | | --- | --- | --- | --- | --- | --- | --- | --- | | N | 4/9 | 2/9 | 0 | 0 | 1/9 | 2/9 | 0 | | M | 0 | 0 | 1/4 | 0 | 3/4 | 0 | 0 | | V | 0 | 0 | 0 | 2/4 | 0 | 1/4 | 1/4 | **Question. Using the fully tagged four-sentence corpus above, calculate the joint score of “will jane spot mary” assigned tags M,N,V,N, including END.** #### Given Required factors from the tables: START→M=1/4; emit will|M=3/4; M→N=1/4; jane|N=2/9; N→V=1/9; spot|V=1/4; V→N=1; mary|N=4/9; N→END=4/9. #### To find The specified tag-path score, including words and sentence end. #### Formula and why it applies $$ P=\pi_M b_M(will)a_{MN}b_N(jane)a_{NV}b_V(spot)a_{VN}b_N(mary)a_{N,END} $$ Both transitions and emissions are needed. A tag transition alone cannot tell us whether a particular word is plausible. #### Solution ##### Step 1. Start and emit will $$ \frac14\times\frac34=\frac3{16} $$ ##### Step 2. Move to N and emit jane $$ \frac3{16}\times\frac14\times\frac29=\frac1{96} $$ ##### Step 3. Move to V and emit spot $$ \frac1{96}\times\frac19\times\frac14=\frac1{3456} $$ ##### Step 4. Move to N and emit mary $$ \frac1{3456}\times1\times\frac49=\frac1{7776} $$ ##### Step 5. End the sentence $$ \frac1{7776}\times\frac49=\frac1{17496}\approx0.0000571559 $$ **Final answer.** The joint score is 1/17496 ≈ 0.0000571559. **Question. For the same tagged corpus and tables, explain the unsmoothed score of “will can spot mary” assigned tags M,V,N,N.** #### Given The proposed second word is can with hidden tag V. In the training corpus, can occurs only with M, so b\_V(can)=0. #### To find Whether the complete path score is zero and what this means. #### Formula and why it applies $$ P(\text{path,words})=(\text{other required factors})\times b_V(can) $$ A product containing one zero factor is zero even if every other factor is positive. #### Solution ##### Step 1. Find the critical emission $$ b_V(can)=C(can,V)/C(V)=0/4=0 $$ ##### Step 2. Use the full-product consequence $$ P(\text{path,words})=0 $$ ##### Step 3. Interpret correctly This unsmoothed corpus model has no evidence for can as a verb. It does not prove that a verb usage of can is impossible in English. **Final answer.** The unsmoothed path score is zero. --- ## TEE · Why exhaustive search grows exponentially Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-hmm--sequence-count/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-hmm/sequence-count Formulas explained through full worked examples If each of T observed words can receive any of N tags, there are N choices at each position. Thus the total number of complete tag sequences is N multiplied by itself T times. This counts possible assignments before grammar rules or zero probabilities eliminate any. $$ \text{Number of tag sequences}=N^T $$ **Question. A four-word sentence is tagged using three possible POS tags per word. How many complete tag sequences are possible?** #### Given N=3 possible tags per position; T=4 word positions. #### To find The number of tag sequences. #### Formula and why it applies $$ M=N^T $$ Every independent position contributes a factor of three choices. #### Solution ##### Step 1. Write one factor per position $$ M=3\times3\times3\times3 $$ ##### Step 2. Multiply $$ 3\times3=9,\quad9\times3=27,\quad27\times3=81 $$ **Final answer.** 81 possible tag sequences. **Question. A five-word sentence has two candidate tags per word. Count its tag sequences.** #### Given N=2; T=5. #### To find M. #### Formula and why it applies $$ M=N^T $$ The length is the exponent, not a multiplier. #### Solution ##### Step 1. Substitute $$ M=2^5 $$ ##### Step 2. Evaluate $$ 2\times2\times2\times2\times2=32 $$ **Final answer.** 32 possible sequences. **Question. Using 45 possible tags, compare the number of tag sequences for sentences of length 2 and 3.** #### Given N=45. First T=2, then T=3. #### To find Both counts and the growth factor. #### Formula and why it applies $$ M_2=45^2,\quad M_3=45^3 $$ Adding one position multiplies the possibilities by the number of tag choices. #### Solution ##### Step 1. Two words $$ 45^2=2025 $$ ##### Step 2. Three words $$ 45^3=2025\times45=91125 $$ ##### Step 3. Compare $$ 91125/2025=45 $$ **Final answer.** 2 words: 2025 paths. 3 words: 91125 paths. One additional word multiplies the count by 45. The slides introduce Viterbi to avoid enumerating every path. It reuses the best prefix score for each ending state. For a dense first-order HMM, it needs O(TN²) recurrence work instead of enumerating NT complete paths. **Yaad rakho:** Choices per word ki power = sentence length. --- ## TEE · Viterbi decoding: compare incoming paths, keep the best, trace back Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-hmm--viterbi-explained/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-hmm/viterbi-explained Formulas explained through full worked examples **The Viterbi algorithm** finds the most likely hidden-state sequence for a fixed observation sequence. At each position, it keeps the best path ending in each state. This is safe because, under the HMM assumptions, future extensions depend on that ending state, not the discarded details of the prefix. $$ \delta_1(j)=\pi_jb_j(o_1) $$ $$ \delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ $$ \psi_t(j)=\arg\max_i[\delta_{t-1}(i)a_{ij}] $$ | Symbol | Meaning in ordinary words | | --- | --- | | δₜ(j) | Score of the best prefix that ends in state j at position t. | | ψₜ(j) | Previous state that supplied that best prefix; keep it as a backpointer. | | max | Choose the larger candidate score; do not add. | | argmax | Remember which previous state won, rather than only its value. | Initialize: start × first emission → For each next state: compare incoming scores → Multiply winner by this state's emission → Keep score + winning predecessor → Choose best final state and trace arrows backward Do not choose the best state independently at each word. The best last state can come from a different best predecessor than a local guess. Backpointers recover one consistent complete path. If scores tie, either tied path is optimal under the model; state the tie. **Question. Use an HMM with π(H)=0.8, π(C)=0.2, transitions H:(0.7,0.3), C:(0.4,0.6) in H,C order, and emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for observations 1,2,3. Find the most likely hidden path for observations 1,3,1 using Viterbi.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find The best hidden-state sequence and its joint score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(o_1),\quad\delta_t(j)=b_j(o_t)\max_i[\delta_{t-1}(i)a_{ij}] $$ Viterbi keeps two best prefix scores per column, one ending H and one ending C. #### Solution ##### Step 1. Initialize at observation 1 $$ \delta_1(H)=0.8(0.2)=0.16 $$ $$ \delta_1(C)=0.2(0.5)=0.10 $$ ##### Step 2. At observation 3, calculate the H cell $$ H\to H:\ 0.16(0.7)=0.112 $$ $$ C\to H:\ 0.10(0.4)=0.040 $$ $$ \delta_2(H)=\max(0.112,0.040)(0.4)=0.0448 $$ Backpointer for this H cell: H. ##### Step 3. At observation 3, calculate the C cell $$ H\to C:\ 0.16(0.3)=0.048 $$ $$ C\to C:\ 0.10(0.6)=0.060 $$ $$ \delta_2(C)=\max(0.048,0.060)(0.1)=0.006 $$ Backpointer for this C cell: C. ##### Step 4. At final observation 1, calculate H $$ H\to H:\ 0.0448(0.7)=0.03136 $$ $$ C\to H:\ 0.006(0.4)=0.0024 $$ $$ \delta_3(H)=0.03136(0.2)=0.006272 $$ Backpointer: H. ##### Step 5. At final observation 1, calculate C $$ H\to C:\ 0.0448(0.3)=0.01344 $$ $$ C\to C:\ 0.006(0.6)=0.0036 $$ $$ \delta_3(C)=0.01344(0.5)=0.00672 $$ Backpointer: H. ##### Step 6. Select the final state 0.00672 > 0.006272, so the best final state is C. ##### Step 7. Trace back Final C came from H at position2. That H came from H at position1. Reading forward gives H,H,C. ##### Step 8. Verify the recovered complete path $$ 0.8(0.2)(0.7)(0.4)(0.3)(0.5)=0.00672 $$ **Final answer.** Best path H→H→C with joint score 0.00672. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 3,1.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best two-state hidden path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(3),\quad\delta_2(j)=b_j(1)\max_i[\delta_1(i)a_{ij}] $$ The same recurrence applies even though the observation order changed. #### Solution ##### Step 1. Initialize observation 3 $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.1)=0.02 $$ ##### Step 2. Calculate the final H $$ \max(0.32(0.7),0.02(0.4))(0.2) $$ $$ =\max(0.224,0.008)(0.2)=0.0448 $$ Winning predecessor H. ##### Step 3. Calculate the final C $$ \max(0.32(0.3),0.02(0.6))(0.5) $$ $$ =\max(0.096,0.012)(0.5)=0.048 $$ Winning predecessor H. ##### Step 4. Select and trace Final C wins with 0.048 and came from H. The path is H,C. **Final answer.** Best path H→C, score 0.048. **Question. For π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3, decode observations 2,2.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find Best path and score. #### Formula and why it applies $$ \delta_1(j)=\pi_jb_j(2),\quad\delta_2(j)=b_j(2)\max_i[\delta_1(i)a_{ij}] $$ Both states emit2 with probability 0.4, but their initial and transition probabilities still differ. #### Solution ##### Step 1. Initialize $$ \delta_1(H)=0.8(0.4)=0.32,\quad\delta_1(C)=0.2(0.4)=0.08 $$ ##### Step 2. Final H $$ \max(0.32(0.7),0.08(0.4))(0.4) $$ $$ =\max(0.224,0.032)(0.4)=0.0896 $$ Predecessor H. ##### Step 3. Final C $$ \max(0.32(0.3),0.08(0.6))(0.4) $$ $$ =\max(0.096,0.048)(0.4)=0.0384 $$ Predecessor H. ##### Step 4. Select and trace The final H score 0.0896 is larger. Its predecessor is H. **Final answer.** Best path H→H with score 0.0896. **Yaad rakho:** Har destination ke liye incoming paths compare; winner ka arrow yaad. --- ## TEE · A path score is not the probability of the observed sentence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-hmm--joint-vs-posterior/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-hmm/joint-vs-posterior Formulas explained through full worked examples **Question. An HMM has π(H)=0.8, π(C)=0.2; transitions H:(0.7,0.3), C:(0.4,0.6); emissions H:(0.2,0.4,0.4), C:(0.5,0.4,0.1) for labels1,2,3. For observations 1,3, calculate the probability of the observations by summing all four two-state paths.** #### Given Initial probabilities: π(H)=0.8, π(C)=0.2. Transition matrix: H→H=0.7, H→C=0.3, C→H=0.4, C→C=0.6. Emissions for labels (1,2,3): H=(0.2,0.4,0.4), C=(0.5,0.4,0.1). There is no END-state factor. #### To find P(observations 1,3), not just the best-path score. #### Formula and why it applies $$ P(o_1,o_2)=\sum_{s_1,s_2}\pi_{s_1}b_{s_1}(o_1)a_{s_1s_2}b_{s_2}(o_2) $$ The hidden path is unspecified, so its mutually exclusive alternatives must be summed. #### Solution ##### Step 1. Score H,H $$ 0.8(0.2)(0.7)(0.4)=0.0448 $$ ##### Step 2. Score H,C $$ 0.8(0.2)(0.3)(0.1)=0.0048 $$ ##### Step 3. Score C,H $$ 0.2(0.5)(0.4)(0.4)=0.016 $$ ##### Step 4. Score C,C $$ 0.2(0.5)(0.6)(0.1)=0.006 $$ ##### Step 5. Add all paths $$ P(1,3)=0.0448+0.0048+0.016+0.006=0.0716 $$ **Final answer.** Observation probability =0.0716. The best single-path score 0.0448 is a different quantity. If asked for the posterior of path H,H given observations 1,3, normalize: 0.0448/0.0716 ≈0.625698. Viterbi chooses the best path using joint scores because the observation denominator is common to every candidate. It does not compute that denominator by taking a maximum. --- ## TEE · Lesk algorithm, from words to overlap scores Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-wsd--u5-lesk/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-wsd/u5-lesk Formulas explained through full worked examples **Lesk is a knowledge-based WSD method that chooses senses whose dictionary descriptions have the greatest overlap with contextual evidence.** Original Lesk compares definitions of neighbouring words' senses. A common simplified version compares each target-sense gloss directly with the sentence context. State which version you use. ### Simplified Lesk procedure 1. Remove the target word from the context; tokenize and normalize the remaining content words. 2. Obtain the gloss for each candidate sense and apply the same normalization. 3. Count shared words between the context and each gloss. 4. Select the sense with maximum overlap. If scores tie, report the tie or use a specified tie-break rule. $$ \operatorname{score}(s)=|C\cap G_s|,\qquad s^*=\arg\max_s\operatorname{score}(s) $$ $$ C $$ is the set of context words, $$ G_s $$ the set of words in sense $$ s $$ 's gloss, $$ \cap $$ means common elements, $$ |\cdot| $$ means count, and $$ \arg\max $$ means “the sense giving the largest score”. This version counts unique matching words once. **Question. Use simplified Lesk to disambiguate “bank” in “The river water rose near the bank.” Use C = {river, water, rose, near}; financial gloss = {institution, money, deposit}; river gloss = {land, river, water, edge}.** #### Given The normalized context and two gloss sets are given. Count exact shared words. #### To find The score of each sense and the selected meaning. #### Formula and why it applies Use $$ \operatorname{score}(s)=|C\cap G_s| $$ because the task specifies context–gloss overlap. #### Step 1. Compare the financial sense No word occurs in both {river, water, rose, near} and {institution, money, deposit}. Overlap = {}. #### Step 2. Count the financial score $$ \operatorname{score}(F)=0 $$ . #### Step 3. Compare the river sense Shared words are {river, water}. “Rose” and “near” are not in the gloss. #### Step 4. Count and select $$ \operatorname{score}(R)=2 $$ . Since $$ 2>0 $$ , choose the river-edge sense. **Answer.** Bank means land beside a river. **Question. Use simplified Lesk for “The dog barked loudly.” Treat barked as bark. Context C = {dog, loudly}; sound gloss = {sharp, cry, dog}; tree gloss = {outer, covering, tree}; ship gloss = {sailing, ship}.** #### Given Three glosses and normalized context {dog, loudly}. #### To find Select the sense of bark. #### Formula and why it applies $$ \operatorname{score}(s)=|C\cap G_s| $$ . Lemmatization lets barked match the target lemma bark; it does not create additional overlap words. #### Step 1. Sound sense Intersection = {dog}; score = 1. #### Step 2. Tree sense Intersection = {}; score = 0. #### Step 3. Ship sense Intersection = {}; score = 0. #### Step 4. Compare The sound sense has the unique maximum: 1 > 0. **Answer.** Bark is the dog's cry. **Question. Apply definition–definition Lesk to “pine cone”. Pine senses: P1 = {evergreen, tree, needle, leaf}; P2 = {waste, sorrow, illness}. Cone senses: C1 = {solid, shape, circular}; C2 = {fruit, evergreen, tree}.** #### Given Two candidate senses for each word. Tokens are already normalized; tree and trees would be reduced to the same form if necessary. #### To find The most compatible pair of senses. #### Formula and why it applies Use $$ |G_{P_i}\cap G_{C_j}| $$ for every pair because this is original-style gloss–gloss comparison. #### Step 1. P1 with C1 No shared word → 0. #### Step 2. P1 with C2 Shared {evergreen, tree} → 2. #### Step 3. P2 with each cone sense P2–C1 has no overlap → 0; P2–C2 has no overlap → 0. #### Step 4. Choose the best pair The four scores are 0, 2, 0, 0. Maximum 2 belongs to P1–C2. **Answer.** Pine is the evergreen tree; cone is its fruit-like reproductive structure. **Limitations:** exact overlap misses synonyms, short glosses may produce zeros, and different preprocessing choices change the score. Extended Lesk may include related synset definitions or longer phrase overlaps; do not silently substitute it for a simple question. Lesk: context aur dictionary mein common words gino; sabse bada overlap jeeta. --- ## TEE · Thesaurus / Walker approach, with three complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-wsd--u5-walker/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-wsd/u5-walker Formulas explained through full worked examples **The Walker approach uses thesaurus categories as semantic evidence.** Associate each candidate sense with a category. Count or weight context words that support that category. Choose the largest total. $$ \operatorname{score}(s)=\sum_{w\in C} I(w,\operatorname{category}(s)) $$ $$ I=1 $$ if the supplied category mapping says the context word supports the sense, otherwise $$ I=0 $$ . The summation symbol means add all those contributions. Weighted questions may replace 1 with a supplied weight. Category membership must be supplied or justified; it is not guessed from word frequency. **Question. Disambiguate bank using Walker scoring. Context words are money, interest, annum, fetch. The finance category supports money, interest and annum; the location category supports none of these words.** #### Given Two categories: Finance and Location. Each supported word contributes 1. #### To find The two totals and selected bank sense. #### Formula and why it applies Sum the category indicators for each sense. #### Step 1. Make a contribution table | Word | Finance | Location | | --- | --- | --- | | money | 1 | 0 | | interest | 1 | 0 | | annum | 1 | 0 | | fetch | 0 | 0 | #### Step 2. Add Finance $$ 1+1+1+0=3 $$ . #### Step 3. Add Location $$ 0+0+0+0=0 $$ . #### Step 4. Select 3 > 0, so select the financial institution. **Answer.** Finance, with score 3 versus 0. **Question. Use Walker scoring for “The fisherman caught bass beside the lake.” Context {fisherman,caught,lake}. The Fish category contains all three clues; the Music category contains none.** #### Given Fish and Music senses, binary category membership given. #### To find The correct bass sense. #### Formula and why it applies Score = sum of context-category matches. #### Step 1. Record Fish matches fisherman → 1; caught → 1; lake → 1. #### Step 2. Add Fish score $$ 1+1+1=3 $$ . #### Step 3. Add Music score $$ 0+0+0=0 $$ . #### Step 4. Choose Fish has the larger score. **Answer.** Bass refers to a fish. **Question. Score “The player used a bat in the match.” Context {player,match}. Sport supports both words with weights 2 and 1; Animal supports neither. Use the supplied weighted Walker scheme.** #### Given Sport weights {player:2, match:1}; Animal weights {player:0, match:0}. #### To find Weighted sense scores. #### Formula and why it applies Use the sum of supplied weights, not an unweighted count, because the question explicitly gives weights. #### Step 1. Sport contribution of player Add 2. #### Step 2. Sport contribution of match Add 1. Sport total = 2 + 1 = 3. #### Step 3. Animal total 0 + 0 = 0. #### Step 4. Compare 3 > 0 selects the sporting implement. **Answer.** Bat is the sports implement. Walker: dictionary ke exact words nahi, context kis topic/category ko support karta hai woh count. --- ## TEE · Multinomial Naive Bayes: the exact solving method Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-wsd--u5-bayes-steps/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-wsd/u5-bayes-steps Formulas explained through full worked examples Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. ### Always keep these counts separate Class priors use **number of examples**. Word likelihoods in this multinomial model use **number of word occurrences**. The vocabulary counts **unique words across all classes**. 1. List the class examples and calculate priors. 2. Write one shared vocabulary and count its size. 3. Count total word occurrences in each class. 4. Count each query word in each class. 5. Apply the given smoothing rule. 6. Multiply each prior by all its query likelihoods. 7. Compare scores; divide each score by their sum only if a normalized posterior is requested. $$ P(s\mid C)=\frac{\operatorname{score}(s)}{\sum_{s'}\operatorname{score}(s')}. $$ **Why smoothing is needed:** an unseen clue gives $$ 0/N_s=0 $$ ; one zero makes the whole product zero. Add-one smoothing redistributes probability across every vocabulary word and prevents these zero factors. Use it only when requested, taught for the task, or state the assumption. **Different model warning:** a Bernoulli Naive Bayes question counts presence/absence per document and may include absence factors. Do not use its denominator in a multinomial word-count question. Pehle sentences se prior; phir words se likelihood; dono counts mix mat kar. --- ## TEE · Mam's bank numerical—every count and every multiplication Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-wsd--u5-bayes-mam/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-wsd/u5-bayes-mam Formulas explained through full worked examples **Question. Classify the sense of “bank” using the training table below. The sentence is: “As the stream flowed gently past the shore, we set up camp by the bank of the river, discussing the possibility of borrowing a loan to buy a boat for future trips.” Use only the highlighted feature words stream, shore and loan, and add-one multinomial smoothing.** #### Given | Training row | Context words | Sense | | --- | --- | --- | | 1 | money, deposit, loan | Finance | | 2 | river, water, shore | Riverbank | | 3 | account, savings, investment | Finance | | 4 | boat, river, stream | Riverbank | | 5 | branch, deposit, check | Finance | | 6 | fishing, shore, river | Riverbank | Query feature set: {stream, shore, loan}. Other sentence words are not included as features in this supplied example. #### To find The most probable bank sense and normalized posteriors. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Calculate priors Finance appears in 3 of the 6 rows; Riverbank appears in 3 of 6. $$ P(F)=\frac36=\frac12,\qquad P(R)=\frac36=\frac12. $$ #### Step 2. Construct the vocabulary {money, deposit, loan, river, water, shore, account, savings, investment, boat, stream, fishing, branch, check}. Thus V=14. Repeated deposit or river is counted once in vocabulary. #### Step 3. Count word occurrences per class There are 3 finance rows × 3 words = 9 word occurrences. There are 3 river rows × 3 words = 9. Thus $$ N_F=N_R=9 $$ , and each smoothed denominator is $$ 9+14=23 $$ . #### Step 4. Count the three query words | Word | Finance count | Riverbank count | | --- | --- | --- | | stream | 0 | 1 | | shore | 0 | 2 | | loan | 1 | 0 | #### Step 5. Calculate finance likelihoods $$ P(\text{stream}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{shore}\mid F)=\frac{0+1}{23}=\frac1{23} $$ $$ P(\text{loan}\mid F)=\frac{1+1}{23}=\frac2{23} $$ #### Step 6. Multiply the finance score $$ \operatorname{score}(F)=\frac12\times\frac1{23}\times\frac1{23}\times\frac2{23}=\frac1{23^3}=\frac1{12167}\approx0.00008219. $$ #### Step 7. Calculate river likelihoods $$ P(\text{stream}\mid R)=\frac{1+1}{23}=\frac2{23} $$ $$ P(\text{shore}\mid R)=\frac{2+1}{23}=\frac3{23} $$ $$ P(\text{loan}\mid R)=\frac{0+1}{23}=\frac1{23} $$ #### Step 8. Multiply the river score $$ \operatorname{score}(R)=\frac12\times\frac2{23}\times\frac3{23}\times\frac1{23}=\frac3{12167}\approx0.00024657. $$ #### Step 9. Compare scores 3/12167 is larger than 1/12167. Riverbank wins even though “loan” is financial evidence: the combined clues matter. #### Step 10. Normalize if a posterior is requested $$ P(F\mid C)=\frac{1/12167}{4/12167}=\frac14=0.25 $$ $$ P(R\mid C)=\frac{3/12167}{4/12167}=\frac34=0.75. $$ **Answer.** Bank is the riverbank sense. Scores are 1/12167 and 3/12167; normalized model posteriors are 0.25 and 0.75. **Without smoothing:** stream and shore are absent in Finance; loan is absent in Riverbank. Both unsmoothed scores become zero, so there is no valid winning class from those products. Also, unnormalized scores should not be labelled as final posterior probabilities. Zero word? Add-one: count+1 upar, total words+vocabulary neeche. Last mein score bada wala sense. --- ## TEE · More WSD numericals: given likelihoods, unequal priors and repeated clues Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-wsd--u5-bayes-practice/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-wsd/u5-bayes-practice Formulas explained through full worked examples **Question. For bass, let P(Fish)=0.6 and P(Music)=0.4. Given Fish likelihoods P(caught|Fish)=0.5, P(lake|Fish)=0.4 and Music likelihoods P(caught|Music)=0.05, P(lake|Music)=0.1, classify context {caught,lake} and normalize.** #### Given Priors and likelihoods are provided directly; no smoothing or training-count estimation is requested. #### To find The winning sense and its posterior. #### Formula and why it applies $$ \operatorname{score}(s)=P(s)P(\text{caught}\mid s)P(\text{lake}\mid s) $$ , because the two features are treated as conditionally independent. #### Step 1. Fish multiplication $$ 0.6\times0.5=0.3 $$ , then $$ 0.3\times0.4=0.12 $$ . #### Step 2. Music multiplication $$ 0.4\times0.05=0.02 $$ , then $$ 0.02\times0.1=0.002 $$ . #### Step 3. Total evidence $$ 0.12+0.002=0.122 $$ . #### Step 4. Normalize $$ P(Fish\mid C)=0.12/0.122\approx0.983607 $$ $$ P(Music\mid C)=0.002/0.122\approx0.016393. $$ **Answer.** Fish, with model posterior approximately 0.983607. **Question. Disambiguate bank from the repeated context tokens {loan, loan, river}. Priors are Finance=2/3 and River=1/3. Vocabulary V=4. Finance has N=8 total tokens, loan count 3 and river count 0; River has N=4, loan count 0 and river count 2. Apply add-one smoothing.** #### Given Multinomial counts are supplied. The two appearances of loan must both contribute. #### To find Scores and normalized Finance posterior. #### Formula and why it applies Naive Bayes selects the sense with the largest prior × likelihood product. The context evidence denominator is identical across candidate senses, so it can be omitted for comparison. $$ \operatorname{score}(s)=P(s)\prod_j P(w_j\mid s). $$ For add-one multinomial smoothing: $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V}. $$ $$ C(w,s) $$ counts occurrences of word $$ w $$ in training contexts of sense $$ s $$ ; $$ N_s $$ is the total number of word occurrences in those contexts; $$ V $$ is the shared vocabulary size. It is not the number of training sentences. Repeated query words contribute repeated factors. #### Step 1. Finance denominator $$ 8+4=12 $$ . $$ P(loan\mid F)=(3+1)/12=1/3 $$ ; $$ P(river\mid F)=(0+1)/12=1/12 $$ . #### Step 2. River denominator $$ 4+4=8 $$ . $$ P(loan\mid R)=(0+1)/8=1/8 $$ ; $$ P(river\mid R)=(2+1)/8=3/8 $$ . #### Step 3. Finance product $$ \operatorname{score}(F)=\frac23\times\frac13\times\frac13\times\frac1{12}=\frac1{162}\approx0.00617284. $$ #### Step 4. River product $$ \operatorname{score}(R)=\frac13\times\frac18\times\frac18\times\frac38=\frac1{512}\approx0.001953125. $$ #### Step 5. Compare and normalize $$ P(F\mid C)=\frac{1/162}{1/162+1/512}=\frac{512}{674}=\frac{256}{337}\approx0.759644. $$ **Answer.** Finance wins. Repeated loan contributes a squared likelihood; do not reduce the query to a unique-word set. ### Answer-writing order in an exam Write the class priors, vocabulary size, class token totals, a likelihood table, both score products and the final class. Keep fractions until the comparison where possible. If the question only asks classification, normalization is optional; if it asks posterior probabilities, normalization is required. Repeated word aaye toh factor bhi repeat. Sirf word-set bana diya toh multinomial answer badal jayega. --- ## TEE · Label and one-hot encoding: three complete examples Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u2-label-onehot/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u2-label-onehot Formulas explained through full worked examples **Label encoding** assigns each category or word a unique integer ID. **One-hot encoding** assigns a length- $$ V $$ vector with one 1 at the word's vocabulary position and zeros elsewhere. $$ \operatorname{onehot}(w)_j=\begin{cases}1,&j=\operatorname{index}(w)\\0,&\text{otherwise.}\end{cases} $$ The index is an ID, not a meaning score. One-hot represents one token/category; a multiword document generally needs multiple token vectors or an aggregation scheme. **Question. Encode “dog cat dog” using vocabulary [cat,dog,fish], IDs starting at 0. Give label IDs and a one-hot vector for every token.** #### Given cat→0, dog→1, fish→2. Vector columns have the same order. #### To find The ID sequence and three one-hot rows. #### Formula and why it applies Replace each word by its ID for labels; set its vocabulary position to 1 for one-hot. #### Step 1. Label IDs dog→1, cat→0, dog→1 → [1,0,1]. #### Step 2. First dog [0,1,0]. #### Step 3. Cat [1,0,0]. #### Step 4. Second dog [0,1,0]. Repeated tokens keep the same representation. **Answer.** IDs [1,0,1]; one-hot rows [0,1,0], [1,0,0], [0,1,0]. **Question. Encode the city sequence Mumbai, Delhi, Mumbai, Pune using vocabulary [Delhi,Mumbai,Pune] and label IDs starting at 1.** #### Given Delhi→1, Mumbai→2, Pune→3. #### To find IDs and one-hot rows. #### Formula and why it applies One-hot uses three coordinates regardless of whether label IDs begin at 0 or 1. #### Step 1. IDs [2,1,2,3]. #### Step 2. Mumbai rows Each Mumbai → [0,1,0]. #### Step 3. Delhi and Pune Delhi→[1,0,0]; Pune→[0,0,1]. **Answer.** Rows [0,1,0], [1,0,0], [0,1,0], [0,0,1]. **Question. One-hot encode categories [red,green,blue] for a linear regression with an intercept. Then give dummy coding after dropping blue.** #### Given Full column order [red,green,blue]. Blue is the baseline. #### To find Full and reduced codes. #### Formula and why it applies Full one-hot rows sum to 1. With an intercept this creates a linear dependency; dropping one category column gives k−1 independent dummy indicators. #### Step 1. Full encoding | Category | red | green | blue | | --- | --- | --- | --- | | red | 1 | 0 | 0 | | green | 0 | 1 | 0 | | blue | 0 | 0 | 1 | #### Step 2. Drop blue column | Category | red | green | | --- | --- | --- | | red | 1 | 0 | | green | 0 | 1 | | blue | 0 | 0 | #### Step 3. Interpret zero row [0,0] now identifies the baseline blue, not missing data. **Answer.** Full width 3; reduced dummy width 2. Dropping a column is model-dependent, not a requirement for every neural embedding input. **Limitations:** one-hot is large and sparse and does not express semantic similarity. Unseen words need a defined unknown-token or vocabulary-update policy. Label IDs are compact but should not be treated as an ordered scalar feature for nominal words. Label = ID number. One-hot = us ID ke box mein 1, baaki 0. --- ## TEE · Bag of words and bag of n-grams Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u2-bow/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u2-bow Formulas explained through full worked examples **Bag of words (BoW) represents a document by the counts of vocabulary terms, discarding their positions.** A binary BoW variant stores only presence/absence; a count BoW stores frequencies. State which is used. $$ x_{d,j}=C(v_j,d). $$ $$ x_{d,j} $$ is document $$ d $$ 's coordinate for vocabulary term $$ v_j $$ . $$ C(v_j,d) $$ is its occurrence count. A **bag of n-grams** uses short contiguous token sequences as features. Bigrams can distinguish “not good” from “good”, but increasing n enlarges the vocabulary and creates more rare features. **Question. Build count and binary BoW for D1=“cat cat dog” and D2=“dog fish”. Use vocabulary [cat,dog,fish] and no stopword removal.** #### Given Two tokenized documents, fixed vocabulary order. #### To find Count vectors and binary vectors. #### Formula and why it applies Count BoW uses $$ C(v_j,d) $$ . Binary BoW is 1 when that count is positive and 0 otherwise. #### Step 1. D1 counts cat=2, dog=1, fish=0 → [2,1,0]. #### Step 2. D2 counts cat=0, dog=1, fish=1 → [0,1,1]. #### Step 3. Convert to presence D1 [1,1,0]; D2 [0,1,1]. **Answer.** Count rows [2,1,0], [0,1,1]; binary rows [1,1,0], [0,1,1]. **Question. After lowercasing and removing {is,a,means}, obtain BoW vectors for D1=“Continuous learning is learning lifetime”, D2=“Continuous learning means lifetime learning”, D3=“Learning is a good practice”. Use [continuous,good,learning,lifetime,practice].** #### Given Three documents and exact stopword set from the supplied slides. #### To find The document–term count matrix. #### Formula and why it applies Remove only the listed stopwords, then count every remaining token in the fixed column order. #### Step 1. Clean D1 [continuous,learning,learning,lifetime] → [1,0,2,1,0]. #### Step 2. Clean D2 [continuous,learning,lifetime,learning] → [1,0,2,1,0]. #### Step 3. Clean D3 [learning,good,practice] → [0,1,1,0,1]. #### Step 4. Compare D1 and D2 They have identical BoW vectors despite different original wording/order. **Answer.** Rows [1,0,2,1,0], [1,0,2,1,0], [0,1,1,0,1]. **Question. Build a bag of bigrams for “not good but not bad” with vocabulary [not good,good but,but not,not bad]. Do not add sentence boundary tokens.** #### Given Five tokens; use adjacent pairs, stride 1. #### To find The bigram count vector. #### Formula and why it applies A sequence of $$ L $$ tokens has $$ L-1 $$ adjacent bigrams when no boundaries are added. #### Step 1. Enumerate pairs (not,good), (good,but), (but,not), (not,bad). #### Step 2. Count each vocabulary bigram Each appears once. #### Step 3. Write vector [1,1,1,1]. **Answer.** Four bigram features, each with count 1. BoW: order bhool, word-count rakh. Bag of bigrams: neighbouring word-pair count. --- ## TEE · TF–IDF from the exact slide convention Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u2-tfidf/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u2-tfidf Formulas explained through full worked examples **TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus.** The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency: $$ \operatorname{TF}(t,d)=\frac{C(t,d)}{N_d} $$ $$ \operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)} $$ $$ \operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t). $$ $$ N_d $$ is the document's token count after the specified preprocessing; $$ D $$ is number of documents; $$ \operatorname{df}(t) $$ is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency. **Why the log?** It compresses the ratio's scale. If a term occurs in all documents, $$ D/df=1 $$ , so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default. **Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.** #### Given D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10. #### To find The TF–IDF weight. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ , then multiply. #### Step 1. TF $$ 3/12=0.25 $$ . #### Step 2. Document ratio $$ 10/2=5 $$ . #### Step 3. IDF $$ \log_{10}5\approx0.698970 $$ . #### Step 4. Weight $$ 0.25\times0.698970\approx0.174743 $$ . **Answer.** TF=0.25, IDF≈0.698970, TF–IDF≈0.174743. **Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.** #### Given C=4, Nd=20, D=8, df=8. #### To find Its weight under unsmoothed base-10 IDF. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ . #### Step 1. TF $$ 4/20=0.2 $$ . #### Step 2. IDF $$ \log_{10}(8/8)=\log_{10}1=0 $$ . #### Step 3. Multiply $$ 0.2\times0=0 $$ . **Answer.** Weight 0: this term does not distinguish documents under this convention. **Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.** #### Given No further preprocessing. Each term occurs in exactly two of the three documents. #### To find All three vectors. #### Formula and why it applies First count df by documents, then compute each document's TF, then multiply by its term's IDF. #### Step 1. Common IDF $$ D=3 $$ , $$ df(cat)=df(dog)=df(fish)=2 $$ . Each IDF $$ =\log_{10}(3/2)\approx0.176091 $$ . #### Step 2. D1 term frequencies Length 3 → $$ [2/3,1/3,0] $$ . #### Step 3. D1 weights $$ [(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0] $$ . #### Step 4. D2 weights Length 2; TF $$ [0,1/2,1/2] $$ . Vector $$ \approx[0,0.088046,0.088046] $$ . #### Step 5. D3 weights Length 3; TF $$ [1/3,0,2/3] $$ . Vector $$ \approx[0.058697,0,0.117394] $$ . **Answer.** D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394]. TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply. --- ## TEE · Cosine: products, lengths and division Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u2-cosine/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u2-cosine Formulas explained through full worked examples **Cosine similarity compares the direction of two nonzero vectors.** Multiply matching coordinates and add to obtain the dot product. To obtain each vector's norm, square its coordinates, add them and take a square root. $$ \cos(x,y)=\frac{\sum_i x_i y_i}{\sqrt{\sum_i x_i^2}\sqrt{\sum_i y_i^2}}. $$ The vectors must use the same coordinate/vocabulary order. For nonnegative BoW or TF–IDF vectors, the result lies from 0 to 1; for general real-valued vectors, it can range from −1 to 1. A zero vector makes this formula undefined. **Question. Find cosine similarity for x=[1,1] and y=[1,0].** #### Given Nonzero vectors, matching coordinate order. #### To find Find cosine similarity for x=[1,1] and y=[1,0]. #### Formula and why it applies Use $$ \cos(x,y)=(x\cdot y)/(\|x\|\|y\|) $$ because direction similarity is required. #### Step 1. Multiply matching coordinates 1 × 1 = 1; 1 × 0 = 0. #### Step 2. Add products 1 + 0 = 1. #### Step 3. Length of x $$ \sqrt{(1)^2+(1)^2}=\sqrt{2}. $$ #### Step 4. Length of y $$ \sqrt{(1)^2+(0)^2}=\sqrt{1}. $$ #### Step 5. Divide $$ \frac{1}{\sqrt{2}\sqrt{1}}\approx 0.707107. $$ **Answer.** Cosine≈0.707107. **Question. Find cosine similarity for x=[1,2] and y=[2,4].** #### Given Nonzero vectors, matching coordinate order. #### To find Find cosine similarity for x=[1,2] and y=[2,4]. #### Formula and why it applies Use $$ \cos(x,y)=(x\cdot y)/(\|x\|\|y\|) $$ because direction similarity is required. #### Step 1. Multiply matching coordinates 1 × 2 = 2; 2 × 4 = 8. #### Step 2. Add products 2 + 8 = 10. #### Step 3. Length of x $$ \sqrt{(1)^2+(2)^2}=\sqrt{5}. $$ #### Step 4. Length of y $$ \sqrt{(2)^2+(4)^2}=\sqrt{20}. $$ #### Step 5. Divide $$ \frac{10}{\sqrt{5}\sqrt{20}}\approx 1.000000. $$ **Answer.** Cosine≈1.000000. **Question. Find cosine similarity for x=[1,2,0] and y=[0,1,2].** #### Given Nonzero vectors, matching coordinate order. #### To find Find cosine similarity for x=[1,2,0] and y=[0,1,2]. #### Formula and why it applies Use $$ \cos(x,y)=(x\cdot y)/(\|x\|\|y\|) $$ because direction similarity is required. #### Step 1. Multiply matching coordinates 1 × 0 = 0; 2 × 1 = 2; 0 × 2 = 0. #### Step 2. Add products 0 + 2 + 0 = 2. #### Step 3. Length of x $$ \sqrt{(1)^2+(2)^2+(0)^2}=\sqrt{5}. $$ #### Step 4. Length of y $$ \sqrt{(0)^2+(1)^2+(2)^2}=\sqrt{5}. $$ #### Step 5. Divide $$ \frac{2}{\sqrt{5}\sqrt{5}}\approx 0.400000. $$ **Answer.** Cosine≈0.400000. Proportional vectors [1,2] and [2,4] have cosine 1 but nonzero Euclidean distance. Direction similarity does not require identical magnitudes. Cosine: matching products add upar; dono lengths multiply neeche. --- ## TEE · Jaccard: intersection, union and distance Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u2-jaccard/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u2-jaccard Formulas explained through full worked examples **Jaccard similarity is the fraction of shared distinct elements among all distinct elements in either set.** $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|}. $$ Intersection means common elements; union means all distinct elements. Repetitions do not count again. Jaccard distance is $$ 1-J $$ , not the same quantity as similarity. Two empty sets require a declared convention because the fraction becomes 0/0. **Question. Find Jaccard similarity and distance for A={cat, dog} and B={dog, fish}.** #### Given Sets contain unique elements. #### To find Find Jaccard similarity and distance for A={cat, dog} and B={dog, fish}. #### Formula and why it applies $$ J=|A\cap B|/|A\cup B| $$ ; $$ d=1-J $$ . #### Step 1. Find intersection {dog}; count 1. #### Step 2. Find union {cat, dog, fish}; count 3. #### Step 3. Divide J=1/3≈0.333333. #### Step 4. Subtract from one Distance=1−0.333333≈0.666667. **Answer.** Similarity≈0.333333; distance≈0.666667. **Question. Find Jaccard similarity and distance for A={red, blue} and B={red, blue}.** #### Given Sets contain unique elements. #### To find Find Jaccard similarity and distance for A={red, blue} and B={red, blue}. #### Formula and why it applies $$ J=|A\cap B|/|A\cup B| $$ ; $$ d=1-J $$ . #### Step 1. Find intersection {red, blue}; count 2. #### Step 2. Find union {red, blue}; count 2. #### Step 3. Divide J=2/2≈1.000000. #### Step 4. Subtract from one Distance=1−1.000000≈0.000000. **Answer.** Similarity≈1.000000; distance≈0.000000. **Question. Find Jaccard similarity and distance for A={apple} and B={orange, pear}.** #### Given Sets contain unique elements. #### To find Find Jaccard similarity and distance for A={apple} and B={orange, pear}. #### Formula and why it applies $$ J=|A\cap B|/|A\cup B| $$ ; $$ d=1-J $$ . #### Step 1. Find intersection {}; count 0. #### Step 2. Find union {apple, orange, pear}; count 3. #### Step 3. Divide J=0/3≈0.000000. #### Step 4. Subtract from one Distance=1−0.000000≈1.000000. **Answer.** Similarity≈0.000000; distance≈1.000000. Jaccard = common unique ÷ total unique. Distance = 1 minus similarity. --- ## TEE · Manhattan, Euclidean and Minkowski: three complete comparisons Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u2-distances/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u2-distances Formulas explained through full worked examples **Distance measures separation; smaller means closer.** Absolute values prevent positive and negative differences from cancelling. $$ d_1=\sum_i|x_i-y_i|\quad\text{Manhattan} $$ $$ d_2=\sqrt{\sum_i(x_i-y_i)^2}\quad\text{Euclidean} $$ $$ d_p=\left(\sum_i|x_i-y_i|^p\right)^{1/p},\quad p\ge1\quad\text{Minkowski}. $$ Manhattan is p=1; Euclidean is p=2. For p=3, cube absolute differences, add, then take a cube root. **Question. Find Manhattan, Euclidean and Minkowski p=3 distances for x=[1,2], y=[4,6].** #### Given Vectors share coordinate order; negative inputs are allowed. #### To find Find Manhattan, Euclidean and Minkowski p=3 distances for x=[1,2], y=[4,6]. #### Formula and why it applies Use $$ d_1=\sum|\Delta_i| $$ , $$ d_2=\sqrt{\sum|\Delta_i|^2} $$ , $$ d_3=\sqrt[3]{\sum|\Delta_i|^3} $$ . #### Step 1. Absolute differences |1 − (4)| = 3; |2 − (6)| = 4. #### Step 2. Manhattan 3 + 4 = 7. #### Step 3. Square and add 3²=9; 4²=16. Sum=25. #### Step 4. Euclidean root $$ \sqrt{25}\approx 5.000000. $$ #### Step 5. Cube and add 3³=27; 4³=64. Sum=91. #### Step 6. Minkowski root $$ \sqrt[3]{91}\approx 4.497941. $$ **Answer.** Manhattan=7; Euclidean≈5.000000; p=3≈4.497941. **Question. Find Manhattan, Euclidean and Minkowski p=3 distances for x=[0,1,2], y=[2,1,0].** #### Given Vectors share coordinate order; negative inputs are allowed. #### To find Find Manhattan, Euclidean and Minkowski p=3 distances for x=[0,1,2], y=[2,1,0]. #### Formula and why it applies Use $$ d_1=\sum|\Delta_i| $$ , $$ d_2=\sqrt{\sum|\Delta_i|^2} $$ , $$ d_3=\sqrt[3]{\sum|\Delta_i|^3} $$ . #### Step 1. Absolute differences |0 − (2)| = 2; |1 − (1)| = 0; |2 − (0)| = 2. #### Step 2. Manhattan 2 + 0 + 2 = 4. #### Step 3. Square and add 2²=4; 0²=0; 2²=4. Sum=8. #### Step 4. Euclidean root $$ \sqrt{8}\approx 2.828427. $$ #### Step 5. Cube and add 2³=8; 0³=0; 2³=8. Sum=16. #### Step 6. Minkowski root $$ \sqrt[3]{16}\approx 2.519842. $$ **Answer.** Manhattan=4; Euclidean≈2.828427; p=3≈2.519842. **Question. Find Manhattan, Euclidean and Minkowski p=3 distances for x=[-1,3], y=[2,-1].** #### Given Vectors share coordinate order; negative inputs are allowed. #### To find Find Manhattan, Euclidean and Minkowski p=3 distances for x=[-1,3], y=[2,-1]. #### Formula and why it applies Use $$ d_1=\sum|\Delta_i| $$ , $$ d_2=\sqrt{\sum|\Delta_i|^2} $$ , $$ d_3=\sqrt[3]{\sum|\Delta_i|^3} $$ . #### Step 1. Absolute differences |-1 − (2)| = 3; |3 − (-1)| = 4. #### Step 2. Manhattan 3 + 4 = 7. #### Step 3. Square and add 3²=9; 4²=16. Sum=25. #### Step 4. Euclidean root $$ \sqrt{25}\approx 5.000000. $$ #### Step 5. Cube and add 3³=27; 4³=64. Sum=91. #### Step 6. Minkowski root $$ \sqrt[3]{91}\approx 4.497941. $$ **Answer.** Manhattan=7; Euclidean≈5.000000; p=3≈4.497941. Manhattan: absolute-add. Euclidean: square-add-root. Minkowski: p-power-add-p-root. --- ## TEE · Distributional hypothesis and a co-occurrence matrix Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u5-distribution/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u5-distribution Formulas explained through full worked examples **The distributional hypothesis states that words occurring in similar contexts tend to have similar meanings or semantic behaviour.** “Cat” and “dog” often occur near pet, feed and animal. Their context patterns can therefore be represented and compared numerically. A **vector** is an ordered list of numbers. Each position must have a fixed meaning. For context order [pet, food, road], cat=[4,3,0] means four co-occurrences with pet, three with food, none with road. A **matrix** stacks vectors as rows in a table. A **word–context co-occurrence matrix** records how often a target word occurs with each context word under a specified context definition. A window may be a sentence, document, or a fixed number of neighbouring tokens. State its width, direction and whether targets count themselves. Choose target words → Choose context/window → Count co-occurrences → Store each row as a vector → Compare vectors **Question. Build the symmetric co-occurrence matrix for “cats eat fish” using a window of one token on either side, no self-counts and vocabulary [cats,eat,fish].** #### Given One sentence, one occurrence of each token. Left and right adjacent tokens are contexts. #### To find A 3×3 target-by-context count matrix. #### Formula and why it applies $$ X_{ij} $$ counts the number of times context word $$ j $$ occurs within the chosen window of target $$ i $$ . #### Step 1. Target cats Only eat is adjacent. Row cats = [0,1,0]. #### Step 2. Target eat cats is left and fish is right. Row eat = [1,0,1]. #### Step 3. Target fish Only eat is adjacent. Row fish = [0,1,0]. #### Step 4. Assemble the matrix | Target / context | cats | eat | fish | | --- | --- | --- | --- | | cats | 0 | 1 | 0 | | eat | 1 | 0 | 1 | | fish | 0 | 1 | 0 | **Answer.** Rows are [0,1,0], [1,0,1], [0,1,0]. The symmetry follows from the symmetric window. **Question. Build a target–context matrix for sentences “cats eat fish” and “dogs eat fish”. Target words are cats and dogs; context columns [eat,fish]; use the entire sentence excluding the target.** #### Given Each sentence contains one target. Count context token occurrences. #### To find Two context vectors. #### Formula and why it applies Count each context word in every sentence containing the target. #### Step 1. Count cats contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 2. Count dogs contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 3. Compare Both targets have identical observed context patterns in this very small corpus. **Answer.** cats=[1,1], dogs=[1,1]. This corpus supplies no context evidence to distinguish them. **Question. For “a b a”, use a right-only window of one token. Build the matrix with rows/columns [a,b].** #### Given Count directed target→next-word events, not both directions. #### To find The co-occurrence matrix. #### Formula and why it applies $$ X_{ij}=C(\text{target }i\text{ immediately followed by }j) $$ . #### Step 1. First target a Next token is b → add 1 to X(a,b). #### Step 2. Target b Next token is a → add 1 to X(b,a). #### Step 3. Final a No following token → no addition. #### Step 4. Write matrix | Target / next | a | b | | --- | --- | --- | | a | 0 | 1 | | b | 1 | 0 | **Answer.** a=[0,1], b=[1,0]. A different context window would produce a different matrix. Word ka meaning: uske aas-paas kaunse words kitni baar aaye, uski number-list. --- ## TEE · Word2Vec: CBOW and skip-gram from training pairs Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u5-word2vec/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u5-word2vec Formulas explained through full worked examples **Word2Vec learns dense word embeddings by predicting words from nearby context.** Its two common architectures are CBOW and skip-gram. A vocabulary ID or one-hot vector is the input representation; the learned embedding table converts that identity into a short trainable vector. | Architecture | Input | Prediction | Memory cue | | --- | --- | --- | --- | | CBOW | Surrounding context words | Centre/target word | Context → centre | | Skip-gram | Centre/target word | Surrounding words | Centre → context | CBOW: the + sat → Average context embeddings → Predict: cat Skip-gram: cat → Embedding of cat → Predict: the and sat For vocabulary size $$ V $$ and embedding dimension $$ d $$ , an input one-hot row is $$ 1\times V $$ ; the input embedding matrix is $$ V\times d $$ ; the hidden row is $$ 1\times d $$ ; the output matrix is $$ d\times V $$ ; output scores are $$ 1\times V $$ . Multiplying a one-hot row by the embedding matrix simply selects one row. $$ P(w_o\mid w_i)=\frac{\exp(u_{w_o}^{T}v_{w_i})}{\sum_{w\in V}\exp(u_w^{T}v_{w_i})}. $$ $$ v_{w_i} $$ is the input word embedding, $$ u_w $$ an output-context embedding, and the dot product is a compatibility score. Softmax turns all scores into probabilities that sum to one. Training increases the probability of observed context words. Negative sampling or hierarchical softmax can reduce computation in larger vocabularies; use the algorithm specified by the question. **Question. For the sentence “the cat sat on mat”, generate CBOW and skip-gram training data for centre word sat with a window of one token on each side.** #### Given Tokens indexed 1 to 5: the, cat, sat, on, mat. Centre = sat. #### To find The CBOW example and the skip-gram pairs. #### Formula and why it applies CBOW predicts the centre from its neighbours; skip-gram predicts each neighbour from the centre. #### Step 1. Identify neighbours Left neighbour = cat; right neighbour = on. “the” and “mat” are outside this window. #### Step 2. CBOW example Input [cat,on] → target sat. #### Step 3. Skip-gram examples (sat,cat) and (sat,on), with first item as input and second as predicted context. **Answer.** CBOW: [cat,on]→sat. Skip-gram: sat→cat and sat→on. **Question. The embeddings of context words cat and on are [1,2] and [3,0]. Compute the CBOW hidden vector using mean aggregation. Output vectors for [sat,ran] are [1,0] and [0,1]. Compute softmax probabilities.** #### Given Context embeddings [1,2], [3,0]; mean aggregation; two candidate output vectors. #### To find The hidden vector, logits and probabilities. #### Formula and why it applies Mean context vector $$ h=(v_1+v_2)/2 $$ . Logit $$ z_w=u_w^Th $$ . Softmax $$ P(w)=e^{z_w}/\sum_j e^{z_j} $$ . #### Step 1. Add coordinates $$ [1+3,\,2+0]=[4,2] $$ . #### Step 2. Divide by number of words $$ h=[4/2,2/2]=[2,1] $$ . #### Step 3. Compute logits $$ z_{sat}=1(2)+0(1)=2 $$ . $$ z_{ran}=0(2)+1(1)=1 $$ . #### Step 4. Exponentials $$ e^2\approx7.389056 $$ , $$ e^1\approx2.718282 $$ ; sum $$ \approx10.107338 $$ . #### Step 5. Divide $$ P(sat)\approx0.731059 $$ , $$ P(ran)\approx0.268941 $$ . **Answer.** h=[2,1]; sat is more probable (0.731059). **Question. A skip-gram centre embedding is v=[1,−1]. Output embeddings for [eat,sleep,drive] are [1,1], [0,1], [1,0]. Calculate the predicted context probabilities.** #### Given One centre vector and three output vectors; full softmax over these three words. #### To find Three logits and three probabilities. #### Formula and why it applies $$ z_j=u_j^Tv $$ , then $$ P(j)=e^{z_j}/\sum_k e^{z_k} $$ . #### Step 1. Eat dot product $$ 1(1)+1(-1)=0 $$ . #### Step 2. Sleep and drive dot products $$ z_{sleep}=0(1)+1(-1)=-1 $$ . $$ z_{drive}=1(1)+0(-1)=1 $$ . #### Step 3. Exponentiate and sum $$ e^0=1 $$ , $$ e^{-1}\approx0.367879 $$ , $$ e^1\approx2.718282 $$ . Sum $$ \approx4.086161 $$ . #### Step 4. Normalize $$ P(eat)\approx0.244728 $$ ; $$ P(sleep)\approx0.090031 $$ ; $$ P(drive)\approx0.665241 $$ . **Answer.** drive has the highest probability in this toy learned model. **Limitations:** a standard static Word2Vec vector merges a word's senses; it has limited representation of subword morphology and out-of-vocabulary words; its local prediction objective does not directly fit a full global co-occurrence table. Similarity is learned from data and is not a guarantee of factual or causal understanding. CBOW = neighbours se missing centre. Skip-gram = centre se neighbours. --- ## TEE · Word analogies and sentence embeddings Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u5-sentence/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u5-sentence Formulas explained through full worked examples **A sentence embedding** represents an entire sentence by one fixed-length vector. A basic method averages word embeddings. More advanced encoders learn a sentence representation from context and task objectives. $$ v_{\text{sentence}}=\frac1n\sum_{i=1}^{n}v_{w_i}. $$ This means add corresponding coordinates and divide each coordinate by the number of words. Averaging ignores order: “dog bites man” and “man bites dog” have the same average if their words are identical. **Question. Average the word vectors [1,0], [0,2] and [2,1] to represent a three-word sentence.** #### Given Three two-dimensional vectors with equal weights. #### To find The two-dimensional sentence vector. #### Formula and why it applies Use coordinate-wise mean $$ v=(v_1+v_2+v_3)/3 $$ . #### Step 1. First coordinate $$ 1+0+2=3 $$ ; $$ 3/3=1 $$ . #### Step 2. Second coordinate $$ 0+2+1=3 $$ ; $$ 3/3=1 $$ . **Answer.** Sentence vector = [1,1]. **Question. Toy word vectors are king=[3,2], man=[1,0], woman=[1,1]. Find king−man+woman and compare to queen=[3,3].** #### Given All vectors use the same two-coordinate space. #### To find The analogy vector and its distance from queen. #### Formula and why it applies Vector arithmetic is coordinate by coordinate. Euclidean distance is the square root of the sum of squared coordinate differences. #### Step 1. Subtract man $$ [3-1,2-0]=[2,2] $$ . #### Step 2. Add woman $$ [2+1,2+1]=[3,3] $$ . #### Step 3. Compare with queen $$ \sqrt{(3-3)^2+(3-3)^2}=0 $$ . **Answer.** The toy analogy exactly equals queen. Real learned embeddings only approximate some analogies. **Contextual encoders** can assign “bank” different representations in different sentences. BERT and RoBERTa produce contextual token representations; obtaining strong sentence similarity vectors requires an appropriate pooling/training method, not simply assuming every raw token vector is a sentence embedding. Universal Sentence Encoder is designed to encode sentence-level meaning. Skip-thought predicts surrounding sentences; InferSent learns from natural-language-inference supervision. Sentence average: har column add karo, words ki count se divide. Word order ka loss yaad rakh. --- ## TEE · The GloVe objective and numerical interpretation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-vectors--u5-glove-loss/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-vectors/u5-glove-loss Formulas explained through full worked examples $$ J=\sum_{i,j:X_{ij}>0}f(X_{ij})\left(w_i^T\widetilde w_j+b_i+\widetilde b_j-\ln X_{ij}\right)^2. $$ **What each piece means:** $$ w_i $$ is a target vector; $$ \widetilde w_j $$ is a separate context vector; the dot product estimates their compatibility; biases adjust word-level baselines. The desired score is the natural log of the observed count. Squaring penalizes both positive and negative errors. Weighting prevents unreliable rare pairs or extremely frequent pairs from dominating inappropriately. $$ f(x)=\begin{cases}(x/x_{\max})^\alpha,&0 0.0078125. These are unnormalised scores; the label is chosen by the larger score. Predicted label: spam. **Yaad rakho:** Class prior × har word ki class-likelihood; sabse bada score wins. --- ## TEE · Precision, recall, F1 and accuracy Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-applications--evaluation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-applications/evaluation Formulas explained through full worked examples Evaluation compares predicted results with known answers. A **true positive** is a correctly predicted positive; a **false positive** is a negative item predicted positive; a **false negative** is a missed positive; a **true negative** is a correctly rejected negative. $$ \text{Precision}=\frac{TP}{TP+FP} $$ $$ \text{Recall}=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN} $$ $$ \text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} $$ Precision asks how many predicted positives were correct. Recall asks how many actual positives were found. F1 balances them by a harmonic mean. If a denominator is zero, state the convention rather than dividing by zero. ### Example 1 · evaluate a classifier A binary text classifier gives TP=6, FP=2, FN=4, TN=8. Compute precision, recall, F1 and accuracy. **Given** TP=6; FP=2; FN=4; TN=8. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{6}{6+2}=0.750000 $$ #### Step 2. Recall $$ R=\frac{6}{6+4}=0.600000 $$ #### Step 3. F1 $$ F_1=\frac{12}{12+2+4}=0.666667 $$ #### Step 4. Accuracy $$ A=\frac{6+8}{6+2+4+8}=0.700000 $$ Precision 0.7500; recall 0.6000; F1 0.6667; accuracy 0.7000. ### Example 2 · evaluate a classifier A binary text classifier gives TP=9, FP=3, FN=1, TN=7. Compute precision, recall, F1 and accuracy. **Given** TP=9; FP=3; FN=1; TN=7. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{9}{9+3}=0.750000 $$ #### Step 2. Recall $$ R=\frac{9}{9+1}=0.900000 $$ #### Step 3. F1 $$ F_1=\frac{18}{18+3+1}=0.818182 $$ #### Step 4. Accuracy $$ A=\frac{9+7}{9+3+1+7}=0.800000 $$ Precision 0.7500; recall 0.9000; F1 0.8182; accuracy 0.8000. ### Example 3 · evaluate a classifier A binary text classifier gives TP=4, FP=0, FN=4, TN=12. Compute precision, recall, F1 and accuracy. **Given** TP=4; FP=0; FN=4; TN=12. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{4}{4+0}=1.000000 $$ #### Step 2. Recall $$ R=\frac{4}{4+4}=0.500000 $$ #### Step 3. F1 $$ F_1=\frac{8}{8+0+4}=0.666667 $$ #### Step 4. Accuracy $$ A=\frac{4+12}{4+0+4+12}=0.800000 $$ Precision 1.0000; recall 0.5000; F1 0.6667; accuracy 0.8000. --- ## TEE · Spelling correction and edit distance Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-applications--spelling/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-applications/spelling Formulas explained through full worked examples **Spelling correction** proposes the intended word or sentence when the observed text contains an error. A non-word error produces an invalid vocabulary item (“recieve”); a real-word error produces an existing word in the wrong context (“I will meat you”). Dictionary lookup can detect many non-words but cannot reliably detect real-word errors. Detect suspicious text → Generate candidates → Score edit + context fit → Choose correction ### Minimum edit distance Levenshtein distance is the minimum total cost of insertions, deletions and substitutions needed to transform one string into another. In the examples below, each edit costs 1; matching characters cost 0. Transposition is not a single Levenshtein operation. State different costs explicitly if the question gives them. $$ D(i,j)=\min\begin{cases}D(i-1,j)+1\\D(i,j-1)+1\\D(i-1,j-1)+[a_i\ne b_j]\end{cases} $$ Rows represent prefixes of the source; columns represent prefixes of the target. D(i,j) is the best cost for transforming the first i source characters to the first j target characters. Initialise D(i,0)=i and D(0,j)=j. The three choices are deletion, insertion and diagonal match/substitution. ### Example 1 · cat → cut Calculate the unit-cost Levenshtein distance from “cat” to “cut”. Show the dynamic-programming table. **Given** Source=cat; target=cut; insertion=deletion=substitution=1; match=0. **To find** Minimum edit cost. ### Formula and why it applies $$ D(i,j)=\min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+c) $$ c = 0 for matching characters, 1 otherwise. Compare all three possible ways to reach each cell. ### Solution #### Step 1. Initialise empty prefixes First row counts insertions; first column counts deletions. #### Step 2. Fill row 1: source character “c” $$ D(1,1)=\min(1+1,\ 1+1,\ 0+0)=0 $$ $$ D(1,2)=\min(2+1,\ 0+1,\ 1+1)=1 $$ $$ D(1,3)=\min(3+1,\ 1+1,\ 2+1)=2 $$ #### Step 3. Fill row 2: source character “a” $$ D(2,1)=\min(0+1,\ 2+1,\ 1+1)=1 $$ $$ D(2,2)=\min(1+1,\ 1+1,\ 0+1)=1 $$ $$ D(2,3)=\min(2+1,\ 1+1,\ 1+1)=2 $$ #### Step 4. Fill row 3: source character “t” $$ D(3,1)=\min(1+1,\ 3+1,\ 2+1)=2 $$ $$ D(3,2)=\min(1+1,\ 2+1,\ 1+1)=2 $$ $$ D(3,3)=\min(2+1,\ 2+1,\ 1+0)=1 $$ #### Step 5. Read the completed matrix | Source / target | ∅ | c | u | t | | --- | --- | --- | --- | --- | | ∅ | 0 | 1 | 2 | 3 | | c | 1 | 0 | 1 | 2 | | a | 2 | 1 | 1 | 2 | | t | 3 | 2 | 2 | 1 | Minimum edit distance = 1. ### Example 2 · at → cat Calculate the unit-cost Levenshtein distance from “at” to “cat”. Show the dynamic-programming table. **Given** Source=at; target=cat; insertion=deletion=substitution=1; match=0. **To find** Minimum edit cost. ### Formula and why it applies $$ D(i,j)=\min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+c) $$ c = 0 for matching characters, 1 otherwise. Compare all three possible ways to reach each cell. ### Solution #### Step 1. Initialise empty prefixes First row counts insertions; first column counts deletions. #### Step 2. Fill row 1: source character “a” $$ D(1,1)=\min(1+1,\ 1+1,\ 0+1)=1 $$ $$ D(1,2)=\min(2+1,\ 1+1,\ 1+0)=1 $$ $$ D(1,3)=\min(3+1,\ 1+1,\ 2+1)=2 $$ #### Step 3. Fill row 2: source character “t” $$ D(2,1)=\min(1+1,\ 2+1,\ 1+1)=2 $$ $$ D(2,2)=\min(1+1,\ 2+1,\ 1+1)=2 $$ $$ D(2,3)=\min(2+1,\ 2+1,\ 1+0)=1 $$ #### Step 4. Read the completed matrix | Source / target | ∅ | c | a | t | | --- | --- | --- | --- | --- | | ∅ | 0 | 1 | 2 | 3 | | a | 1 | 1 | 1 | 2 | | t | 2 | 2 | 2 | 1 | Minimum edit distance = 1. ### Example 3 · cats → cat Calculate the unit-cost Levenshtein distance from “cats” to “cat”. Show the dynamic-programming table. **Given** Source=cats; target=cat; insertion=deletion=substitution=1; match=0. **To find** Minimum edit cost. ### Formula and why it applies $$ D(i,j)=\min(D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+c) $$ c = 0 for matching characters, 1 otherwise. Compare all three possible ways to reach each cell. ### Solution #### Step 1. Initialise empty prefixes First row counts insertions; first column counts deletions. #### Step 2. Fill row 1: source character “c” $$ D(1,1)=\min(1+1,\ 1+1,\ 0+0)=0 $$ $$ D(1,2)=\min(2+1,\ 0+1,\ 1+1)=1 $$ $$ D(1,3)=\min(3+1,\ 1+1,\ 2+1)=2 $$ #### Step 3. Fill row 2: source character “a” $$ D(2,1)=\min(0+1,\ 2+1,\ 1+1)=1 $$ $$ D(2,2)=\min(1+1,\ 1+1,\ 0+0)=0 $$ $$ D(2,3)=\min(2+1,\ 0+1,\ 1+1)=1 $$ #### Step 4. Fill row 3: source character “t” $$ D(3,1)=\min(1+1,\ 3+1,\ 2+1)=2 $$ $$ D(3,2)=\min(0+1,\ 2+1,\ 1+1)=1 $$ $$ D(3,3)=\min(1+1,\ 1+1,\ 0+0)=0 $$ #### Step 5. Fill row 4: source character “s” $$ D(4,1)=\min(2+1,\ 4+1,\ 3+1)=3 $$ $$ D(4,2)=\min(1+1,\ 3+1,\ 2+1)=2 $$ $$ D(4,3)=\min(0+1,\ 2+1,\ 1+1)=1 $$ #### Step 6. Read the completed matrix | Source / target | ∅ | c | a | t | | --- | --- | --- | --- | --- | | ∅ | 0 | 1 | 2 | 3 | | c | 1 | 0 | 1 | 2 | | a | 2 | 1 | 0 | 1 | | t | 3 | 2 | 1 | 0 | | s | 4 | 3 | 2 | 1 | Minimum edit distance = 1. ### Noisy-channel correction $$ \hat w=\arg\max_w P(x\mid w)P(w) $$ x is the observed misspelling and w a candidate intended word. P(x | w) is the error-model likelihood; P(w) is the language-model prior. In context, replace the isolated-word prior with an appropriate contextual probability. A frequent word with a plausible typo may beat a rarer candidate, but context can reverse the ranking. ### Candidate ranking For observed “teh”, candidate “the” has prior 0.05 and error likelihood 0.02. Candidate “ten” has prior 0.01 and likelihood 0.01. Select the intended word. **Given** Two candidates and their prior/error-model likelihoods are supplied. **To find** Largest noisy-channel score. ### Formula and why it applies $$ s(w)=P(x\mid w)P(w) $$ Multiply likelihood by prior; the same evidence normaliser is irrelevant to ranking. ### Solution #### Step 1. Score the $$ s(\text{the})=0.02\times0.05=0.001 $$ #### Step 2. Score ten $$ s(\text{ten})=0.01\times0.01=0.0001 $$ #### Step 3. Compare 0.001 is ten times 0.0001. Choose “the”. --- ## TEE · Information retrieval Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-applications--retrieval/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-applications/retrieval Formulas explained through full worked examples **Information retrieval (IR)** locates and ranks items relevant to an information need, usually expressed as a query. The collection may contain documents, product descriptions or scientific papers. Retrieval estimates relevance; it does not itself establish that every retrieved claim is true. Documents → cleaning → Build index → Query → same representation → Match & rank → Return relevant results ### The inverted index An inverted index maps a term to the documents containing it, often with positions or counts. For D1 “cats drink milk” and D2 “dogs drink water”, the posting list for drink is {D1,D2}, while milk maps only to D1. This avoids rescanning every document for every query. | Term | Posting list | | --- | --- | | cats | D1 | | drink | D1, D2 | | milk | D1 | | dogs | D2 | | water | D2 | ### Matching and ranking Boolean retrieval combines term membership using AND, OR and NOT. Vector-space retrieval represents documents and queries as weighted vectors, for example with TF-IDF, and compares them with cosine similarity. Learned retrieval can use dense embeddings for meaning-based matching. The representation used at query time must be compatible with the document index. ### Search engines and recommendation A search engine receives an explicit query and ranks matching indexed pages. A recommender may derive an implicit query from a user’s liked items or profile and retrieve similar candidates. Both require candidate retrieval and ranking, but the source of the information need differs. ### Evaluate the results For retrieval, TP is the number of retrieved relevant documents, FP retrieved irrelevant documents, and FN relevant documents not retrieved. Precision and recall use the same formulas as classification. Ranking measures also consider the positions of relevant items; a relevant result at rank 1 is more useful than one far below. ### Retrieval evaluation A search system retrieves 8 documents; 6 are relevant. There are 10 relevant documents in the collection. Find precision, recall and F1. **Given** Retrieved=8; relevant retrieved=6; all relevant=10. **To find** P, R and F1. ### Formula and why it applies $$ P=\frac{\text{relevant retrieved}}{\text{retrieved}},\quad R=\frac{\text{relevant retrieved}}{\text{all relevant}} $$ $$ F_1=\frac{2PR}{P+R} $$ The denominator is 8 for precision and 10 for recall because they ask different questions. ### Solution #### Step 1. Identify the confusion counts TP=6; FP=8−6=2; FN=10−6=4. #### Step 2. Precision $$ P=\frac68=0.75 $$ #### Step 3. Recall $$ R=\frac6{10}=0.6 $$ #### Step 4. F1 $$ F_1=\frac{2(0.75)(0.6)}{0.75+0.6}=\frac{0.9}{1.35}=\frac23\approx0.666667 $$ Precision=0.75, recall=0.60, F1≈0.666667. **Yaad rakho:** Precision: jo laaye, unmein kitne sahi? Recall: jo sahi the, unmein kitne laaye? --- ## TEE · N-gram interpolation: weighted evidence from several histories Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-deep-u3-smoothing-extensions--interpolation/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-deep-u3-smoothing-extensions/interpolation Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Faculty topic; not an observed numerical PYQ.** Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Interpolation combines several context lengths for every prediction.** A trigram estimate is specific but may have little evidence. Bigram and unigram estimates use less context and can support unseen longer sequences. A weighted mixture combines their information. $$ P_{\mathrm{mix}}(w\mid u,v)=\lambda_3P(w\mid u,v)+\lambda_2P(w\mid v)+\lambda_1P(w) $$ $$ \lambda_1,\lambda_2,\lambda_3\ge0,\qquad\lambda_1+\lambda_2+\lambda_3=1. $$ Each component must be a normalized distribution over the same vocabulary. Then summing the mixture over all possible words gives λ₁+λ₂+λ₃=1. A zero higher-order count can be rescued by positive lower-order evidence. Unlike simple backoff, interpolation can combine all levels even when the trigram was seen. **Question. For predicting tea after I like, suppose trigram, bigram and unigram probabilities are 0, 0.2 and 0.1. Use weights λ₃=0.5, λ₂=0.3 and λ₁=0.2 to calculate the interpolated probability. Repeat for coffee with probabilities 0.4, 0.1 and 0.05. [Authored worked example]** ### Answer #### Given The three distributions use one vocabulary. Nonnegative weights sum to 1. These are supplied probabilities; no count estimation is needed. #### To find Pmix(tea | I,like) and Pmix(coffee | I,like). #### Formula and why it applies $$ P_{mix}=\lambda_3P_3+\lambda_2P_2+\lambda_1P_1 $$ We add weighted alternatives from models, rather than multiply sequential word events. #### Solution ##### Step 1. Check the weights $$ 0.5+0.3+0.2=1. $$ ##### Step 2. Calculate the three tea contributions $$ 0.5(0)=0,\qquad0.3(0.2)=0.06,\qquad0.2(0.1)=0.02. $$ ##### Step 3. Add the tea contributions $$ P_{mix}(tea)=0+0.06+0.02=0.08. $$ ##### Step 4. Calculate the coffee contributions $$ 0.5(0.4)=0.20,\qquad0.3(0.1)=0.03,\qquad0.2(0.05)=0.01. $$ ##### Step 5. Add and compare $$ P_{mix}(coffee)=0.20+0.03+0.01=0.24. $$ Coffee gets three times tea’s probability in this model. Other vocabulary outcomes carry the remaining mass; these two displayed entries are not the whole distribution. **Final answer.** Tea=0.08 and coffee=0.24. A zero trigram probability does not force a zero mixture. **Yaad rakho:** Interpolation = sab context-lengths ka weighted mix. Word-sequence product aur model-mixture sum alag operations hain. --- ## TEE · Backoff and the important score-versus-probability distinction Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-deep-u3-smoothing-extensions--stupid-backoff/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-deep-u3-smoothing-extensions/stupid-backoff Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Faculty topic; not an observed numerical PYQ.** Unit 3 PDF p147 names these methods. Neither supplied TEE paper asks their numerical calculation; the worked values below are authored examples. **Backoff uses a shorter history when the longer-history estimate lacks evidence.** A properly normalized backoff model discounts some seen-event probability and allocates the remaining mass across unseen continuations using lower-order probabilities and normalization factors. **Stupid Backoff** is a specifically named efficient scoring method in the slides’ list. It uses an observed n-gram relative frequency when available; otherwise it multiplies the shorter-history score by a fixed penalty, commonly 0.4. Its output is generally an **unnormalized score**, not a conditional probability distribution. Do not insert raw Stupid Backoff scores into standard probability-based perplexity and call the result a normalized-model perplexity. $$ S(w\mid h)=\begin{cases}C(h,w)/C(h),&C(h,w)>0\\ \alpha S(w\mid h'),&C(h,w)=0.\end{cases} $$ Here h′ drops the oldest history word; α is the backoff penalty. At the unigram base, use C(w)/N. The letter α here is a penalty, not the forward-HMM variable introduced later. **Question. A bigram scorer has complete vocabulary {a,b}. After h, C(h,a)=2 and C(h,b)=0. The unigram probabilities are P(a)=P(b)=0.5. Apply Stupid Backoff with penalty 0.4 and show whether the scores form a normalized row. [Authored worked example]** ### Answer #### Given C(h)=2. The a bigram is observed; the b bigram is unseen. Unigram probabilities are supplied. #### To find S(a|h), S(b|h), their sum, and whether they are probabilities. #### Formula and why it applies Use relative frequency for an observed pair and α times the unigram base for an unseen pair. The method does not discount a’s observed score in this example. #### Solution ##### Step 1. Score the observed pair $$ S(a\mid h)=2/2=1. $$ ##### Step 2. Back off for the unseen pair $$ S(b\mid h)=0.4(0.5)=0.2. $$ ##### Step 3. Add the complete row $$ 1+0.2=1.2\ne1. $$ ##### Step 4. Interpret the failure of normalization The scores can rank a above b, but the raw row cannot be called a probability distribution. Merely labelling the method “smoothing” does not change this. **Final answer.** Scores are 1 and 0.2; their sum is 1.2. They are useful ranking scores under this rule, not normalized probabilities. | Method | Operation | Normalization | | --- | --- | --- | | Add-one/add-k | Adjust all allowed outcome counts and the total | Normalized with a fixed vocabulary | | Interpolation | Mix normalized distributions using nonnegative weights summing to 1 | Normalized | | Normalized backoff | Discount seen events and distribute remaining mass over unseen events | Normalized when its factors are correctly computed | | Stupid Backoff | Keep seen relative frequencies; penalize shorter-history scores for unseen n-grams | Generally not normalized | Scope: Unit 3 slide 147 lists Stupid Backoff and n-gram interpolation; the class calculations emphasize Laplace. These short worked extensions explain the listed methods without claiming extra official exam weight. --- ## TEE · Prepare a grammar for CYK / CKY Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-deep-u4-forward-cky--cnf/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-deep-u4-forward-cky/cnf Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. **Faculty topic; not an observed numerical PYQ.** Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. **CYK and CKY are two names for the same dynamic-programming parsing family.** A chart stores which grammatical categories can generate each contiguous span of the sentence. Small spans are solved first and reused to build larger spans. Basic CKY uses a context-free grammar in Chomsky normal form (CNF): a production has either two nonterminals on its right, A→B C, or one terminal, A→word. Empty-string handling is a separate special case; the examples here have no empty production. | Rule | Basic CKY status | Reason | | --- | --- | --- | | S → NP VP | Allowed | Two nonterminals | | N → cat | Allowed | One terminal | | NP → Det Adj N | Convert first | Three nonterminals | | NP → N | Eliminate or handle unary closure | One nonterminal, not a lexical terminal | | VP → eats NP | Convert first | Mixed terminal and nonterminal | **Question. Convert NP→Det Adj N, NP→N, N→cat and VP→eats NP into basic CNF-compatible productions. Preserve their possible word yields. [Authored worked example]** ### Answer #### Given The symbols NP,Det,Adj,N,VP are nonterminals; cat and eats are terminals. Other needed lexical rules remain as supplied. #### To find Equivalent binary or terminal productions for these cases. #### Formula and why it applies Introduce helper categories for a long right-hand side or a terminal mixed with nonterminals. Eliminate a unary nonterminal rule by inheriting the reachable non-unit productions. #### Solution ##### Step 1. Binarize the three-part NP Introduce X: NP→Det X and X→Adj N. X is a helper, not a word; the terminal order remains Det, Adj, N. ##### Step 2. Remove the unit production NP→N and N→cat allow NP to derive cat. Add NP→cat, retain N→cat where needed, and remove NP→N. With more rules reachable through unit chains, inherit every corresponding non-unit rule, not only the first one. ##### Step 3. Replace the mixed terminal Introduce Vlex→eats and replace VP→eats NP by VP→Vlex NP. ##### Step 4. Read the converted forms NP→Det X | cat; X→Adj N; N→cat; VP→Vlex NP; Vlex→eats. All displayed productions are binary nonterminal or single-terminal rules. **Final answer.** The converted rules preserve the original word sequences while satisfying the displayed basic CKY forms. Helper nodes change tree shape; they can be removed when presenting the original grammar’s tree. General CNF conversion may also require eliminating nullable productions and useless symbols while preserving the intended language. Do not claim that simply splitting long rules handles every arbitrary CFG. An Earley parser instead supports general CFG rules using prediction, scanning and completion; it combines top-down expectations with bottom-up evidence. LR and operator-precedence parsers are other methods named in the faculty list, with grammar restrictions of their own. --- ## TEE · CYK / CKY: complete charts, backpointers and ambiguity Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/f-deep-u4-forward-cky--cky/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/f-deep-u4-forward-cky/cky Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. **Faculty topic; not an observed numerical PYQ.** Unit 4 PDF p105 names CYK. The supplied papers ask top-down / bottom-up CFG trees, not CKY charts. These traces explain the taught topic. Number the words 1 to n. Let **T[i,j]** contain all categories that derive the span from word i through word j, inclusive. Start with one-word cells from lexical rules. For every longer span try **every split k** between i and j: if B belongs to T[i,k], C belongs to T[k+1,j], and A→B C is a rule, put A in T[i,j]. Store the rule and split as backpointers when a tree is required. $$ A\in T[i,j]\quad\text{if}\quad A\to BC,\ B\in T[i,k],\ C\in T[k+1,j],\ i\le kj); ∅ is an evaluated but empty valid span. Read the chart by increasing span length, not left-to-right across a row. ##### Step 7. Follow the stored backpointers S[1,5]→NP[1,2] VP[3,5]; NP[1,2]→Det[1,1] N[2,2]; VP[3,5]→V[3,3] NP[4,5]; NP[4,5]→Det[4,4] N[5,5]. ``` S ├── NP │ ├── Det: the │ └── N: cat └── VP ├── V: eats └── NP ├── Det: a └── N: fish ``` **Final answer.** Accepted because S∈T[1,5]. The reconstructed tree’s leaves are exactly the five input words in order. **Question. A grammar has S→A A, A→A A and A→a. Use CKY for “a a a” and explain how two parse trees are preserved. [Authored worked example]** ### Answer #### Given Three identical terminal tokens. S is the start symbol. No unit or empty rules are present. #### To find Chart categories and both successful split choices for the root. #### Formula and why it applies A chart cell is a set of possible categories. For each category keep all successful backpointers when the task requires every parse. #### Solution ##### Step 1. Initialize the diagonal Each of T[1,1], T[2,2], T[3,3] is {A}, from A→a. ##### Step 2. Fill two-word spans T[1,2] and T[2,3] each contain {A,S}: A+A can use A→A A or S→A A. ##### Step 3. Try both root splits k=1: A[1,1] + A[2,3] permits S. k=2: A[1,2] + A[3,3] also permits S. The complete cell is {A,S}, but S has two distinct backpointer alternatives. ##### Step 4. Draw both bracketed trees [S [A a] [A [A a] [A a]]] and [S [A [A a] [A a]] [A a]]. Both yield a a a; the constituent grouping differs. **Final answer.** Accepted, with two S-rooted parses. A category appearing once in a set does not mean there is only one derivation. **Question. With S→NP VP, NP→Det N, VP→V NP; Det→the|a; N→cat|fish; V→eats, test “the eats cat” using CKY. [Authored worked example]** ### Answer #### Given Tokens are the/eats/cat, with no punctuation. Use exactly the supplied grammar. #### To find Whether the grammar accepts the whole input. #### Formula and why it applies A parse exists only if the complete chart cell contains the start symbol. #### Solution ##### Step 1. Initialize the diagonal T[1,1]={Det}; T[2,2]={V}; T[3,3]={N}. ##### Step 2. Try length-two combinations Det+V matches no binary rule; V+N also matches no rule (VP requires V+NP). Both two-word cells are empty. ##### Step 3. Try the two full-span splits Each split has an empty child, so T[1,3]=∅ and S is absent. **Final answer.** Rejected by this grammar. Do not invent NP→N to force acceptance. Grammar rejection and a universal judgment about every possible English grammar are different claims. **Yaad rakho:** Diagonal par words ke tags; phir chhote spans jodo. Har split dekho. Full sentence cell mein S aaya toh parse possible. --- ## TEE · TEE formula sheet: vectors and applications Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/formulas/tee-formula-index--all-tee/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/formulas/tee-formula-index/all-tee TEE · used together with the M2 formula sheet These formulas extend the M2 sheet to the complete TEE syllabus. Each coordinate, token-count convention and log base must be defined before calculation. Worked-example chapters follow this sheet. ### Type–token ratio $$ TTR=\frac{\#\text{types}}{\#\text{tokens}} $$ Distinct vocabulary types divided by all token occurrences. Often expressed as a percentage. ### Normalized term frequency $$ TF(t,d)=\frac{C(t,d)}{\sum_w C(w,d)} $$ The slides use count of the term divided by the document token total after the specified preprocessing. ### Inverse document frequency $$ IDF(t)=\log_{10}\frac{D}{df(t)} $$ D is the number of documents; df counts documents containing the term at least once. This is the course slide convention. ### TF–IDF $$ TF\!\operatorname{-}\!IDF(t,d)=TF(t,d)IDF(t) $$ Multiply the local frequency weight by the global rarity weight. ### Dot product $$ a\cdot b=\sum_i a_ib_i $$ Multiply corresponding coordinates, then add. Vectors need the same coordinate order. ### Vector length $$ \|a\|_2=\sqrt{\sum_i a_i^2} $$ Square each coordinate, add, then take the square root. ### Cosine similarity $$ \cos(a,b)=\frac{\sum_i a_ib_i}{\sqrt{\sum_i a_i^2}\sqrt{\sum_i b_i^2}} $$ Compare direction; undefined for a zero-length vector. ### Jaccard similarity $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|} $$ Set membership, not repeated word counts. Jaccard distance = 1 − J. ### Manhattan distance $$ d_1(a,b)=\sum_i|a_i-b_i| $$ Add coordinate-wise absolute differences. ### Euclidean distance $$ d_2(a,b)=\sqrt{\sum_i(a_i-b_i)^2} $$ Square differences, add, square root. ### Minkowski distance $$ d_p(a,b)=\left(\sum_i|a_i-b_i|^p\right)^{1/p} $$ p≥1; Manhattan p=1, Euclidean p=2. ### Mean sentence embedding $$ v_{sentence}=\frac1n\sum_{i=1}^n v_{w_i} $$ Add word vectors coordinate by coordinate; divide each coordinate by n. This baseline loses word order. ### GloVe objective $$ J=\sum_{i,j}f(X_{ij})(w_i^T\widetilde w_j+b_i+\widetilde b_j-\log X_{ij})^2 $$ X is a co-occurrence count. Compute the residual, square it and apply the weighting function. Use observed positive counts. ### Precision and recall $$ P=\frac{TP}{TP+FP},\qquad R=\frac{TP}{TP+FN} $$ Precision checks predicted positives; recall checks actual positives. ### F1 score $$ F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN} $$ Harmonic mean of precision and recall. ### Accuracy $$ A=\frac{TP+TN}{TP+TN+FP+FN} $$ All correct predictions divided by all predictions. ### Noisy-channel correction $$ \hat w=\arg\max_w P(x\mid w)P(w) $$ Rank intended word w using observed typo x; likelihood × prior. ### Unit-cost edit distance $$ D(i,j)=\min\{D(i-1,j)+1,D(i,j-1)+1,D(i-1,j-1)+[a_i\ne b_j]\} $$ Initialise first row/column by insertion/deletion counts; fill all cells using min. --- ## TEE · Q1A · NLU and NLG Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q1a Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Differentiate between Natural Language Understanding (NLU) and Natural Language Generation (NLG) in NLP with suitable examples. [5 marks] ### Answer **Natural Language Understanding (NLU)** interprets language to identify its meaning, intent and relevant information. **Natural Language Generation (NLG)** produces language that expresses information in a useful form. | Basis | NLU | NLG | | --- | --- | --- | | Direction | Language → meaning or structured information | Meaning or structured information → language | | Typical input | A user utterance, document or sentence | Facts, a database record, a prompt or a representation | | Typical output | Intent, entities, sentiment or a semantic representation | A sentence, explanation, summary or dialogue response | | Main difficulty | Resolving ambiguity and context | Choosing accurate content and producing coherent, fluent wording | ### Example: a weather assistant User: “Will it rain in Mumbai tomorrow?” NLU recognises a weather-query intent, the location Mumbai and the date tomorrow. After a weather service supplies the forecast, NLG produces “Rain is likely in Mumbai tomorrow; carry an umbrella.” User language → NLU: identify request → Retrieve forecast → NLG: express answer The two functions often work together in a chatbot. Understanding the request and wording the reply are separate tasks; fluent generation alone does not prove correct understanding. **Yaad rakho:** NLU mein meaning nikalo; NLG mein meaning ko sentence banao. --- ## TEE · Q1B · Chunking and chinking Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q1b Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Explain the concepts of chunking and chinking in Natural Language Processing (NLP). Discuss how Part-of-Speech (POS) tagging supports these processes, and illustrate your explanation with a suitable example. [5 marks] ### Answer **Chunking** groups adjacent tagged words into shallow phrases, such as noun phrases, without building a complete sentence parse. **Chinking** removes a specified sequence of tags or words from a chunk, splitting a larger candidate chunk when required. ### POS tags provide the pattern A POS tag describes a grammatical category: DT = determiner, JJ = adjective, NN = singular noun, VBD = past-tense verb and IN = preposition. A noun-phrase rule can be written as `NP: {
?*+}`: an optional determiner, zero or more adjectives, followed by one or more nouns. ### Worked example | Word | The | small | cat | sat | on | the | mat | | --- | --- | --- | --- | --- | --- | --- | --- | | Tag | DT | JJ | NN | VBD | IN | DT | NN | Applying the NP rule gives **[The small cat] sat on [the mat]**. The first chunk contains DT–JJ–NN; the second contains DT–NN. “sat on” is not included because it does not match the noun-phrase rule. For chinking, begin with one broad chunk: **[The small cat sat on the mat]**. Remove the VBD and IN sequence “sat on”, for example with `}+{`. The result is **[The small cat] sat on [the mat]**. POS tagging therefore supplies grammatical labels; chunking uses labels to select phrases, while chinking uses labels to exclude unwanted material. Incorrect tags can lead to incorrect chunks. **Yaad rakho:** Chunking = phrase ko jodo; chinking = beech ka unwanted part nikalo. --- ## TEE · Q1C · Hyponymy, hypernymy and WSD Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q1c Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Define hyponymy and hypernymy with suitable linguistic examples. How can hypernymy be applied in Word Sense Disambiguation (WSD) to resolve the correct meaning of polysemous words? Provide an example? [5 marks] ### Answer **Hyponymy** is the “is-a-kind-of” relation from a specific concept to a broader concept. **Hypernymy** is the inverse relation from the broader concept to the specific one. “Rose” is a hyponym of “flower”; “flower” is a hypernym of “rose”. rose / tulip → flower → plant The arrow points towards a broader category. Hyponymy is not a part–whole relation: a wheel is part of a car, but it is not a kind of car. ### Using the hierarchy for WSD WSD selects the intended sense of a word from its context. Obtain candidate senses from a lexical resource such as WordNet, inspect their hypernym chains, and favour the chain most compatible with nearby words. 1. Sentence: “The mouse clicked the icon.” 2. Candidate 1: mouse → rodent → mammal → animal. Candidate 2: mouse → pointing device → computer peripheral. 3. The context words “clicked” and “icon” belong to the computer-use domain. They support the device hierarchy, rather than the animal hierarchy. 4. Select the computer-pointing-device sense. In “The mouse ate cheese,” the animal hierarchy is more compatible. Hypernym evidence supplies semantic generalisation; it is a useful signal rather than a guarantee, because short or unusual contexts may still be ambiguous. **Yaad rakho:** Hyponym specific hai; hypernym uska bada category hai. --- ## TEE · Q1D · Pragmatics and discourse Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q1d Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Compare pragmatic and discourse analysis with the help of appropriate examples. [5 marks] ### Answer **Pragmatic analysis** interprets an utterance using the situation, speaker intention, shared knowledge and conversational expectations. **Discourse analysis** studies how multiple utterances form a connected text or conversation. | Basis | Pragmatic analysis | Discourse analysis | | --- | --- | --- | | Focus | Intended meaning in context | Relationships and coherence across utterances | | Useful information | Speaker, listener, place, time and common knowledge | Previous sentences, entities, topics and rhetorical relations | | Typical tasks | Interpret requests, implicatures and deixis | Resolve references and analyse cohesion, coherence and discourse structure | | Example | “It is cold here” may be a request to close a window | “Riya bought a book. She read it.” links She to Riya and it to book | In “Can you pass the salt?”, literal syntax asks about ability; pragmatic interpretation usually identifies a polite request. In “The road flooded. Therefore, the school closed,” discourse analysis identifies a cause–result link between the sentences. The levels interact: resolving “she” across a conversation may require both earlier text and real-world context. They are complementary rather than mutually exclusive. **Yaad rakho:** Pragmatics: bolne ka intention. Discourse: sentences ka connection. --- ## TEE · Q2A · Morpheme, lexeme and NER Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q2a Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. i. Differentiate between Morpheme and Lexeme with examples. ii. Discuss the significance of NER in three different NLP applications. [10 marks] ### Answer ### i. Morpheme and lexeme A **morpheme** is the smallest linguistic unit that contributes meaning or grammatical function. A **lexeme** is an abstract vocabulary item representing a family of inflected word forms. | Basis | Morpheme | Lexeme | | --- | --- | --- | | Unit | A meaningful component of a word | An abstract dictionary vocabulary item | | Examples | un-, happy, -ness; cat and plural -s | RUN covers run, runs, ran and running | | Independence | Free morphemes can stand alone; bound morphemes cannot | A lexeme is realised through one or more word forms | | NLP relevance | Morphological segmentation, stemming and analysis | Lemmatization and lexical meaning | “Cats” contains two morphemes: cat + plural -s. “cat” and “cats” are forms of the lexeme CAT. “Unhappiness” can be segmented into un + happy + ness, although its spelling changes at the boundary. A morpheme and a syllable are different: syllables concern pronunciation, while morphemes concern meaning or grammar. ### ii. Named Entity Recognition in applications **NER** locates spans that name entities and assigns types such as PERSON, ORGANIZATION, LOCATION, DATE or domain-specific categories. | Application | Entities and example | Why NER matters | | --- | --- | --- | | News information extraction | “Asha joined Infosys in Pune.” → Asha/PERSON, Infosys/ORG, Pune/LOCATION | Creates searchable people–organisation–location records and supports later relation extraction | | Healthcare document processing | Find disease, drug and dosage spans in a clinical note | Supports record indexing and medication extraction; specialist labels and validation are required | | Customer support and commerce | Extract product names, dates and order identifiers from “My Pixel arrived damaged on Monday.” | Routes complaints, links the correct product and fills structured support fields | NER answers “Which entity spans occur here, and what type are they?” It does not by itself establish every relation between those entities or resolve all names to unique real-world identities. **Yaad rakho:** Morpheme = word ka meaningful piece; lexeme = word family. NER = naam dhoondho aur type do. --- ## TEE · Q2B · Laplace-smoothed bigram probability Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q2b Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Consider the following training corpus: 1. artists paint murals in city 2. musicians play songs in city 3. painters paint murals on walls 4. signs are near city Using a bigram language model with Laplace smoothing, evaluate the probability of the test sentence: “artists paint murals near city”. Assume the and for each sentence. [10 marks] ### Answer ### Given - artists paint murals in city - musicians play songs in city - painters paint murals on walls - signs are near city Test: artists paint murals near city. Add one start token and one end token to every training and test sentence. ### To find The Laplace-smoothed bigram probability of the complete test sentence, including its end token. ### Formula: which one and why $$ P_{+1}(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)+1}{C(w_{i-1})+V} $$ C(previous,current) counts that adjacent pair in the training corpus. C(previous) counts its occurrences as a history. V is the vocabulary size. Add-one smoothing is necessary because an unseen pair must not force the whole sentence probability to zero. $$ P(w_1,\ldots,w_n,\langle/s\rangle\mid\langle s\rangle)=\prod_{i=1}^{n+1}P(w_i\mid w_{i-1}) $$ ### Solution ### Step 1. Fix the token and vocabulary convention Lowercase words and remove the final punctuation. The supplied course key counts both boundary tokens in V. Following that convention: are, artists, city, in, murals, musicians, near, on, paint, painters, play, signs, songs, walls, , gives **V = 16**. The start marker supplies history; it is not an extra probability factor before the first word. ### Step 2. List every required transition The complete test chain is → artists → paint → murals → near → city → . ### Step 3. Count first, then add one | Required bigram | History count | Pair count | Smoothed probability | | --- | --- | --- | --- | | → artists | 4 | 1 | (1+1)/(4+16) = 1/10 | | artists → paint | 1 | 1 | (1+1)/(1+16) = 2/17 | | paint → murals | 2 | 2 | (2+1)/(2+16) = 1/6 | | murals → near | 2 | 0 | (0+1)/(2+16) = 1/18 | | near → city | 1 | 1 | (1+1)/(1+16) = 2/17 | | city → | 3 | 3 | (3+1)/(3+16) = 4/19 | Count only adjacent words, never words that merely appear in the same sentence. The unseen pairs have count zero, so their numerator becomes one. ### Step 4. Multiply all six probabilities $$ P=\frac{2}{20}\times\frac{2}{17}\times\frac{3}{18}\times\frac{1}{18}\times\frac{2}{17}\times\frac{4}{19} $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. → artists | $$ 1\times\frac{1}{10}=\frac{1}{10} $$ | | 2. artists → paint | $$ \frac{1}{10}\times\frac{2}{17}=\frac{1}{85} $$ | | 3. paint → murals | $$ \frac{1}{85}\times\frac{1}{6}=\frac{1}{510} $$ | | 4. murals → near | $$ \frac{1}{510}\times\frac{1}{18}=\frac{1}{9180} $$ | | 5. near → city | $$ \frac{1}{9180}\times\frac{2}{17}=\frac{1}{78030} $$ | | 6. city → | $$ \frac{1}{78030}\times\frac{4}{19}=\frac{2}{741285} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{2}{741285}\approx 2.69801763\times10^{-6} $$ ### Step 5. Interpret the result This is a joint sentence probability, so a small value is normal. Do not add transition probabilities; each word must occur in sequence, so the chain multiplies them. ### Final answer **P ≈ 2.69801763 × 10-6** with the stated course vocabulary convention. **Yaad rakho:** Pair count mein 1 jodo, previous-word count mein V; phir saare fractions multiply karo. **Convention note:** Some implementations exclude from the predicted vocabulary. Their V is one smaller and their numerical probability differs. In an exam, state V explicitly and follow the convention specified in the question or class solution. --- ## TEE · Q3A · Ambiguity and bottom-up parsing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q3a Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. i. Explain the various types of ambiguities encountered in Natural Language Processing (NLP). Illustrate each type with suitable examples. ii. Compare top-down and bottom-up parsing and apply bottom-up syntax parsing approach to the sentence “A cat chased the mouse” and draw a parse tree using the grammar rules: S → NP VP; NP → Pronoun | Proper-Noun | Det Nominal; Nominal → Noun | Nominal Noun | Nominal PP; VP → Verb | Verb NP | VP PP; PP → Prep NP. Lexicon: Det → {A, the}; Noun → {cat, mouse}; Verb → {chased}. [10 marks] ### Answer ### i. Types of ambiguity Ambiguity exists when the same language expression permits more than one interpretation. The correct interpretation usually requires grammatical, semantic or contextual evidence. | Type | Meaning | Example and readings | | --- | --- | --- | | Lexical | One word has multiple meanings | “She visited the bank”: financial institution or riverbank | | Syntactic | The sentence permits different structural attachments | “I saw the man with a telescope”: I used the telescope, or the man had it | | Semantic / scope | A structured sentence permits different interpretations | “Every student read a book”: one shared book, or possibly a different book per student | | Referential | A referring expression has more than one candidate referent | “Asha told Riya that she won”: she could refer to Asha or Riya | | Pragmatic | Intention depends on the situation | “Can you open the door?” can ask about ability or request an action | ### ii. Parsing approaches | Aspect | Top-down | Bottom-up | | --- | --- | --- | | Start | The start symbol S | The input words | | Operation | Expand a nonterminal using grammar productions | Combine words and constituents using productions in reverse | | Success | Leaves match the complete input | All input is reduced to S | | Possible difficulty | Naive recursive descent can loop on left recursion or need backtracking | A naive shift-reduce parser may face shift/reduce conflicts | ### Given The five tokens are A / cat / chased / the / mouse. The supplied grammar permits determiner + nominal noun phrases and a verb + noun phrase verb phrase. ### To find A bottom-up sequence of reductions and a parse tree whose root is S and whose leaves preserve the input order. ### Rule: which one and why A production A → B C permits B C to be reduced to A. Bottom-up parsing begins with the words; lexical rules assign their categories, and phrase rules combine those categories. ### Step 1. Assign lexical categories A → Det; cat → Noun; chased → Verb; the → Det; mouse → Noun. ### Step 2. Build the first noun phrase cat is Noun. Reduce Noun to Nominal using Nominal → Noun. Then reduce Det Nominal, representing “A cat”, to NP. ### Step 3. Build the object noun phrase mouse is Noun. Reduce it to Nominal. Reduce “the” + “mouse”, now Det Nominal, to NP. ### Step 4. Build the verb phrase “chased” is Verb and “the mouse” is NP. Reduce Verb NP to VP. ### Step 5. Build the sentence The remaining constituents are NP (“A cat”) and VP (“chased the mouse”). Apply S → NP VP in reverse to obtain S. Every input token is used exactly once. SNPVPDetNominalVerbNPNounDetNominalNounAcatchasedthemouse Bottom-up parse tree for A cat chased the mouse. **Final answer:** [S [NP [Det A] [Nominal [Noun cat]]] [VP [Verb chased] [NP [Det the] [Nominal [Noun mouse]]]]]. **Yaad rakho:** Bottom-up: words se phrases, phrases se S. Top-down: S se words. --- ## TEE · Q3B · Co-occurrence matrix and cosine Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q3b Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Consider the given paragraph: The teacher wrote on the board. The students listened to the teacher. The board was full of notes. The student asked a question. The teacher answered from the book. Target words: teacher, student, board, book. Context words: wrote, listened, notes, question. i. Construct a co-occurrence matrix for the target and context words. Take the window size as one whole sentence. ii. Evaluate the similarity between the vectors of the target words “teacher-student” and “teacher-board” using the cosine similarity. iii. Plot the target word vectors by considering only two dimensions: “wrote” and “notes”. [10 marks] ### Answer ### Given Five sentences, four target rows and four context columns. Treat students and student as the same target lemma, and count each target–context pair once when both occur in a sentence. ### To find The 4 × 4 sentence-window co-occurrence matrix, two cosine similarities and the requested 2-D plot. ### Formula: which one and why $$ M_{ij}=\sum_{s=1}^{5}\mathbf{1}[\text{target}_i\text{ and context}_j\text{ occur in sentence }s] $$ The indicator is 1 when both words appear in that sentence and 0 otherwise. Each matrix row is the target word’s vector: its list of four context counts. $$ \cos(\mathbf a,\mathbf b)=\frac{\mathbf a\cdot\mathbf b}{\|\mathbf a\|\|\mathbf b\|} $$ The dot product multiplies corresponding coordinates and adds them. The length is the square root of the sum of squared coordinates. Cosine compares directions while allowing different vector lengths. ### Solution ### Step 1. Record the contribution of each sentence | Sentence | Targets present | Relevant context | Counts added | | --- | --- | --- | --- | | 1 | teacher, board | wrote | teacher–wrote +1; board–wrote +1 | | 2 | student, teacher | listened | student–listened +1; teacher–listened +1 | | 3 | board | notes | board–notes +1 | | 4 | student | question | student–question +1 | | 5 | teacher, book | none | No requested context count | ### Step 2. Construct every requested column | Target | wrote | listened | notes | question | | --- | --- | --- | --- | --- | | teacher | 1 | 1 | 0 | 0 | | student | 0 | 1 | 0 | 1 | | board | 1 | 0 | 1 | 0 | | book | 0 | 0 | 0 | 0 | ### Step 3. Calculate teacher–student $$ \mathbf t\cdot\mathbf s=(1)(0)+(1)(1)+(0)(0)+(0)(1)=1 $$ $$ \|\mathbf t\|=\sqrt{1^2+1^2+0^2+0^2}=\sqrt2 $$ $$ \|\mathbf s\|=\sqrt{0^2+1^2+0^2+1^2}=\sqrt2 $$ $$ \cos(\mathbf t,\mathbf s)=\frac1{\sqrt2\sqrt2}=\frac12=0.5 $$ ### Step 4. Calculate teacher–board $$ \mathbf t\cdot\mathbf b=(1)(1)+(1)(0)+(0)(1)+(0)(0)=1 $$ $$ \|\mathbf b\|=\sqrt{1^2+0^2+1^2+0^2}=\sqrt2 $$ $$ \cos(\mathbf t,\mathbf b)=\frac1{\sqrt2\sqrt2}=0.5 $$ ### Step 5. Select only the two requested plot coordinates | Word | x = wrote | y = notes | | --- | --- | --- | | teacher | 1 | 0 | | student | 0 | 0 | | board | 1 | 1 | | book | 0 | 0 | 001122teacherstudent, bookboardwrotenotes The plot uses only the requested two coordinates; cosine uses all context columns. ### Final answer Both requested cosine similarities are **0.5**. Student and book overlap at the origin of the 2-D projection, but their complete four-dimensional vectors differ. The book row is zero, so a cosine involving book would be undefined without an explicit convention. **Yaad rakho:** Har sentence mein target aur context saath? Ek count. Cosine ke liye poori row lo, plot ke liye sirf do columns. --- ## TEE · Q4A · Three preprocessing techniques Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q4a Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Explain any three basic text preprocessing techniques used in Natural Language Processing (NLP). Discuss the purpose of each technique and illustrate them with suitable examples. [10 marks] ### Answer Text preprocessing transforms raw text into a representation suitable for a chosen NLP task. The best steps depend on the task; removing information indiscriminately can reduce performance. ### 1. Tokenization Tokenization divides text into units called tokens, such as words, subwords or sentences. These units are required for counting words and building many model inputs. “The cats are sleeping.” can become [The, cats, are, sleeping, .] under a punctuation-preserving word tokenizer. A different tokenizer may split contractions or punctuation differently, so the convention must be consistent during training and prediction. ### 2. Lowercasing Lowercasing maps uppercase letters to lowercase so that variants such as “Book”, “BOOK” and “book” can share one feature. This reduces vocabulary duplication in a case-insensitive search system. However, “US” and “us” are different, and capitals help NER recognise names. Therefore lowercasing should be chosen according to the application. ### 3. Lemmatization Lemmatization maps an inflected word to its dictionary base form, often using its POS and context. Examples: cats → cat; running/VBG → run; better/JJR → good. It allows a search for “run” to match relevant inflections. A stemmer instead uses simpler truncation rules and can produce a non-word stem; lemmatization is linguistically informed and can need more processing. ### Combined example | Stage | Representation | | --- | --- | | Raw | The cats are running. | | Tokenized | The | cats | are | running | . | | Lowercased | the | cats | are | running | . | | Lemmatized with correct POS | the | cat | be | run | . | These steps do not automatically remove stop words or punctuation. Those are separate choices. For sentiment analysis, preserve negation such as “not”; for syntax parsing, function words and word order are often essential. **Yaad rakho:** Preprocess task dekh ke karo: har word ko blindly delete mat karo. --- ## TEE · Q4B · HMM emission, transition and sequence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q4b Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Consider the following POS tagged corpus. Apply Hidden Markov Model (HMM) approach to: 1. Cat/Noun chase/Verb mice/Noun 2. The/Det dog/Noun runs/Verb 3. A/Det cat/Noun eats/Verb cheese/Noun i. Construct the emission probability matrix. ii. Construct the transmission probability matrix. iii. Calculate the probability of sequence of events: The/Det cat/Noun eats/Verb mice/Noun. Note: Include start and end characters for all the sentences. [10 marks] ### Answer ### Given | Sentence | Tagged sequence | | --- | --- | | 1 | cat/Noun chase/Verb mice/Noun | | 2 | the/Det dog/Noun runs/Verb | | 3 | a/Det cat/Noun eats/Verb cheese/Noun | Test words: the cat eats mice. Given tag sequence: Det → Noun → Verb → Noun. Include Start and End for all sentences. Lowercase the corpus so sentence-initial Cat and cat are one word. ### To find Emission probabilities, transition probabilities and the joint probability of the specified tag-and-word sequence. ### Formula: which one and why $$ P(w\mid t)=\frac{C(t,w)}{C(t)} $$ An emission probability asks: among tokens carrying this tag t, what fraction are the word w? $$ P(t_j\mid t_i)=\frac{C(t_i,t_j)}{\sum_k C(t_i,t_k)} $$ A transition probability asks: among transitions leaving tag tᵢ, what fraction go to tⱼ? Include End in the outgoing counts. The paper says “transmission”; the standard HMM term is transition. $$ P(\mathbf w,\mathbf t)=P(t_1\mid\mathrm{Start})\prod_{i=1}^{n}P(w_i\mid t_i)\prod_{i=2}^{n}P(t_i\mid t_{i-1})P(\mathrm{End}\mid t_n) $$ ### Solution ### Step 1. Count tokens for each tag Det has 2 tokens; Noun has 5 tokens; Verb has 3 tokens. The repeated noun cat occurs twice with Noun; every other displayed noun occurs once. The three verb types each occur once. ### Step 2. Build the emission matrix | Tag / word | a | cat | chase | cheese | dog | eats | mice | runs | the | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Det | 1/2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1/2 | | Noun | 0 | 2/5 | 0 | 1/5 | 1/5 | 0 | 1/5 | 0 | 0 | | Verb | 0 | 0 | 1/3 | 0 | 0 | 1/3 | 0 | 1/3 | 0 | Each row sums to one. A zero means no occurrence under that tag in this unsmoothed training corpus; no add-one smoothing was requested. ### Step 3. Write the three tag chains Start → Noun → Verb → Noun → End Start → Det → Noun → Verb → End Start → Det → Noun → Verb → Noun → End ### Step 4. Count outgoing transitions | From | To Det | To Noun | To Verb | To End | Outgoing total | | --- | --- | --- | --- | --- | --- | | Start | 2 | 1 | 0 | 0 | 3 | | Det | 0 | 2 | 0 | 0 | 2 | | Noun | 0 | 0 | 3 | 2 | 5 | | Verb | 0 | 2 | 0 | 1 | 3 | ### Step 5. Divide each row by its outgoing total | From / To | Det | Noun | Verb | End | | --- | --- | --- | --- | --- | | Start | 2/3 | 1/3 | 0 | 0 | | Det | 0 | 1 | 0 | 0 | | Noun | 0 | 0 | 3/5 | 2/5 | | Verb | 0 | 2/3 | 0 | 1/3 | End is terminal, so it has no estimated outgoing row. The four nonterminal rows each sum to one. ### Step 6. Choose the factors for the given sequence | Event in order | Probability | | --- | --- | | Start → Det | 2/3 | | Det emits the | 1/2 | | Det → Noun | 1 | | Noun emits cat | 2/5 | | Noun → Verb | 3/5 | | Verb emits eats | 1/3 | | Verb → Noun | 2/3 | | Noun emits mice | 1/5 | | Noun → End | 2/5 | ### Step 7. Multiply, including the end transition $$ P=\frac23\times\frac12\times1\times\frac25\times\frac35\times\frac13\times\frac23\times\frac15\times\frac25 $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. Start → Det | $$ 1\times\frac{2}{3}=\frac{2}{3} $$ | | 2. Det emits the | $$ \frac{2}{3}\times\frac{1}{2}=\frac{1}{3} $$ | | 3. Det → Noun | $$ \frac{1}{3}\times1=\frac{1}{3} $$ | | 4. Noun emits cat | $$ \frac{1}{3}\times\frac{2}{5}=\frac{2}{15} $$ | | 5. Noun → Verb | $$ \frac{2}{15}\times\frac{3}{5}=\frac{2}{25} $$ | | 6. Verb emits eats | $$ \frac{2}{25}\times\frac{1}{3}=\frac{2}{75} $$ | | 7. Verb → Noun | $$ \frac{2}{75}\times\frac{2}{3}=\frac{4}{225} $$ | | 8. Noun emits mice | $$ \frac{4}{225}\times\frac{1}{5}=\frac{4}{1125} $$ | | 9. Noun → End | $$ \frac{4}{1125}\times\frac{2}{5}=\frac{8}{5625} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac8{5625}\approx0.0014222222 $$ ### Final answer **Joint sequence probability = 8/5625 ≈ 0.0014222222.** This is the probability of the specified words and tags together. It is not the probability of the word sequence summed over every possible tag path. **Yaad rakho:** Tag se tag = transition; tag se word = emission. Dono alternate multiply, end ko mat bhoolo. **Boundary reminder:** The initial factor is P(Det | Start) = 2/3, based on three sentence starts, and the final factor is P(End | Noun) = 2/5. The overall fraction of Det tokens, 2/10, is not the start probability. Using it and omitting End, as in the handwritten key’s last line, computes a different quantity. --- ## TEE · Q5A · TF–IDF matrix Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q5a Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. For the following two sentences: Sentence A: “The student read a science book in the library” Sentence B: “The teacher read a history book in the classroom” 1. Construct the TF-IDF matrix for these sentences after removing the stop words {The, a, in}. 2. Identify and comment on the important terms in each sentence based on their TF-IDF scores. [10 marks] ### Answer ### Given Two documents, N = 2. Remove the listed stop words case-insensitively, so both “The” and “the” are removed. Use raw count TF and natural-log IDF, the convention used in the course key. ### To find The term-by-document TF–IDF matrix and the terms that distinguish each sentence. ### Formula: which one and why $$ \mathrm{TF}(t,d)=C(t,d),\qquad \mathrm{IDF}(t)=\ln\frac{N}{\mathrm{df}(t)} $$ $$ \mathrm{TFIDF}(t,d)=\mathrm{TF}(t,d)\,\mathrm{IDF}(t) $$ C(t,d) is the number of occurrences of term t in document d. df(t) is the number of documents containing t, not its total token count. N is the number of documents. IDF downweights terms shared by many documents. ### Solution ### Step 1. Clean and list the tokens A: student, read, science, book, library. B: teacher, read, history, book, classroom. Each retained token appears once in its sentence. ### Step 2. Fix one common vocabulary order student, read, science, book, library, teacher, history, classroom. Each row must refer to the same term in both documents. ### Step 3. Count document frequency and calculate IDF | Term type | df | IDF calculation | | --- | --- | --- | | read, book | 2 | ln(2/2) = ln(1) = 0 | | Every other term | 1 | ln(2/1) = ln(2) ≈ 0.693147 | A term absent from a document has TF = 0. A present term has TF = 1 here. ### Step 4. Multiply TF by IDF | Term | TF in A | TF in B | IDF | TF–IDF A | TF–IDF B | | --- | --- | --- | --- | --- | --- | | student | 1 | 0 | 0.693147 | 0.693147 | 0 | | read | 1 | 1 | 0 | 0 | 0 | | science | 1 | 0 | 0.693147 | 0.693147 | 0 | | book | 1 | 1 | 0 | 0 | 0 | | library | 1 | 0 | 0.693147 | 0.693147 | 0 | | teacher | 0 | 1 | 0.693147 | 0 | 0.693147 | | history | 0 | 1 | 0.693147 | 0 | 0.693147 | | classroom | 0 | 1 | 0.693147 | 0 | 0.693147 | ### Step 5. Interpret the weights A is distinguished by student, science and library. B is distinguished by teacher, history and classroom. “read” and “book” have zero IDF in this tiny unsmoothed two-document collection because they appear in every document; that does not mean those words are meaningless in English. ### Final answer All six document-specific terms receive **0.693147** in their own document and 0 in the other. Shared read and book receive 0 in both. **Yaad rakho:** TF: is document mein kitni baar? IDF: kitne documents mein common? Rare-across-documents term ko zyada weight. If normalized TF is explicitly requested, divide each present-token count by five; each nonzero weight becomes 0.693147/5 = 0.1386294. Do not silently switch conventions. --- ## TEE · Q5B · Naïve Bayes word-sense classification Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q5b Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Consider the sentence: “The bat hit the ball hard on the cricket field.” Two possible senses of “bat”: Sense 1: Bat (Sports equipment – cricket/baseball); Sense 2: Bat (Nocturnal flying mammal). | Sentence ID | Context words | Sense | | --- | --- | --- | | 1 | ball, cricket, score, stadium | Sports | | 2 | night, cave, wings, fly | Animal | | 3 | player, run, match, bat | Sports | | 4 | nocturnal, vampire, insect | Animal | | 5 | team, field, bowl, bat | Sports | | 6 | dark, flying, cave, night | Animal | a) Using the Naïve Bayes algorithm, classify the sense of “bat” in the given sentence. Apply Laplace Smoothing wherever applicable. b) Discuss the significance of Laplace Smoothing in Naïve Bayes Classification. [10 marks] ### Answer ### Given There are three Sports examples and three Animal examples. Treat training context lists as bag-of-words token counts. Exclude the ambiguous target “bat” from the test context. Use the fixed training vocabulary; ignore out-of-vocabulary test words hit and hard. The usable test features are ball, cricket and field. ### To find The posterior-score comparison for Sports and Animal, and the role of smoothing. ### Formula: which one and why $$ \mathrm{score}(s)=P(s)\prod_{w\in\mathrm{context}}P(w\mid s) $$ $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V} $$ P(s) is the class prior, C(w,s) counts occurrences of context word w in class s, Nₛ is the total number of context tokens in that class and V is the shared vocabulary size. Conditional independence lets us multiply word probabilities. The common denominator P(context) is unnecessary when only comparing classes. ### Solution ### Step 1. Calculate the priors $$ P(S)=\frac36=0.5,\qquad P(A)=\frac36=0.5 $$ ### Step 2. Count class tokens and vocabulary Sports has 4 + 4 + 4 = 12 tokens. Animal has 4 + 3 + 4 = 11 tokens. The shared training vocabulary has 20 distinct words: ball, cricket, score, stadium, player, run, match, bat, team, field, bowl, night, cave, wings, fly, nocturnal, vampire, insect, dark, flying. “bat”, “night” and “cave” repeat but count only once each towards V. ### Step 3. Look up each usable context word | Feature | Sports count | Animal count | P(word | Sports) | P(word | Animal) | | --- | --- | --- | --- | --- | | ball | 1 | 0 | (1+1)/(12+20) = 2/32 | (0+1)/(11+20) = 1/31 | | cricket | 1 | 0 | 2/32 | 1/31 | | field | 1 | 0 | 2/32 | 1/31 | ### Step 4. Multiply the Sports score $$ \mathrm{score}(S)=0.5\times\frac2{32}\times\frac2{32}\times\frac2{32}=0.0001220703125 $$ ### Step 5. Multiply the Animal score $$ \mathrm{score}(A)=0.5\times\frac1{31}\times\frac1{31}\times\frac1{31}\approx0.00001678359236 $$ ### Step 6. Compare the scores 0.0001220703125 is greater than 0.00001678359236. Therefore select Sports. Optional normalization gives P(Sports | retained features) ≈ 0.87913; normalization is not needed for the decision. ### Final answer The intended sense is **bat as sports equipment**. Ball, cricket and field favour that sense. **Yaad rakho:** Pehle class prior, phir har context-word likelihood multiply; bigger score wali sense choose karo. ### Why Laplace smoothing matters Without smoothing, all three test features have zero count in Animal. One zero factor makes the entire class score zero, even if other evidence could support that class. Add-one smoothing assigns nonzero probability to every word in the defined vocabulary and redistributes probability mass. It does not mean every unseen word outside the vocabulary can be inserted without changing the vocabulary model. ### If following the supplied paper key exactly The key also multiplies probabilities for hit and hard using the same denominators 32 and 31. That convention produces Sports = 1.19209290 × 10⁻⁷ and Animal ≈ 1.74647163 × 10⁻⁸, again selecting Sports. Its treatment of out-of-vocabulary words is not a normalized fixed-vocabulary model. State your feature convention clearly; the complete calculation above uses a consistent fixed training vocabulary. --- ## TEE · Q6A · Coreference, pronouns and Word2Vec Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q6a Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. i. Differentiate between co-reference resolution and pronoun resolution with suitable examples. ii. Explain the working principle of the Continuous Bag of Words (CBOW) model and Skipgram model in Word2Vec. Illustrate how it predicts the target word from its surrounding context. [10 marks] ### Answer ### i. Coreference and pronoun resolution **Coreference resolution** groups expressions that refer to the same entity, including names, descriptions and pronouns. **Pronoun resolution** identifies the referent of a pronoun; it is narrower in scope. | Expression / task | Example | | --- | --- | | Coreference cluster | “Dr Rao arrived. The surgeon opened her bag.” → {Dr Rao, the surgeon, her} | | Pronoun resolution | For the same text, resolve “her” to Dr Rao. | | Difference | Resolving the description “the surgeon” is coreference work even when no pronoun is involved. | ### ii. Word2Vec Word2Vec learns dense word embeddings by predicting words from local context. A dense embedding is a short learned list of real numbers for a word; words used in similar contexts can obtain related vectors. CBOW and Skip-gram use opposite prediction directions. | Feature | CBOW | Skip-gram | | --- | --- | --- | | Input | Surrounding context words | One centre word | | Prediction target | The centre word | Surrounding context words | | Training example | [the, cat, on, the] → sat | sat → the; sat → cat; sat → on; sat → the | | Representation | Combine context embeddings, often by averaging | Use the centre-word embedding for each context prediction | Example sentence: “The cat sat on the mat.” With a radius-two window around sat, the context is [The, cat, on, the]. In CBOW, obtain the four context embeddings, average or sum them, calculate scores for vocabulary words, and train the model to assign high probability to sat. In Skip-gram, use sat as input and create four centre–context training pairs. CBOW: nearby words → combine embeddings → predict centre word Skip-gram: centre word → its embedding → predict nearby words Training adjusts embedding parameters to improve prediction; the final word vectors are the learned representation. Thus the question’s phrase “predicts the target word from surrounding context” describes CBOW specifically; Skip-gram reverses that direction. **Yaad rakho:** CBOW: neighbours se centre. Skip-gram: centre se neighbours. --- ## TEE · Q6B · Hobbs: Ravi and Suresh Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q6b Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. i. Explain Hobb’s Algorithm. Discuss the steps with help of example and state its limitation. ii. Sentence: Ravi met Suresh at the station after he arrived from Delhi. Using Hobbs’ algorithm, determine the most likely antecedent of the pronoun “he”. Show the steps of traversal according to the algorithm. [10 marks] ### Answer ### i. Method **Hobbs’ algorithm** is a syntax-based pronoun-resolution procedure that searches constituency parse trees in a specified order. It proposes noun phrases as antecedents and filters incompatible candidates, for example by number and gender agreement. **Read the tree labels first:** NP means noun phrase; S means sentence/clause; VP means verb phrase. X is the current NP or S search node, and p is the upward path used to reach it. Breadth-first means shallower nodes before deeper nodes; left-to-right fixes the order at the same depth. An N-bar (N′) node is the noun-centred part of an NP in a parse using that label. ### Complete ordered procedure 1. **Locate the pronoun.** Start at the NP immediately dominating the pronoun: the smallest NP containing it. 2. **Make the first climb.** Move upward to the first NP or S above the starting NP. Call it X and record the path p. 3. **Search initially to the left.** Traverse branches below X to the left of p, breadth-first and left-to-right. Propose an NP only if another NP or S lies between that candidate and X. Check agreement and other applicable constraints for each proposed candidate; accept the first eligible compatible one. 4. **Check the highest sentence.** If X is the highest S and no candidate has been accepted, search previous sentence trees, most recent first. In each, traverse breadth-first and left-to-right and propose NPs. Accept the first eligible compatible candidate; if the available context contains none, report no resolved antecedent. 5. **Otherwise climb again.** Move from X to the next NP or S above it. Make that node the new X and record the new upward path p. 6. **Test a higher NP.** If the new X is an NP, propose X only when p did not pass through its nominal N-bar child. Apply the same compatibility checks. Use the labels in the supplied parse; do not invent a missing N-bar branch. 7. **Search left again.** Traverse the new X’s branches to the left of p, breadth-first and left-to-right, proposing NPs. The extra intervening-NP/S requirement in Step 3 belongs to the initial search. 8. **Search permitted right branches.** If X is an S, traverse branches to the right of p, breadth-first and left-to-right, but do not descend below any NP or S encountered. Propose the encountered eligible NPs and test their compatibility. 9. **Repeat or finish.** Return to Step 4 until a compatible candidate is accepted or the available sentence trees are exhausted. Record the candidate order and why each candidate was accepted or rejected. **Why the initial restriction matters:** in “John saw him”, the subject NP John is directly under the clause S, with no additional NP or S between them. Step 3 therefore does not propose John as the ordinary local antecedent of him. A reflexive such as himself is a different grammatical case. The worked question below shows the specific parse and traversal; selecting an antecedent is not simply choosing the nearest name. ### ii. Apply the method **Given:** the pronoun he occurs as the subject of the subordinate clause “after he arrived from Delhi”. Assume that this clause is attached to the main verb phrase. The relevant main-clause NPs are Ravi, Suresh and the station. **To find:** the first eligible singular-male antecedent in Hobbs’ syntactic search order. ``` S (main) ├── NP: Ravi ← first higher-left candidate └── VP ├── VP: met [NP Suresh] at [NP the station] └── SBAR: after └── S (embedded) ← first X ├── NP: he ← start └── VP: arrived from Delhi ``` ### Step 1. Start at the pronoun NP Locate NP “he” inside the after-clause. Move to that clause’s S node. No earlier noun phrase in this small clause provides a candidate. ### Step 2. Move to the higher sentence The embedded S is not the highest S, so Step 5 climbs through SBAR and VP to the main S. Step 7 now searches to the left of that new path. This is the later left search, so the extra initial-search restriction from Step 3 is not imposed again. ### Step 3. Examine candidates in structural order The main subject NP Ravi is encountered before the deeper object NP Suresh in the relevant breadth-first search. Ravi is singular and compatible with “he”. “the station” would fail the ordinary person constraint. ### Step 4. Choose the first compatible candidate Select Ravi under the assumed parse and agreement settings. Suresh remains a possible interpretation in ordinary language, but the syntactic preference chooses Ravi first. ### Limitations - The result depends on the accuracy and attachment choices of the parse tree. - Syntactic ordering alone does not model all semantic plausibility, discourse focus or world knowledge. - Agreement can eliminate candidates but cannot distinguish two compatible male singular people. - Long-distance references and sentences with multiple possible parses can remain ambiguous. **Final answer:** Hobbs’ syntactic preference is **Ravi** for this analysis. This is an algorithmic selection, not proof of the speaker’s intended referent. **Yaad rakho:** Nearest naam blindly nahi: tree ki search order aur agreement follow karo. --- ## TEE · Q7A · Text classification pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q7a Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. Explain the working of text classification in Natural Language Processing (NLP) and discuss the key steps involved in building a text classification system in context of real-world applications such as spam detection or news categorization. [10 marks] ### Answer **Text classification** assigns one or more predefined labels to a document or message based on its content. In spam detection the labels may be spam and not-spam; in news categorization they may be sports, business and politics. Labelled text → Prepare and split data → Represent text → Train classifier → Evaluate → Classify new text 1. **Define the labels and task.** Decide whether each text gets one label or multiple labels. Establish what counts as spam so annotators use a consistent rule. 2. **Collect and label representative examples.** Gather messages from the intended application and assign reliable labels. Include enough examples of each class and protect private information. 3. **Split training, validation and test data.** Use training data to learn, validation data to choose settings and an untouched test set for final evaluation. Keep duplicate or closely related texts from leaking between splits. 4. **Preprocess consistently.** Tokenize and normalize only as appropriate. In spam detection, URLs, currency symbols and capitalisation may contain useful information. 5. **Create features.** A classical baseline uses BoW or TF–IDF. A neural system may use embeddings or a pretrained contextual model. Fit the vocabulary and IDF on training data only. 6. **Train a classifier.** For example, multinomial Naïve Bayes estimates class priors and word likelihoods; logistic regression learns a decision boundary. Training uses the labelled examples. 7. **Evaluate.** Inspect a confusion matrix, precision, recall and F1 as well as accuracy. False spam flags may hide legitimate mail, so the costs of errors matter. 8. **Deploy and monitor.** Apply the same preprocessing and feature mapping to new texts. Monitor changing vocabulary and error patterns, and retrain when suitable new labelled data becomes available. ### Example prediction A message such as “Claim your free prize now” is transformed into its feature representation. The classifier compares the evidence for the two labels and outputs spam if its decision score crosses the chosen threshold. The label comes from learned patterns; the mere presence of one word should not be treated as a universally correct rule. **Yaad rakho:** Labelled examples → features → model → evaluation → new text ka label. --- ## TEE · Q7B · Corpora and Jaccard similarity Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/final-2025/final-2025-q7b Actual final examination · 2 December 2025 · 100 marks · 3 hours Question. i. Differentiate between a flat text corpus and an annotated corpus, and discuss how each can be applied in various NLP tasks with suitable examples. ii. Explain a real-world scenario where the Jaccard Similarity measure is more appropriate than Euclidean or Manhattan distance, and justify why Jaccard is the best choice in that case. [10 marks] ### Answer ### i. Flat and annotated corpora A **corpus** is a collection of language data. A flat text corpus stores text without the linguistic labels needed by a particular task. An annotated corpus adds labels or structure such as POS tags, entity spans, parse trees or sentence sentiment. | Basis | Flat text corpus | Annotated corpus | | --- | --- | --- | | Example | A collection of plain news articles | The same articles with PERSON/ORG spans marked | | Information | The original text and possibly simple metadata | Text plus explicit task-related linguistic labels | | Applications | Word frequency, collocations and unsupervised language-model training | Supervised POS tagging, NER or parser training and evaluation | | Preparation | Relatively inexpensive to gather, but still needs cleaning | Requires annotation guidelines, labour and quality checks | “Flat” describes the lack of the annotations being discussed, not an absence of all file organization. Both corpus types may still have document boundaries and metadata. ### ii. Jaccard in a set-based application Consider recommending articles using their topic-tag sets. Article A has {AI, NLP, ethics}; article B has {AI, NLP, healthcare}. We care about overlap among present tags, not how many irrelevant tags are absent from both. ### Given A = {AI, NLP, ethics}; B = {AI, NLP, healthcare}. ### To find The fraction of distinct present tags shared by the articles. ### Formula and why $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|} $$ The intersection counts common items; the union counts all distinct items present in either set. ### Step 1. Find common tags A ∩ B = {AI, NLP}, so the intersection size is 2. ### Step 2. Find all distinct present tags A ∪ B = {AI, NLP, ethics, healthcare}, so the union size is 4. ### Step 3. Divide $$ J(A,B)=\frac24=0.5 $$ Jaccard directly measures proportional overlap and ignores joint absences. Euclidean or Manhattan distance on binary tag vectors counts mismatches but does not by itself normalize by the union size. If frequency or magnitude were meaningful, those distance measures could be more appropriate. Here, membership-only tags and sparse sets make Jaccard a suitable choice. **Yaad rakho:** Jaccard: common present items / total distinct present items. --- ## TEE · Q1A · Generic NLP pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q1a Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Draw block diagram of generic NLP pipeline and explain the task associated with each block. [5 marks] ### Answer An NLP pipeline is an ordered workflow that transforms language data into a task-specific result. The exact components depend on the application; a generic supervised pipeline is shown below. Collect text → Clean and preprocess → Represent text → Train / apply model → Evaluate and deliver output | Block | Task | Example for sentiment analysis | | --- | --- | --- | | Data acquisition | Collect representative text and any required labels | Reviews labelled positive or negative | | Cleaning and preprocessing | Handle noise, tokenize and normalize appropriately | Remove HTML while preserving “not” | | Text representation | Convert tokens into usable model features | TF–IDF values or contextual embeddings | | Modeling | Learn patterns from training data and predict new labels | Train a classifier on labelled review features | | Evaluation | Measure performance on unseen held-out data and inspect errors | Precision, recall, F1 and confusion matrix | | Deployment and feedback | Deliver predictions and monitor changing data | Show review sentiment and monitor new product vocabulary | Training/validation/test separation must precede fitting the learned preprocessing and feature statistics. New inputs must use the same feature mapping as training. Not every task uses every linguistic level explicitly: a modern end-to-end model may combine several operations internally. **Yaad rakho:** Text lo → prepare karo → numbers banao → model lagao → result check aur use karo. --- ## TEE · Q1B · Content and function words Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q1b Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. i. Define function words and content words. ii. Which category (function words and content words) belong to open class and closed class? Justify your answer. iii. For the sentence “The small cat is sleeping under the tree”, identify the function words and content words. [5 marks] ### Answer **Content words** carry much of a sentence’s lexical meaning, usually including nouns, main verbs, adjectives and many adverbs. **Function words** primarily express grammatical relationships, including determiners, prepositions, conjunctions, pronouns and auxiliary verbs. | Category | Typical class | Reason | | --- | --- | --- | | Content words | Open class | New nouns, verbs and adjectives enter the language readily, such as new technology names | | Function words | Closed class | The grammatical inventory is relatively stable; new determiners or pronouns are much less frequently added | | Word | Role in this sentence | Category | | --- | --- | --- | | The | Determiner | Function | | small | Adjective | Content | | cat | Noun | Content | | is | Auxiliary supporting progressive tense | Function | | sleeping | Main verb | Content | | under | Preposition | Function | | the | Determiner | Function | | tree | Noun | Content | **Final answer:** content words = small, cat, sleeping, tree; function words = The, is, under, the. “is” is auxiliary here; a word’s role must be judged in context. “Closed” means relatively resistant to new members, not that change is logically impossible. **Yaad rakho:** Content words scene batate hain; function words grammar ka connection banate hain. --- ## TEE · Q1C · Word2Vec and GloVe Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q1c Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Differentiate between word2vec and GloVe in terms of their learning strategies. Which type of corpus would be more suitable for each method, and why? [5 marks] ### Answer Both Word2Vec and GloVe learn dense word vectors from distributional information: words occurring in related contexts tend to receive related representations. They differ mainly in the training objective and how corpus co-occurrence evidence is used. | Basis | Word2Vec | GloVe | | --- | --- | --- | | Learning strategy | Predictive: CBOW predicts a centre word; Skip-gram predicts nearby words | Count-based objective fitted to global word–context co-occurrence statistics | | Training evidence | Local context windows converted into prediction examples | A corpus-wide co-occurrence matrix and relationships among counts | | Parameters learned | Word/context embeddings that make context predictions accurate | Word/context embeddings and biases that fit weighted log co-occurrence counts | | Corpus considerations | Large token sequences with useful local context, including domain-specific text | A sufficiently large corpus from which reliable global co-occurrence statistics can be collected | For a domain such as medicine, either method can benefit from a representative medical corpus. Word2Vec is convenient when generating local training windows directly from a stream of tokens. GloVe is natural when a global sparse co-occurrence matrix can be constructed and stored. Corpus size, domain match, preprocessing and available compute matter; there is no rule that Word2Vec requires informal text while GloVe requires formal text. Both standard forms produce one static vector per vocabulary word, so multiple senses are blended. They do not automatically provide a different vector for the same word in each sentence. **Yaad rakho:** Word2Vec predicts local neighbours; GloVe fits global co-occurrence counts. --- ## TEE · Q1D · Anaphora, cataphora and coreference Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q1d Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Differentiate between cataphora and anaphora in terms of direction of reference. How coreference resolution is different from anaphora reference? [5 marks] ### Answer **Anaphora** points backward to an expression or information introduced earlier. **Cataphora** points forward to an expression introduced later. | Relation | Example | Direction | | --- | --- | --- | | Anaphora | “Riya arrived. She smiled.” She refers to the earlier Riya. | Backward | | Cataphora | “Before she spoke, Riya checked the microphone.” She refers to the later Riya. | Forward | **Coreference** means two or more expressions refer to the same entity. Coreference resolution groups such expressions, regardless of whether they are names, descriptions or pronouns and regardless of which is mentioned first. For “Dr Sen arrived. The surgeon smiled. She waved,” {Dr Sen, the surgeon, She} is a coreference cluster. The latter two mentions are also backward-looking references. Coreference is an identity relation; anaphora describes a dependency on prior discourse and is not always identity. For example, “I bought a house. The roof leaks” contains a bridging reference: the roof depends on the house for interpretation, but roof and house are not the same entity. **Yaad rakho:** Ana = pehle ki taraf; cata = aage ki taraf; coreference = same entity. --- ## TEE · Q2A · NLP tasks and syntax versus semantics Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q2a Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. i. Discuss five different tasks associated with Natural Language Processing. ii. Compare and contrast syntactic analysis and semantic analysis in NLP. Support your explanation with examples. [10 marks] ### Answer ### i. Five NLP tasks | Task | What it does | Example | | --- | --- | --- | | Machine translation | Expresses source-language content in a target language | English “Good morning” → Hindi greeting | | Sentiment analysis | Identifies the opinion or attitude expressed | “The service was excellent” → positive sentiment | | Named Entity Recognition | Finds and types entity spans | “Asha joined Google” → PERSON and ORGANIZATION | | Question answering | Returns an answer supported by a question’s relevant information | From a passage, identify who discovered a described result | | Text classification | Assigns predefined category labels | Classify an email as spam or legitimate | ### ii. Syntactic and semantic analysis | Basis | Syntactic analysis | Semantic analysis | | --- | --- | --- | | Main question | How are words grammatically organised? | What do the words and sentence mean? | | Typical output | POS tags, constituency or dependency structure | Word senses, semantic relations or a meaning representation | | Knowledge used | Grammar rules and structural patterns | Lexical meaning, context and semantic constraints | | Example | Identify “The boy” as subject NP and “ate an apple” as VP | Interpret the boy as eater and apple as the thing eaten | “Colourless green ideas sleep furiously” can have a plausible English grammatical structure while remaining semantically unusual. Syntax alone therefore does not guarantee ordinary meaning. Conversely, “boy eat apple yesterday” conveys a possible event despite grammatical errors. The analyses support each other. In “I saw the man with a telescope,” syntactic PP attachment determines which meaning is available, and semantic knowledge can help choose a plausible attachment. A full NLP system often needs both. **Yaad rakho:** Syntax = sentence ka structure; semantics = us structure ka meaning. --- ## TEE · Q2B · Laplace smoothing on a second corpus Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q2b Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Consider the following training corpus: 1. cats chase mice in the garden 2. dogs bark loudly at night 3. birds sing in the morning 4. children play games in the park Using a bigram language model with Laplace smoothing, evaluate the probability of the test sentence: “cats bark in the park.” Assume and as start and end of the sentence. [10 marks] ### Answer ### Given - cats chase mice in the garden - dogs bark loudly at night - birds sing in the morning - children play games in the park Test: cats bark in the park. Add one start token and one end token to every training and test sentence. ### To find The Laplace-smoothed bigram probability of the complete test sentence, including its end token. ### Formula: which one and why $$ P_{+1}(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)+1}{C(w_{i-1})+V} $$ C(previous,current) counts that adjacent pair in the training corpus. C(previous) counts its occurrences as a history. V is the vocabulary size. Add-one smoothing is necessary because an unseen pair must not force the whole sentence probability to zero. $$ P(w_1,\ldots,w_n,\langle/s\rangle\mid\langle s\rangle)=\prod_{i=1}^{n+1}P(w_i\mid w_{i-1}) $$ ### Solution ### Step 1. Fix the token and vocabulary convention Lowercase words and remove the final punctuation. The supplied course key counts both boundary tokens in V. Following that convention: at, bark, birds, cats, chase, children, dogs, games, garden, in, loudly, mice, morning, night, park, play, sing, the, , gives **V = 20**. The start marker supplies history; it is not an extra probability factor before the first word. ### Step 2. List every required transition The complete test chain is → cats → bark → in → the → park → . ### Step 3. Count first, then add one | Required bigram | History count | Pair count | Smoothed probability | | --- | --- | --- | --- | | → cats | 4 | 1 | (1+1)/(4+20) = 1/12 | | cats → bark | 1 | 0 | (0+1)/(1+20) = 1/21 | | bark → in | 1 | 0 | (0+1)/(1+20) = 1/21 | | in → the | 3 | 3 | (3+1)/(3+20) = 4/23 | | the → park | 3 | 1 | (1+1)/(3+20) = 2/23 | | park → | 1 | 1 | (1+1)/(1+20) = 2/21 | Count only adjacent words, never words that merely appear in the same sentence. The unseen pairs have count zero, so their numerator becomes one. ### Step 4. Multiply all six probabilities $$ P=\frac{2}{24}\times\frac{1}{21}\times\frac{1}{21}\times\frac{4}{23}\times\frac{2}{23}\times\frac{2}{21} $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. → cats | $$ 1\times\frac{1}{12}=\frac{1}{12} $$ | | 2. cats → bark | $$ \frac{1}{12}\times\frac{1}{21}=\frac{1}{252} $$ | | 3. bark → in | $$ \frac{1}{252}\times\frac{1}{21}=\frac{1}{5292} $$ | | 4. in → the | $$ \frac{1}{5292}\times\frac{4}{23}=\frac{1}{30429} $$ | | 5. the → park | $$ \frac{1}{30429}\times\frac{2}{23}=\frac{2}{699867} $$ | | 6. park → | $$ \frac{2}{699867}\times\frac{2}{21}=\frac{4}{14697207} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{4}{14697207}\approx 2.72160554\times10^{-7} $$ ### Step 5. Interpret the result This is a joint sentence probability, so a small value is normal. Do not add transition probabilities; each word must occur in sequence, so the chain multiplies them. ### Final answer **P ≈ 2.72160554 × 10-7** with the stated course vocabulary convention. **Yaad rakho:** Pair count mein 1 jodo, previous-word count mein V; phir saare fractions multiply karo. **Convention note:** Some implementations exclude from the predicted vocabulary. Their V is one smaller and their numerical probability differs. In an exam, state V explicitly and follow the convention specified in the question or class solution. --- ## TEE · Q3A · Top-down parsing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q3a Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. In the context of natural language processing, explain the concept of top-down and bottom-up parsing approaches. Apply top-down syntax parsing approach to the sentence “The tall boy saw a girl in the park.” and draw a parse tree using the CFG grammar rules: S → NP VP; NP → Det Adj N | Det N; VP → V NP | VP PP; PP → P NP. Lexicon: Det → {the | a}; Adj → {tall}; N → {boy | girl | park}; V → {saw}; P → {in}. [10 marks] ### Answer | Aspect | Top-down | Bottom-up | | --- | --- | --- | | Start | The start symbol S | The input words | | Operation | Expand a nonterminal using grammar productions | Combine words and constituents using productions in reverse | | Success | Leaves match the complete input | All input is reduced to S | | Possible difficulty | Naive recursive descent can loop on left recursion or need backtracking | A naive shift-reduce parser may face shift/reduce conflicts | ### Given The sentence contains the tokens the / tall / boy / saw / a / girl / in / the / park. Lowercase the initial “The” to match the lexicon. The full stop is punctuation outside the listed grammar. ### To find A derivation beginning at S and a parse tree matching the whole sentence. ### Rule: which one and why A context-free production replaces one nonterminal with its right-hand sequence. A top-down derivation begins at S and expands the desired structure. The supplied grammar attaches the PP through VP → VP PP; it does not provide NP → NP PP. ### Step 1. Expand the sentence S ⇒ NP VP. Choose NP → Det Adj N for “the tall boy”. ### Step 2. Match the subject words Det ⇒ the, Adj ⇒ tall, N ⇒ boy. The remaining material must be generated by VP. ### Step 3. Allow a verb phrase followed by a PP Expand the outer VP using VP → VP PP. Then expand the inner VP using VP → V NP. This finite derivation selects the recursive rule once. ### Step 4. Match the verb and object V ⇒ saw. Choose NP → Det N, then Det ⇒ a and N ⇒ girl. ### Step 5. Expand the prepositional phrase PP ⇒ P NP. P ⇒ in. NP ⇒ Det N, with Det ⇒ the and N ⇒ park. ### Step 6. Check the complete leaf order The leaves read “the tall boy saw a girl in the park”, exactly the normalized input. The PP “in the park” attaches to the VP under this supplied grammar. SNPVPDetAdjNVPPPVNPPNPDetNDetNThetallboysawagirlinthepark Top-down parse tree showing the PP attached to VP. ### Complete derivation S ⇒ NP VP ⇒ Det Adj N VP ⇒ the tall boy VP ⇒ the tall boy VP PP ⇒ the tall boy V NP PP ⇒ the tall boy saw Det N PP ⇒ the tall boy saw a girl P NP ⇒ the tall boy saw a girl in Det N ⇒ the tall boy saw a girl in the park. **Implementation caution:** VP → VP PP is left-recursive. A naive recursive-descent implementation may loop; it must transform or otherwise handle the grammar. A valid finite top-down derivation, as shown above, still exists. **Yaad rakho:** Rule jo diya hai wahi use karo: yahan “in the park” VP ka part hai. --- ## TEE · Q3B · Lesk word-sense disambiguation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q3b Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Consider the sentence: “The bank can ensure your deposits are safe.” The word “bank” has the following two possible senses (WordNet-style glosses): Bank₁: Sloping land especially beside a body of water. Bank₂: A financial institution that accepts deposits and channels the money into lending activities. i. Apply the LESK algorithm to determine the correct sense of the word “bank” in the given sentence. Show the steps clearly. ii. Briefly discuss limitations of the LESK algorithm in handling ambiguous words in real-world contexts. [10 marks] ### Answer ### Given Context sentence and the two supplied dictionary glosses. Use a simplified Lesk method: lowercase, remove function words, exclude the target bank and compare distinct content-word overlap. Do not add synonyms that the question does not supply. ### To find Which candidate gloss has the greatest overlap with the local context. ### Formula: which one and why $$ \mathrm{score}(s)=|C\cap G_s|,\qquad \hat s=\arg\max_s\mathrm{score}(s) $$ C is the set of context content words. Gₛ is the content-word set of the gloss for sense s. The intersection contains words appearing in both; its size is the score. This is an overlap count, not a probability. ### Solution ### Step 1. Prepare the context Remove The, can, your, are and the target bank. A reasonable content set is C = {ensure, deposits, safe}. If “can” is retained under another fixed stop-list it matches neither gloss and does not change the result. ### Step 2. Prepare Bank₁’s gloss G₁ = {sloping, land, especially, beside, body, water}. There is no shared word with {ensure, deposits, safe}. ### Step 3. Calculate Bank₁’s score $$ C\cap G_1=\varnothing,\qquad \mathrm{score}(1)=0 $$ ### Step 4. Prepare Bank₂’s gloss and count overlap G₂ = {financial, institution, accepts, deposits, channels, money, lending, activities}. The shared word is deposits. ### Step 5. Calculate Bank₂’s score and select $$ C\cap G_2=\{\mathrm{deposits}\},\qquad \mathrm{score}(2)=1 $$ 1 > 0, so choose the financial-institution sense. ### Final answer **Bank₂: financial institution.** The explicit overlap “deposits” supports this choice. **Yaad rakho:** Lesk: context aur har meaning ki definition mein common words count karo; highest overlap choose karo. ### ii. Limitations - Short contexts or short glosses may produce no overlaps or ties. - Exact word matching misses related words and synonyms, such as deposit versus savings, unless an extension is defined. - Irrelevant frequent words can create misleading overlaps if preprocessing is poor. - Dictionary glosses may omit a domain-specific sense or fail to express the context’s wording. - Simplified Lesk uses limited local information and may miss discourse or real-world knowledge. Lemmatization, richer glosses and semantic similarity can improve variants, but the exact variant and tie-breaking rule should be stated. --- ## TEE · Q4A · Bag of Words and three cosine similarities Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q4a Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Consider the following three documents: Document A: “The dog chased the cat.” Document B: “The cat chased the mouse.” Document C: “The dog ran after the cat.” Using the Bag of Words (BoW) model, represent each document as a vector after removing the stop words {The, by, was}. i. Construct the term-document matrix. ii. Compute the cosine similarity between each pair of documents (A–B, A–C, and B–C). iii. Interpret and comment on the results obtained in terms of document similarity. [10 marks] ### Answer ### Given Three documents. Lowercase and remove punctuation. Remove the specified stop words case-insensitively; “the” is removed everywhere. Keep “after” because it is not in the supplied stop list. Use raw word counts. ### To find A common-vocabulary count matrix, three cosine values and their interpretation. ### Formula: which one and why $$ \mathrm{BoW}_j(d)=C(v_j,d) $$ vⱼ is vocabulary term j. The coordinate stores its count in document d; order in the sentence is not retained. $$ \cos(\mathbf a,\mathbf b)=\frac{\sum_j a_jb_j}{\sqrt{\sum_j a_j^2}\sqrt{\sum_j b_j^2}} $$ ### Solution ### Step 1. Clean each document A: dog chased cat. B: cat chased mouse. C: dog ran after cat. ### Step 2. Fix vocabulary and matrix Vocabulary: dog, chased, cat, mouse, ran, after. | Term | A | B | C | | --- | --- | --- | --- | | dog | 1 | 0 | 1 | | chased | 1 | 1 | 0 | | cat | 1 | 1 | 1 | | mouse | 0 | 1 | 0 | | ran | 0 | 0 | 1 | | after | 0 | 0 | 1 | A = [1,1,1,0,0,0]; B = [0,1,1,1,0,0]; C = [1,0,1,0,1,1]. ### Step 3. Calculate vector lengths $$ \|A\|=\sqrt{1^2+1^2+1^2}=\sqrt3 $$ $$ \|B\|=\sqrt{1^2+1^2+1^2}=\sqrt3 $$ $$ \|C\|=\sqrt{1^2+1^2+1^2+1^2}=2 $$ ### Step 4. Calculate A–B $$ A\cdot B=(1)(0)+(1)(1)+(1)(1)+(0)(1)+(0)(0)+(0)(0)=2 $$ $$ \cos(A,B)=\frac2{\sqrt3\sqrt3}=\frac23\approx0.666667 $$ ### Step 5. Calculate A–C $$ A\cdot C=(1)(1)+(1)(0)+(1)(1)+(0)(0)+(0)(1)+(0)(1)=2 $$ $$ \cos(A,C)=\frac2{\sqrt3\cdot2}=\frac1{\sqrt3}\approx0.577350 $$ ### Step 6. Calculate B–C $$ B\cdot C=(0)(1)+(1)(0)+(1)(1)+(1)(0)+(0)(1)+(0)(1)=1 $$ $$ \cos(B,C)=\frac1{\sqrt3\cdot2}\approx0.288675 $$ ### Final answer **A–B = 0.666667; A–C = 0.577350; B–C = 0.288675.** A and B have the highest lexical overlap under this representation. B and C share only cat. BoW similarity reflects token overlap, not full semantic equivalence or word order. **Yaad rakho:** Same vocabulary order rakho. Corresponding counts multiply-add; phir dono lengths se divide. --- ## TEE · Q4B · A second HMM sequence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q4b Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Consider the following POS tagged corpus. Apply Hidden Markov Model (HMM) approach to: 1. Dog/Noun eats/Verb bread/Noun 2. The/Det cat/Noun sleeps/Verb 3. A/Det dog/Noun eats/Verb fish/Noun i. Construct the emission probability matrix. ii. Construct the transmission probability matrix. iii. Calculate the probability of the given sequence of events: The/Det dog/Noun eats/Verb bread/Noun. Note: Include start and end characters for all the sentences. [10 marks] ### Answer The repeated verb eats has a different emission count from the final paper; recalculate it rather than copying the previous numerical result. ### Given | Sentence | Tagged sequence | | --- | --- | | 1 | dog/Noun eats/Verb bread/Noun | | 2 | the/Det cat/Noun sleeps/Verb | | 3 | a/Det dog/Noun eats/Verb fish/Noun | Lowercase the corpus. Test sequence: the/Det dog/Noun eats/Verb bread/Noun. Add Start and End markers. ### To find Emission and transition matrices, then the probability of the supplied tagged sequence. ### Formula: which one and why $$ P(w\mid t)=\frac{C(t,w)}{C(t)} $$ $$ P(t_j\mid t_i)=\frac{C(t_i,t_j)}{\sum_kC(t_i,t_k)} $$ Emission divides the word’s tag-specific count by the total tokens with that tag. Transition divides the tag-pair count by all transitions leaving the previous tag, including End. $$ P(\mathbf w,\mathbf t)=P(t_1\mid\mathrm{Start})\prod_iP(w_i\mid t_i)\prod_{i=2}^{n}P(t_i\mid t_{i-1})P(\mathrm{End}\mid t_n) $$ ### Solution ### Step 1. Count tag totals and word frequencies Det: the=1, a=1, total2. Noun: dog=2, bread=1, cat=1, fish=1, total5. Verb: eats=2, sleeps=1, total3. ### Step 2. Construct emission matrix | Tag / word | a | bread | cat | dog | eats | fish | sleeps | the | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Det | 1/2 | 0 | 0 | 0 | 0 | 0 | 0 | 1/2 | | Noun | 0 | 1/5 | 1/5 | 2/5 | 0 | 1/5 | 0 | 0 | | Verb | 0 | 0 | 0 | 0 | 2/3 | 0 | 1/3 | 0 | ### Step 3. Count transitions from the three chains Start → Noun → Verb → Noun → End Start → Det → Noun → Verb → End Start → Det → Noun → Verb → Noun → End | From | Det count | Noun count | Verb count | End count | Total | | --- | --- | --- | --- | --- | --- | | Start | 2 | 1 | 0 | 0 | 3 | | Det | 0 | 2 | 0 | 0 | 2 | | Noun | 0 | 0 | 3 | 2 | 5 | | Verb | 0 | 2 | 0 | 1 | 3 | ### Step 4. Divide outgoing counts | From / to | Det | Noun | Verb | End | | --- | --- | --- | --- | --- | | Start | 2/3 | 1/3 | 0 | 0 | | Det | 0 | 1 | 0 | 0 | | Noun | 0 | 0 | 3/5 | 2/5 | | Verb | 0 | 2/3 | 0 | 1/3 | All nonterminal rows sum to one. No smoothing is requested. ### Step 5. Write every sequence factor | Event | Probability | | --- | --- | | Start → Det | 2/3 | | Det emits the | 1/2 | | Det → Noun | 1 | | Noun emits dog | 2/5 | | Noun → Verb | 3/5 | | Verb emits eats | 2/3 | | Verb → Noun | 2/3 | | Noun emits bread | 1/5 | | Noun → End | 2/5 | ### Step 6. Multiply all factors $$ P=\frac23\times\frac12\times1\times\frac25\times\frac35\times\frac23\times\frac23\times\frac15\times\frac25 $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. Start → Det | $$ 1\times\frac{2}{3}=\frac{2}{3} $$ | | 2. Det emits the | $$ \frac{2}{3}\times\frac{1}{2}=\frac{1}{3} $$ | | 3. Det → Noun | $$ \frac{1}{3}\times1=\frac{1}{3} $$ | | 4. Noun emits dog | $$ \frac{1}{3}\times\frac{2}{5}=\frac{2}{15} $$ | | 5. Noun → Verb | $$ \frac{2}{15}\times\frac{3}{5}=\frac{2}{25} $$ | | 6. Verb emits eats | $$ \frac{2}{25}\times\frac{2}{3}=\frac{4}{75} $$ | | 7. Verb → Noun | $$ \frac{4}{75}\times\frac{2}{3}=\frac{8}{225} $$ | | 8. Noun emits bread | $$ \frac{8}{225}\times\frac{1}{5}=\frac{8}{1125} $$ | | 9. Noun → End | $$ \frac{8}{1125}\times\frac{2}{5}=\frac{16}{5625} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{16}{5625}\approx0.0028444444 $$ ### Final answer **P = 16/5625 ≈ 0.0028444444** for these words and specified tags jointly. The two occurrences of eats give P(eats | Verb) = 2/3. **Yaad rakho:** Same tag structure ho sakta hai, lekin word emission counts dobara count karna. **Boundary reminder:** Include P(Det | Start) = 2/3 and P(End | Noun) = 2/5. The handwritten key’s final line instead inserts the overall Det fraction 2/10 and omits End; the complete boundary-aware probability requested here is 0.0028444444. --- ## TEE · Q5A · Why TF–IDF is useful Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q5a Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Justify with an appropriate example the importance of TF-IDF technique in the document representation and information retrieval task. [10 marks] ### Answer **TF–IDF** represents each document using weights that combine within-document term frequency with how unusual a term is across the collection. It supports information retrieval by giving larger weight to terms that are frequent in a document but not present in every document. ### Given Use three short documents: D₁ = “solar energy solar”; D₂ = “wind energy”; D₃ = “energy policy”. Use raw-count TF and natural-log unsmoothed IDF. N = 3. ### To find A TF–IDF representation illustrating why distinctive terms help retrieve relevant documents. ### Formula: which one and why $$ \mathrm{TFIDF}(t,d)=C(t,d)\ln\frac{N}{\mathrm{df}(t)} $$ C counts occurrences within a document; df counts documents containing the term. The logarithm reduces the scale of very large rarity ratios. ### Solution ### Step 1. Count term occurrences | Term | D₁ | D₂ | D₃ | df | | --- | --- | --- | --- | --- | | solar | 2 | 0 | 0 | 1 | | energy | 1 | 1 | 1 | 3 | | wind | 0 | 1 | 0 | 1 | | policy | 0 | 0 | 1 | 1 | ### Step 2. Calculate IDF $$ \mathrm{IDF}(solar)=\mathrm{IDF}(wind)=\mathrm{IDF}(policy)=\ln3\approx1.098612 $$ $$ \mathrm{IDF}(energy)=\ln(3/3)=0 $$ ### Step 3. Multiply each count by its IDF | Term | D₁ weight | D₂ weight | D₃ weight | | --- | --- | --- | --- | | solar | 2 × 1.098612 = 2.197224 | 0 | 0 | | energy | 0 | 0 | 0 | | wind | 0 | 1.098612 | 0 | | policy | 0 | 0 | 1.098612 | ### Step 4. Connect the representation to retrieval A query containing “solar” matches D₁’s high solar coordinate. A query containing “wind” matches D₂. “energy” alone cannot distinguish these documents under this unsmoothed collection because it occurs in all three. A retrieval system can compare query and document vectors, commonly by cosine similarity. ### Final answer TF–IDF preserves distinctive lexical evidence and reduces dominance by collection-wide frequent terms. D₁ is most relevant to a solar query under these features. **Yaad rakho:** Document mein frequent + collection mein rare = high TF–IDF. ### Strengths and limits The representation is sparse, interpretable and useful as a fast baseline. However, basic TF–IDF ignores word order and does not recognise synonyms or multiple word senses automatically. Its weights depend on the collection and on the chosen TF/IDF convention. Fit IDF on training or indexed documents, not on a hidden test set. --- ## TEE · Q5B · Sentence-window vectors and cosine Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q5b Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. Consider the paragraph: The farmer planted seeds in the soil. The rain watered the soil. The sun helped the plants grow. The farmer harvested the plants. The soil became rich and fertile. Target words: farmer, soil, sun, plant. Context words: plant, water, grow, harvest. i. Construct a co-occurrence matrix for the target and context words. Take the window size as one whole sentence. ii. Evaluate the similarity between the vectors of the target words “farmer-soil”, “farmer-sun”, and “farmer-plant” using the cosine similarity. iii. Plot the target word vectors by considering only two dimensions: “planted” and “harvested”. [10 marks] ### Answer ### Given Five sentence windows. For alignment with the supplied matrix, target plant means the noun plants, occurring in sentences 3 and 4. Context plant covers planted and plants; watered → water; harvested → harvest. Count a target–context pair once per sentence in which both are present, including the plant–plant sentence-presence diagonal. ### To find The target–context matrix, three cosine similarities and a two-coordinate plot. ### Formula: which one and why $$ M_{ij}=\sum_s\mathbf1[\text{target}_i\text{ and context}_j\text{ present in }s] $$ $$ \cos(\mathbf a,\mathbf b)=\frac{\sum_j a_jb_j}{\sqrt{\sum_j a_j^2}\sqrt{\sum_j b_j^2}} $$ This is a sentence-presence matrix, not a position-based sliding window that excludes a token from its own context. Stating the convention is necessary because those alternatives give a different diagonal. ### Solution ### Step 1. Normalize and identify each sentence’s contribution | Sentence | Targets present | Context lemmas present | | --- | --- | --- | | 1 | farmer, soil | plant | | 2 | soil | water | | 3 | sun, plant | plant, grow | | 4 | farmer, plant | plant, harvest | | 5 | soil | none | ### Step 2. Fill all matrix rows | Target | plant | water | grow | harvest | | --- | --- | --- | --- | --- | | farmer | 2 | 0 | 0 | 1 | | soil | 1 | 1 | 0 | 0 | | sun | 1 | 0 | 1 | 0 | | plant | 2 | 0 | 1 | 1 | Farmer meets context plant in sentences 1 and 4. Soil meets plant in sentence 1 and water in sentence 2. Sun meets plant and grow in sentence 3. The noun plant occurs in sentences 3 and 4, producing its last row. ### Step 3. Compute farmer–soil $$ \mathbf f\cdot\mathbf s=(2)(1)+(0)(1)+(0)(0)+(1)(0)=2 $$ $$ \|\mathbf f\|=\sqrt{2^2+0^2+0^2+1^2}=\sqrt5,\quad\|\mathbf s\|=\sqrt2 $$ $$ \cos(\mathbf f,\mathbf s)=\frac2{\sqrt{10}}\approx0.632456 $$ ### Step 4. Compute farmer–sun $$ \mathbf f\cdot\mathbf u=(2)(1)+(0)(0)+(0)(1)+(1)(0)=2 $$ $$ \|\mathbf u\|=\sqrt2,\quad\cos(\mathbf f,\mathbf u)=\frac2{\sqrt{10}}\approx0.632456 $$ ### Step 5. Compute farmer–plant $$ \mathbf f\cdot\mathbf p=(2)(2)+(0)(0)+(0)(1)+(1)(1)=5 $$ $$ \|\mathbf p\|=\sqrt{2^2+0^2+1^2+1^2}=\sqrt6 $$ $$ \cos(\mathbf f,\mathbf p)=\frac5{\sqrt{30}}\approx0.912871 $$ ### Step 6. Plot the requested two coordinates Interpret the axis names planted and harvested as their normalized context features plant and harvest. Farmer=(2,1), soil=(1,0), sun=(1,0), plant=(2,1). Farmer and plant overlap; soil and sun overlap. 001122farmer, plantsoil, sunplant (planted)harvest (harvested) The plot uses only the requested two coordinates; cosine uses all context columns. ### Final answer Using the stated noun-target convention: **farmer–soil = 0.632456, farmer–sun = 0.632456, farmer–plant = 0.912871.** **Yaad rakho:** Plant noun aur plant verb ko same maana ya separate? Pehle convention likho, phir count karo. ### Two details that prevent an incorrect answer **Arithmetic:** 5/√30 = 0.912871; the source key’s printed approximation 0.9444 does not equal its own fraction. Keep the exact fraction until the last step. **Normalization:** If the question instead requires all noun and verb forms of plant to be merged as targets, sentence 1 also contains the plant target. Its row becomes [3,0,1,1], and farmer–plant becomes 7/√55 ≈ 0.943880. The paper leaves this choice unstated; the main worked solution explicitly uses the noun-target interpretation that reproduces its supplied matrix. --- ## TEE · Q6A · Pronoun approaches and lexical relations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q6a Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. i. Briefly describe any two approaches to pronoun resolution and mention one advantage of each. ii. Describe the difference between Homonymy and Synonymy with suitable examples. Discuss how these two relations differently affect natural language processing applications? [10 marks] ### Answer ### i. Two approaches to pronoun resolution | Approach | How it works | Advantage | | --- | --- | --- | | Rule-based / syntactic | Uses parse structure, agreement, recency and linguistic constraints. Hobbs’ algorithm searches parse trees for eligible antecedents. | Interpretable: a proposed antecedent can be traced to explicit rules. | | Statistical or neural | Learns candidate–pronoun compatibility or whole-document coreference patterns from labelled examples, using context and linguistic features or contextual embeddings. | Can learn interactions and contextual cues that are difficult to enumerate as hand-written rules. | For “Asha dropped the glass because it was slippery,” a system should evaluate candidate referents and context rather than merely choose the most recent noun. Rules can constrain number and syntax; learned models can use compatibility patterns, though neither approach is guaranteed to resolve every ambiguity. ### ii. Homonymy and synonymy | Relation | Definition | Example | | --- | --- | --- | | Homonymy | Same spelling or sound, unrelated meanings | bank as financial institution versus bank as river edge | | Synonymy | Different words with the same or closely related meanings in a context | car and automobile; big and large | Homonymy creates an ambiguity problem: a search for bank may retrieve finance or rivers. WSD uses context to choose the intended sense. Synonymy creates a vocabulary-matching problem: a document saying automobile may be relevant to a query saying car. Query expansion or semantic representations can improve recall by connecting related terms. The remedies differ. Do not merge unrelated homonym senses simply because their spelling matches. Do not insist on exact surface-form matching when different words express the same concept. Synonyms are also context-sensitive: “big mistake” and “large mistake” are not always equally natural. **Yaad rakho:** Homonym: same word-form, different meaning. Synonym: different words, similar meaning. --- ## TEE · Q6B · Hobbs: Ramesh and Shyam Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q6b Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. i. Explain Hobbs’ Algorithm for pronoun resolution. State its main steps and limitations. ii. Apply Hobbs’ Algorithm to the sentence: “When Ramesh gave Shyam a book, he smiled.” Identify the antecedent of “he” by showing traversal steps. [10 marks] ### Answer **Hobbs’ algorithm** is a syntax-based pronoun-resolution procedure that searches constituency parse trees in a specified order. It proposes noun phrases as antecedents and filters incompatible candidates, for example by number and gender agreement. **Read the tree labels first:** NP means noun phrase; S means sentence/clause; VP means verb phrase. X is the current NP or S search node, and p is the upward path used to reach it. Breadth-first means shallower nodes before deeper nodes; left-to-right fixes the order at the same depth. An N-bar (N′) node is the noun-centred part of an NP in a parse using that label. ### Complete ordered procedure 1. **Locate the pronoun.** Start at the NP immediately dominating the pronoun: the smallest NP containing it. 2. **Make the first climb.** Move upward to the first NP or S above the starting NP. Call it X and record the path p. 3. **Search initially to the left.** Traverse branches below X to the left of p, breadth-first and left-to-right. Propose an NP only if another NP or S lies between that candidate and X. Check agreement and other applicable constraints for each proposed candidate; accept the first eligible compatible one. 4. **Check the highest sentence.** If X is the highest S and no candidate has been accepted, search previous sentence trees, most recent first. In each, traverse breadth-first and left-to-right and propose NPs. Accept the first eligible compatible candidate; if the available context contains none, report no resolved antecedent. 5. **Otherwise climb again.** Move from X to the next NP or S above it. Make that node the new X and record the new upward path p. 6. **Test a higher NP.** If the new X is an NP, propose X only when p did not pass through its nominal N-bar child. Apply the same compatibility checks. Use the labels in the supplied parse; do not invent a missing N-bar branch. 7. **Search left again.** Traverse the new X’s branches to the left of p, breadth-first and left-to-right, proposing NPs. The extra intervening-NP/S requirement in Step 3 belongs to the initial search. 8. **Search permitted right branches.** If X is an S, traverse branches to the right of p, breadth-first and left-to-right, but do not descend below any NP or S encountered. Propose the encountered eligible NPs and test their compatibility. 9. **Repeat or finish.** Return to Step 4 until a compatible candidate is accepted or the available sentence trees are exhausted. Record the candidate order and why each candidate was accepted or rejected. **Why the initial restriction matters:** in “John saw him”, the subject NP John is directly under the clause S, with no additional NP or S between them. Step 3 therefore does not propose John as the ordinary local antecedent of him. A reflexive such as himself is a different grammatical case. The worked question below shows the specific parse and traversal; selecting an antecedent is not simply choosing the nearest name. ### Given The pronoun he is the subject of the main clause. The preceding subordinate clause is “When Ramesh gave Shyam a book”. Assume the temporal clause attaches to the whole main sentence. ### To find The first suitable antecedent in the left-to-right breadth-first search of the preceding clause. ``` S (main) ← first X ├── SBAR: When │ └── S (embedded) ← intervening S │ ├── NP: Ramesh ← first candidate │ └── VP │ ├── V: gave │ ├── NP: Shyam │ └── NP: a book ├── NP: he ← start └── VP: smiled ``` ### Step 1. Locate the pronoun NP Start at NP “he” and move upward to its S ancestor. Record the path from he. ### Step 2. Search branches before the pronoun The initial when-clause lies to the left of the path. Apply Step 3: its embedded S lies between each of its noun phrases and the main S, so these NPs satisfy the initial intervening-NP/S requirement. Traverse left-to-right and breadth-first. ### Step 3. Propose noun phrases The subject NP Ramesh is encountered before the deeper object NP Shyam and object NP a book. Ramesh agrees with the assumed singular-male pronoun. ### Step 4. Accept the first compatible antecedent Select Ramesh. Shyam is also a semantically possible referent in ordinary discourse, but the algorithm’s stated syntactic preference selects Ramesh first. ### Limitations - Requires a suitable and accurate constituency parse. - Agreement information may be unknown or insufficient when several people match. - Ignores much of the event semantics, discourse focus and world knowledge. - An ambiguous parse or different attachment can change candidate order. **Final answer:** **Ramesh**, under the shown parse and ordinary agreement assumptions. A syntactic preference is not a guarantee about the author’s intended meaning. **Yaad rakho:** Left branches ko breadth-first dekho; pehla eligible matching NP choose karo. --- ## TEE · Q7A · Information retrieval Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q7a Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. i. Explain the concept of Information Retrieval, highlighting its role in Recommender Systems and Search Engines. ii. Discuss the various steps involved in the Information Retrieval process. [10 marks] ### Answer ### i. Concept and roles **Information Retrieval (IR)** finds and ranks documents or other information objects relevant to a user’s information need. A query is the user’s expression of that need. IR typically returns a ranked set of items; information extraction instead identifies structured facts or fields inside those items. | Application | Role of IR | Example | | --- | --- | --- | | Search engine | Matches a submitted query against a large indexed collection and ranks candidate pages | A search for “symptoms of dehydration” returns relevant health pages | | Recommender system | Retrieves candidate items using a user profile, recent activity or a content representation; a later model may personalize ranking | Retrieve articles similar to topics the user reads | Retrieval is a component of many recommenders; recommendation also involves preferences, feedback, novelty and business constraints. A recommender need not wait for an explicit textual query. ### ii. Retrieval process Collect → Preprocess → Index → Process query → Retrieve candidates → Rank → Evaluate 1. **Collect and define documents.** Crawl or ingest documents, record metadata and remove unwanted duplicates. 2. **Preprocess.** Tokenize and normalize using rules suitable for both documents and queries. Preserve task-relevant information. 3. **Build representations and an index.** An inverted index maps each term to the documents containing it; a dense retrieval system stores vector representations. 4. **Process the query.** Apply compatible tokenization, spelling handling or expansion, then obtain query features. 5. **Retrieve candidates.** Use the index to identify potentially relevant documents efficiently. 6. **Score and rank.** A lexical method may use TF–IDF/BM25-style evidence; a semantic method compares representations. Rank scores estimate relevance rather than proving correctness. 7. **Present results.** Return titles, snippets and links, possibly with filtering or diversification. 8. **Evaluate and update.** Use relevance judgments and measures such as precision, recall or ranking metrics. Refresh the index as documents change. Example: for “solar energy storage”, an inverted index retrieves documents containing those terms; a ranking function rewards useful matches. A semantic system may also retrieve relevant material using related terms such as batteries. **Yaad rakho:** IR mein relevant document dhoondho aur rank karo; extraction mein document ke andar facts nikalo. --- ## TEE · Q7B · Gutenberg corpus and normalization Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/reexam-2026/reexam-2026-q7b Actual re-examination · 9 February 2026 · 100 marks · 3 hours Question. i. Discuss how the Gutenberg Corpus can be utilized for various text analysis tasks in Natural Language Processing? ii. Explain the importance of text normalization in Natural Language Processing (NLP). Illustrate your answer with a suitable example of an application where text normalization plays a crucial role. [10 marks] ### Answer ### i. Using the Gutenberg corpus The Gutenberg corpus provides literary texts, commonly used through tools such as NLTK’s Gutenberg collection. It supplies naturally occurring sentences and longer documents for exploring language patterns. The available subset and licensing should be checked before reuse in a new distribution. | Task | Use of the corpus | | --- | --- | | Frequency and vocabulary study | Count tokens and word types, inspect common words and compare lexical diversity across texts | | Concordances and collocations | View a word in context and discover words that occur near each other | | N-gram language modeling | Estimate unigram or bigram counts and compare sequence probabilities | | Stylistic or authorship analysis | Compare sentence lengths, vocabulary and function-word patterns across authors | | Text preprocessing experiments | Study the effect of tokenization, case normalization and lemmatization on counts | A sound workflow selects files, inspects encoding and boundaries, tokenizes consistently, computes the relevant statistics and interprets them against corpus composition. Literary text is not automatically representative of chat, medical records or present-day search queries; domain mismatch matters. ### ii. Text normalization **Normalization** converts selected surface variants into a consistent form so that equivalent expressions do not unnecessarily produce different features. Possible steps include case folding, whitespace cleanup, Unicode normalization, contraction expansion and lemmatization. Choose these according to the task. ### Example: product search 1. Raw query: “ RUNNING shoes ”. Product description: “Comfortable run shoes”. 2. Trim extra whitespace and apply consistent case: “running shoes”. 3. If the application chooses lemmatization, running → run. The query and the indexed description now share the intended run and shoes features. 4. Apply the same normalization to stored documents and future queries, otherwise matching remains inconsistent. Normalization reduces avoidable vocabulary variation and can improve recall. It must preserve distinctions important to the task: lowercasing US to us may lose meaning, and removing “not” can reverse sentiment. Aggressive stemming or synonym replacement can create incorrect matches. **Yaad rakho:** Same meaning ke surface variants ko consistent banao; useful meaning delete mat karo. --- ## TEE · Q1a · BoW and one-hot encoding Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q1a Previous examination · complete worked answers **Question. Explain Bag of Words and one-hot encoding with appropriate examples.** [individual allocation not printed; Q1 totals 20] ### Answer **Bag of words (BoW) represents a document by the counts of vocabulary terms, discarding their positions.** A binary BoW variant stores only presence/absence; a count BoW stores frequencies. State which is used. $$ x_{d,j}=C(v_j,d). $$ $$ x_{d,j} $$ is document $$ d $$ 's coordinate for vocabulary term $$ v_j $$ . $$ C(v_j,d) $$ is its occurrence count. A **bag of n-grams** uses short contiguous token sequences as features. Bigrams can distinguish “not good” from “good”, but increasing n enlarges the vocabulary and creates more rare features. **Question. Build count and binary BoW for D1=“cat cat dog” and D2=“dog fish”. Use vocabulary [cat,dog,fish] and no stopword removal.** #### Given Two tokenized documents, fixed vocabulary order. #### To find Count vectors and binary vectors. #### Formula and why it applies Count BoW uses $$ C(v_j,d) $$ . Binary BoW is 1 when that count is positive and 0 otherwise. #### Step 1. D1 counts cat=2, dog=1, fish=0 → [2,1,0]. #### Step 2. D2 counts cat=0, dog=1, fish=1 → [0,1,1]. #### Step 3. Convert to presence D1 [1,1,0]; D2 [0,1,1]. **Answer.** Count rows [2,1,0], [0,1,1]; binary rows [1,1,0], [0,1,1]. **Question. After lowercasing and removing {is,a,means}, obtain BoW vectors for D1=“Continuous learning is learning lifetime”, D2=“Continuous learning means lifetime learning”, D3=“Learning is a good practice”. Use [continuous,good,learning,lifetime,practice].** #### Given Three documents and exact stopword set from the supplied slides. #### To find The document–term count matrix. #### Formula and why it applies Remove only the listed stopwords, then count every remaining token in the fixed column order. #### Step 1. Clean D1 [continuous,learning,learning,lifetime] → [1,0,2,1,0]. #### Step 2. Clean D2 [continuous,learning,lifetime,learning] → [1,0,2,1,0]. #### Step 3. Clean D3 [learning,good,practice] → [0,1,1,0,1]. #### Step 4. Compare D1 and D2 They have identical BoW vectors despite different original wording/order. **Answer.** Rows [1,0,2,1,0], [1,0,2,1,0], [0,1,1,0,1]. **Question. Build a bag of bigrams for “not good but not bad” with vocabulary [not good,good but,but not,not bad]. Do not add sentence boundary tokens.** #### Given Five tokens; use adjacent pairs, stride 1. #### To find The bigram count vector. #### Formula and why it applies A sequence of $$ L $$ tokens has $$ L-1 $$ adjacent bigrams when no boundaries are added. #### Step 1. Enumerate pairs (not,good), (good,but), (but,not), (not,bad). #### Step 2. Count each vocabulary bigram Each appears once. #### Step 3. Write vector [1,1,1,1]. **Answer.** Four bigram features, each with count 1. BoW: order bhool, word-count rakh. Bag of bigrams: neighbouring word-pair count. **Label encoding** assigns each category or word a unique integer ID. **One-hot encoding** assigns a length- $$ V $$ vector with one 1 at the word's vocabulary position and zeros elsewhere. $$ \operatorname{onehot}(w)_j=\begin{cases}1,&j=\operatorname{index}(w)\\0,&\text{otherwise.}\end{cases} $$ The index is an ID, not a meaning score. One-hot represents one token/category; a multiword document generally needs multiple token vectors or an aggregation scheme. **Question. Encode “dog cat dog” using vocabulary [cat,dog,fish], IDs starting at 0. Give label IDs and a one-hot vector for every token.** #### Given cat→0, dog→1, fish→2. Vector columns have the same order. #### To find The ID sequence and three one-hot rows. #### Formula and why it applies Replace each word by its ID for labels; set its vocabulary position to 1 for one-hot. #### Step 1. Label IDs dog→1, cat→0, dog→1 → [1,0,1]. #### Step 2. First dog [0,1,0]. #### Step 3. Cat [1,0,0]. #### Step 4. Second dog [0,1,0]. Repeated tokens keep the same representation. **Answer.** IDs [1,0,1]; one-hot rows [0,1,0], [1,0,0], [0,1,0]. **Question. Encode the city sequence Mumbai, Delhi, Mumbai, Pune using vocabulary [Delhi,Mumbai,Pune] and label IDs starting at 1.** #### Given Delhi→1, Mumbai→2, Pune→3. #### To find IDs and one-hot rows. #### Formula and why it applies One-hot uses three coordinates regardless of whether label IDs begin at 0 or 1. #### Step 1. IDs [2,1,2,3]. #### Step 2. Mumbai rows Each Mumbai → [0,1,0]. #### Step 3. Delhi and Pune Delhi→[1,0,0]; Pune→[0,0,1]. **Answer.** Rows [0,1,0], [1,0,0], [0,1,0], [0,0,1]. **Question. One-hot encode categories [red,green,blue] for a linear regression with an intercept. Then give dummy coding after dropping blue.** #### Given Full column order [red,green,blue]. Blue is the baseline. #### To find Full and reduced codes. #### Formula and why it applies Full one-hot rows sum to 1. With an intercept this creates a linear dependency; dropping one category column gives k−1 independent dummy indicators. #### Step 1. Full encoding | Category | red | green | blue | | --- | --- | --- | --- | | red | 1 | 0 | 0 | | green | 0 | 1 | 0 | | blue | 0 | 0 | 1 | #### Step 2. Drop blue column | Category | red | green | | --- | --- | --- | | red | 1 | 0 | | green | 0 | 1 | | blue | 0 | 0 | #### Step 3. Interpret zero row [0,0] now identifies the baseline blue, not missing data. **Answer.** Full width 3; reduced dummy width 2. Dropping a column is model-dependent, not a requirement for every neural embedding input. **Limitations:** one-hot is large and sparse and does not express semantic similarity. Unseen words need a defined unknown-token or vocabulary-update policy. Label IDs are compact but should not be treated as an ordered scalar feature for nominal words. Label = ID number. One-hot = us ID ke box mein 1, baaki 0. --- ## TEE · Q1b · Lesk WSD Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q1b Previous examination · complete worked answers **Question. Discuss Lesk word-sense disambiguation with a suitable example.** [individual allocation not printed; Q1 totals 20] ### Answer **Lesk is a knowledge-based WSD method that chooses senses whose dictionary descriptions have the greatest overlap with contextual evidence.** Original Lesk compares definitions of neighbouring words' senses. A common simplified version compares each target-sense gloss directly with the sentence context. State which version you use. ### Simplified Lesk procedure 1. Remove the target word from the context; tokenize and normalize the remaining content words. 2. Obtain the gloss for each candidate sense and apply the same normalization. 3. Count shared words between the context and each gloss. 4. Select the sense with maximum overlap. If scores tie, report the tie or use a specified tie-break rule. $$ \operatorname{score}(s)=|C\cap G_s|,\qquad s^*=\arg\max_s\operatorname{score}(s) $$ $$ C $$ is the set of context words, $$ G_s $$ the set of words in sense $$ s $$ 's gloss, $$ \cap $$ means common elements, $$ |\cdot| $$ means count, and $$ \arg\max $$ means “the sense giving the largest score”. This version counts unique matching words once. **Question. Use simplified Lesk to disambiguate “bank” in “The river water rose near the bank.” Use C = {river, water, rose, near}; financial gloss = {institution, money, deposit}; river gloss = {land, river, water, edge}.** #### Given The normalized context and two gloss sets are given. Count exact shared words. #### To find The score of each sense and the selected meaning. #### Formula and why it applies Use $$ \operatorname{score}(s)=|C\cap G_s| $$ because the task specifies context–gloss overlap. #### Step 1. Compare the financial sense No word occurs in both {river, water, rose, near} and {institution, money, deposit}. Overlap = {}. #### Step 2. Count the financial score $$ \operatorname{score}(F)=0 $$ . #### Step 3. Compare the river sense Shared words are {river, water}. “Rose” and “near” are not in the gloss. #### Step 4. Count and select $$ \operatorname{score}(R)=2 $$ . Since $$ 2>0 $$ , choose the river-edge sense. **Answer.** Bank means land beside a river. **Question. Use simplified Lesk for “The dog barked loudly.” Treat barked as bark. Context C = {dog, loudly}; sound gloss = {sharp, cry, dog}; tree gloss = {outer, covering, tree}; ship gloss = {sailing, ship}.** #### Given Three glosses and normalized context {dog, loudly}. #### To find Select the sense of bark. #### Formula and why it applies $$ \operatorname{score}(s)=|C\cap G_s| $$ . Lemmatization lets barked match the target lemma bark; it does not create additional overlap words. #### Step 1. Sound sense Intersection = {dog}; score = 1. #### Step 2. Tree sense Intersection = {}; score = 0. #### Step 3. Ship sense Intersection = {}; score = 0. #### Step 4. Compare The sound sense has the unique maximum: 1 > 0. **Answer.** Bark is the dog's cry. **Question. Apply definition–definition Lesk to “pine cone”. Pine senses: P1 = {evergreen, tree, needle, leaf}; P2 = {waste, sorrow, illness}. Cone senses: C1 = {solid, shape, circular}; C2 = {fruit, evergreen, tree}.** #### Given Two candidate senses for each word. Tokens are already normalized; tree and trees would be reduced to the same form if necessary. #### To find The most compatible pair of senses. #### Formula and why it applies Use $$ |G_{P_i}\cap G_{C_j}| $$ for every pair because this is original-style gloss–gloss comparison. #### Step 1. P1 with C1 No shared word → 0. #### Step 2. P1 with C2 Shared {evergreen, tree} → 2. #### Step 3. P2 with each cone sense P2–C1 has no overlap → 0; P2–C2 has no overlap → 0. #### Step 4. Choose the best pair The four scores are 0, 2, 0, 0. Maximum 2 belongs to P1–C2. **Answer.** Pine is the evergreen tree; cone is its fruit-like reproductive structure. **Limitations:** exact overlap misses synonyms, short glosses may produce zeros, and different preprocessing choices change the score. Extended Lesk may include related synset definitions or longer phrase overlaps; do not silently substitute it for a simple question. Lesk: context aur dictionary mein common words gino; sabse bada overlap jeeta. --- ## TEE · Q1c · Syntax parsing approaches Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q1c Previous examination · complete worked answers **Question. Explain syntax parsing and compare top-down and bottom-up approaches.** [individual allocation not printed; Q1 totals 20] ### Answer **A context-free grammar (CFG)** specifies a language using production rules whose left side is a single nonterminal. Formally, G=(N,Σ,R,S): N is the set of nonterminals, Σ the terminal vocabulary, R the production rules, and S the start symbol. A rule can expand its nonterminal regardless of the neighboring symbols; that is the meaning of “context-free”. | Concept | Plain explanation | Example | | --- | --- | --- | | Terminal | A word at a leaf; it is not expanded further | the, girl, read | | Nonterminal | A grammatical category that can be expanded | S, NP, VP, Det, N, V | | Production | A permitted replacement | S → NP VP | | Start symbol | Category for a complete sentence | S | | Lexicon | Word-to-category rules | N → girl | paper | | Derivation | Sequence of rule applications | S ⇒ NP VP ⇒ Det N VP ⇒ … | | Parse tree | Hierarchical picture of those expansions | Root S, internal phrases, word leaves | $$ S\to NP\ VP,\quad NP\to Det\ N,\quad VP\to V\ NP $$ Read S→NP VP as “a sentence can contain a noun phrase followed by a verb phrase”. It does not mean subtraction or multiplication. A vertical bar in a grammar rule means alternatives: Det→the | a allows either the or a. This is different from the probability bar meaning “given”. S: the whole sentence → NP: the girl + VP: read the paper → NP→Det N; VP→V NP → Leaves in order: the / girl / read / the / paper **A constituent** is a group of words that acts as a grammatical unit, such as “the girl”. **Syntactic parsing** assigns a grammatical structure to an input sentence using a grammar. It can support grammar checking, information extraction, translation and later semantic analysis. Failure under one restricted grammar means that grammar could not derive the sentence; it does not necessarily prove the sentence is invalid English. **A treebank** is a corpus annotated with parse trees. It supplies training/evaluation structures for parsers and lets linguists study recurring syntactic patterns. A tagged corpus gives word tags; a treebank additionally records relationships among phrases. | Representation | Example information retained | | --- | --- | | Bag of words | Words occur, but word order/grammar is lost | | POS tags | Each word's local grammatical category | | Chunks | Short phrase boundaries | | Parse tree | Nested constituent structure of the whole sentence | **Yaad rakho:** Grammar rule ek expansion instruction hai: S ko NP aur VP mein kholo. **Top-down parsing** starts at the grammar's start symbol S and expands nonterminals until the leaves match the input words in order. **Recursive descent** commonly implements each nonterminal as a procedure that tries its productions. It builds from the root toward the leaves. Start with S → Choose an allowed S production → Expand its next nonterminal → Match the resulting terminal to the input → Continue, or backtrack if the choice fails **Algorithm.** Start with S as the current goal. Choose a production for the leftmost unexpanded nonterminal. Replace that nonterminal by the production's right side. If a terminal appears, compare it with the next input token. If it matches, consume the token; if it does not, reject that branch and try another production. Accept only when the complete input is consumed and all grammar goals are satisfied. A **leftmost derivation** always expands the leftmost remaining nonterminal first. Top-down parsing can make guesses that the actual words cannot support, so it may need backtracking. A naive recursive-descent parser can loop on left-recursive rules such as NP→NP PP unless the grammar or algorithm is changed. Predictive LL(1) parsing chooses productions using one lookahead token when the grammar permits; not every English CFG is LL(1). | Approach | Begins with | Direction | Common difficulty | | --- | --- | --- | --- | | Top-down | Start symbol S | Root to word leaves | Predictions that do not match input; left recursion | | Bottom-up | Actual input tokens | Leaves to root S | Reductions that cannot contribute to a complete sentence | **Yaad rakho:** Top-down: sentence ka structure pehle, words tak neeche aao. **Bottom-up parsing** starts from the input tokens and combines them into larger constituents until the start symbol is reached. A **shift-reduce parser** maintains a stack of recognized material and a list of remaining input tokens. | Action | Exact operation | Example | | --- | --- | --- | | Shift | Move the next input token onto the stack | Stack: the; remaining input: dog runs | | Reduce | Replace a suffix of the stack matching a rule's RHS with its LHS | Det N becomes NP using NP→Det N | | Accept | Stack is S and input is exhausted | Stack: S; input: empty | | Fail/backtrack | Chosen reductions cannot complete a parse | A noun reading of book blocks an imperative parse | Reduction uses a production in reverse: the grammar NP→Det N expands an NP top-down, but reduces Det N into NP bottom-up. The stack's **rightmost item is its top** in the traces below. Each row shows the state **after** the listed action, avoiding an off-by-one mismatch between actions and stack contents. A lexical reduction such as N→dog still counts as a reduction. Do not skip the sequence from word to category to phrase when the question asks for a complete trace. Backtracking means returning to an earlier choice and trying another applicable rule, not changing the original input. **Yaad rakho:** Shift = input se stack. Reduce = chhote pieces se bada grammatical unit. --- ## TEE · Q1d · NLU versus NLG and applications Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q1d Previous examination · complete worked answers **Question. Compare Natural Language Understanding and Natural Language Generation. List real-world NLP applications that use these concepts.** [individual allocation not printed; Q1 totals 20] ### Answer **Natural language understanding (NLU)** maps language into an interpretation useful for a task: intents, entities, semantic relations or another meaning representation. **Natural language generation (NLG)** produces human-language output from data, a representation or another source. “Book a ticket to Pune tomorrow.” → NLU: intent=book; destination=Pune; date=tomorrow → Application: search available journeys → NLG: “Here are the available journeys to Pune.” | Feature | NLU | NLG | | --- | --- | --- | | Direction | Language → interpretation | Data/representation → language | | Main concern | Meaning, intent, ambiguity | Content choice, organization, fluent wording | | Example | Find destination in a request | Write the booking confirmation | | Common failure | Wrong destination or sense | Unclear, unsupported or ungrammatical output | ### NLG stages in a classical pipeline 1. **Content determination:** choose facts to communicate. 2. **Data interpretation:** identify relationships or trends. 3. **Document planning:** organize facts in a useful order. 4. **Sentence aggregation:** combine related facts without repetition. 5. **Lexical and grammatical realization:** choose words and construct grammatical sentences. 6. **Presentation:** format the generated message for its audience. For a weather report, facts {city=Pune, temperature=28°C, rain=yes} can become “Pune will have rain with a temperature of 28°C.” NLU would instead extract facts or intent from an incoming language message. NLU = language se meaning andar. NLG = data se language bahar. A question-answering assistant interprets intent/entities through NLU and composes an answer through NLG. Translation interprets source meaning and generates a target-language sequence. A customer-support system classifies requests and generates a contextual reply. --- ## TEE · Q2a · TF–IDF cosine after a fixed stoplist Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q2a Previous examination · complete worked answers **Question. For “The cat sat on the mat.”, “The dog barked at the cat.” and “I love my dog and my cat.” remove stopwords {I,the,on,my,at,and}; construct TF–IDF and calculate cosine similarity between the first two documents.** [10 marks] ### Answer ### Given | Document | Original text | | --- | --- | | 1 | The cat sat on the mat. | | 2 | The dog barked at the cat. | | 3 | I love my dog and my cat. | Stopwords: at, my, i, on, the, and ### To find BoW, IDF and TF–IDF plus requested cosine ### Formula: which one and why Use lowercase alphabetic tokens and exactly the stated stoplist. If no stoplist is supplied, retain function words. Adopt raw-count TF and natural-log IDF; the paper does not prescribe another TF/log convention. $$ \mathrm{TFIDF}(w,d)=c(w,d)\ln\frac{N}{\mathrm{df}(w)} $$ A term shared by all N documents has IDF ln(1)=0. Counts and TF–IDF weights are different representations. ### Step 1. Tokenize each document | Document | Tokens | | --- | --- | | 1 | cat, sat, mat | | 2 | dog, barked, cat | | 3 | love, dog, cat | ### Step 2. Fix the vocabulary order barked, cat, dog, love, mat, sat ### Step 3. Count occurrences (Bag of Words) | Document | barked | cat | dog | love | mat | sat | | --- | --- | --- | --- | --- | --- | --- | | 1 | 0 | 1 | 0 | 0 | 1 | 1 | | 2 | 1 | 1 | 1 | 0 | 0 | 0 | | 3 | 0 | 1 | 1 | 1 | 0 | 0 | ### Step 4. Calculate document frequency and IDF | Term | df | N / df | ln(N/df) | | --- | --- | --- | --- | | barked | 1 | 3 | 1.098612289 | | cat | 3 | 1 | 0.000000000 | | dog | 2 | 3/2 | 0.405465108 | | love | 1 | 3 | 1.098612289 | | mat | 1 | 3 | 1.098612289 | | sat | 1 | 3 | 1.098612289 | ### Step 5. Multiply each TF by its IDF | Document | barked | cat | dog | love | mat | sat | | --- | --- | --- | --- | --- | --- | --- | | 1 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 1.098612 | | 2 | 1.098612 | 0.000000 | 0.405465 | 0.000000 | 0.000000 | 0.000000 | | 3 | 0.000000 | 0.000000 | 0.405465 | 1.098612 | 0.000000 | 0.000000 | ### Step 6. Calculate cosine from these same TF–IDF rows $$ \cos(u,v)=\frac{\sum_i u_i v_i}{\sqrt{\sum_i u_i^2}\sqrt{\sum_i v_i^2}} $$ Coordinate products: 0.000000000, 0.000000000, 0.000000000, 0.000000000, 0.000000000, 0.000000000 Dot product=0.000000000; squared norms=2.413897922, 1.371350915. Cosine=0.000000000 ### Final answer The final table is the TF–IDF matrix in the declared vocabulary order. A different explicitly stated log base rescales all IDFs; length-normalized TF changes each document scale. Cosine is 0.0. Shared words whose IDF is zero contribute nothing to the TF–IDF dot product. ### Which terms are prominent? D1: sat and mat have IDF ln3; D2: barked has ln3 and dog has ln(3/2); D3: love has ln3 and dog has ln(3/2). Cat occurs in all three documents and has zero unsmoothed IDF. Thus shared cat does not make D1/D2 TF–IDF cosine positive. --- ## TEE · Q2b · Weather Naive Bayes Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q2b Previous examination · complete worked answers **Question. Use the supplied class/conditional probabilities to predict Play Tennis for Outlook=Sunny, Temperature=Cool, Humidity=High, Windy=True.** [10 marks] ### Answer ### Given | Class | Prior | Sunny | Cool | High | True | | --- | --- | --- | --- | --- | --- | | Yes | 9/14 | 2/9 | 3/9 | 3/9 | 3/9 | | No | 5/14 | 3/5 | 1/5 | 4/5 | 3/5 | ### To find Compare the Yes and No class scores. ### Formula: which one and why $$ \mathrm{score}(c)=P(c)P(Sunny\mid c)P(Cool\mid c)P(High\mid c)P(True\mid c) $$ ### Step 1. Multiply the Yes factors 9/14 × 2/9 × 3/9 × 3/9 × 3/9 = 1/189 = 0.005291005 ### Step 2. Multiply the No factors 5/14 × 3/5 × 1/5 × 4/5 × 3/5 = 18/875 = 0.020571429 ### Step 3. Compare and normalize Yes=0.204582651, No=0.795417349 ### Final answer **Predict No.** Apply the supplied categorical probabilities directly; adding vocabulary smoothing would solve a different problem. --- ## TEE · Q3a · Coreference and anaphora Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q3a Previous examination · complete worked answers **Question. Explain coreference and anaphoric reference with examples.** [5 marks] ### Answer **Pragmatic analysis** interprets an utterance using its situation and the speaker’s intended meaning. **Discourse analysis** studies how utterances connect across a text or conversation. Syntax and literal word meanings alone cannot settle every interpretation. | Text | Literal content | Contextual interpretation | | --- | --- | --- | | “Can you close the window?” | A question about ability. | Usually a polite request. | | “Meera bought a laptop. It was expensive.” | Two sentences. | “It” connects the second sentence to the laptop. | | “Meet me here tomorrow.” | An instruction. | Me, here and tomorrow depend on speaker, place and date. | ### Reference, mention and antecedent A **referring expression** points to an entity; a **mention** is its occurrence in the text. Its **referent** is the entity being discussed. An **antecedent** is a preceding expression used to interpret a later referring expression. Reference resolution identifies the intended referent. Coreference resolution groups mentions that refer to the same entity. “Meera”name mention → Person entitysame referent → “she” / “the engineer”other mentions | Concept | Definition | Example | | --- | --- | --- | | Anaphora | Reference back to something introduced earlier. | “Riya arrived. She sat down.” She → Riya. | | Cataphora | Reference forward to a later expression. | “Before she spoke, Riya checked the microphone.” she → Riya. | | Exophora | Reference to the situation outside the text. | “Put it there”, said while pointing. | | Coreference | Two or more mentions refer to the same entity. | “Dr Rao”, “the lecturer”, “she”. | | Pronoun resolution | Find the referent of a pronoun. | Resolve “it” in “The laptop stopped because it overheated.” | Coreference is broader than pronoun resolution: repeated names and descriptions also form a chain. Anaphora describes direction/dependence; coreference describes identity. In “I bought a car. The engine is noisy”, engine is related to the car but is not the same entity. **Yaad rakho:** Reference = kiski baat? Coreference = alag mentions, same entity. Pronoun resolution = he/she/it kis se linked? ### Cohesion, coherence and discourse markers **Cohesion** is the explicit linguistic linking of a text through reference, repeated or related words, conjunctions and other devices. In “Rita bought a book. She read it”, she and it connect the second sentence to Rita and the book. **Coherence** is the conceptual unity that makes a discourse meaningful. “The road flooded, so the match was cancelled” presents a plausible cause and consequence. A reader relates the propositions using context and knowledge. **Discourse markers** signal relationships or organize speech: however signals contrast; therefore signals a result; first and next signal sequence; for example introduces an illustration. These concepts differ: “I like apples. Therefore the moon is square” has a linking word but no sensible inference. A connector alone does not guarantee coherence. **Yaad rakho:** Cohesion = words se link; coherence = meaning se link; marker = relation ka signal. ### A practical resolution procedure First detect the pronoun and candidate noun phrases. Eliminate candidates inconsistent with person, number, grammatical binding or relevant semantic constraints. Rank the surviving candidates using syntactic prominence, recency and discourse context. Preserve ambiguity if the available evidence does not decide. | Approach | How it works | Strength | Limitation | | --- | --- | --- | --- | | Rule based / syntactic | Use parse trees, agreement and explicit search order; Hobbs is a classic example. | Can explain each decision without labelled training data. | Parsing errors and world knowledge gaps cause errors. | | Salience based | Score candidates using recency, subjecthood and other discourse features; select the strongest compatible candidate. | Captures the tendency to refer again to prominent entities. | Feature weights and competing candidates can mislead. | | Supervised statistical | Learn candidate/pronoun compatibility from labelled examples. | Combines many features from data. | Needs representative annotated examples. | | Neural / contextual | Encode the surrounding text and score mention pairs or antecedents. | Can learn contextual patterns and long-range dependencies. | Needs data/compute and can still make confident mistakes. | ### Worked analysis 1 · number and semantics Resolve “they” in: “Riya put the books on the shelf because they were heavy.” **Given** Candidates: Riya (singular person), books (plural objects), shelf (singular object); pronoun they is plural here. **To find** The antecedent and a justified resolution procedure. ### Formula and why it applies Use number agreement first, then semantic compatibility. There is no multiplication formula in this linguistic task. ### Solution #### Step 1. List candidates Riya, the books, the shelf. #### Step 2. Apply number agreement Riya and shelf are singular. Books is plural, matching they. #### Step 3. Check meaning Books can be heavy; the causal explanation fits carrying/placing books. “They” refers to “the books”. ### Worked analysis 2 · a complete coreference chain Identify the coreference chain in: “Dr Sen entered. The surgeon greeted Ravi. She opened her notebook.” **Given** Mentions: Dr Sen, the surgeon, Ravi, she, her, her notebook. Assume the surgeon is Dr Sen and she/her refers to that surgeon. **To find** The person chain, and a mention that must not be placed in it. ### Formula and why it applies Group expressions by identity of referent; a possessive expression and the owned object are different entities. ### Solution #### Step 1. Link the name and description Dr Sen and the surgeon denote the same person under the given context. #### Step 2. Link the pronouns She and her refer to the surgeon. #### Step 3. Keep the notebook separate “Her notebook” is an object owned by the person; it is not the person herself. Person chain: {Dr Sen, the surgeon, She, her}. “Her notebook” is a separate object mention. ### Worked analysis 3 · honest ambiguity Discuss the pronoun in: “Asha called Nisha after she finished work.” **Given** Both named candidates are singular; the sentence gives no further context. **To find** Whether one referent is logically guaranteed. ### Formula and why it applies Agreement filters incompatible candidates but cannot always select between equally compatible candidates. ### Solution #### Step 1. Check agreement Both Asha and Nisha are compatible with she. #### Step 2. Check the event Either person could have finished work. #### Step 3. State the limitation A heuristic may prefer a candidate, but grammar and this sentence alone do not prove the intended referent. Ambiguous without further context. State any heuristic and parse assumption before giving its preferred candidate. --- ## TEE · Q3b · Cohesion, coherence and discourse markers Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q3b Previous examination · complete worked answers **Question. Explain cohesion, coherence and discourse markers.** [5 marks] ### Answer **Pragmatic analysis** interprets an utterance using its situation and the speaker’s intended meaning. **Discourse analysis** studies how utterances connect across a text or conversation. Syntax and literal word meanings alone cannot settle every interpretation. | Text | Literal content | Contextual interpretation | | --- | --- | --- | | “Can you close the window?” | A question about ability. | Usually a polite request. | | “Meera bought a laptop. It was expensive.” | Two sentences. | “It” connects the second sentence to the laptop. | | “Meet me here tomorrow.” | An instruction. | Me, here and tomorrow depend on speaker, place and date. | ### Reference, mention and antecedent A **referring expression** points to an entity; a **mention** is its occurrence in the text. Its **referent** is the entity being discussed. An **antecedent** is a preceding expression used to interpret a later referring expression. Reference resolution identifies the intended referent. Coreference resolution groups mentions that refer to the same entity. “Meera”name mention → Person entitysame referent → “she” / “the engineer”other mentions | Concept | Definition | Example | | --- | --- | --- | | Anaphora | Reference back to something introduced earlier. | “Riya arrived. She sat down.” She → Riya. | | Cataphora | Reference forward to a later expression. | “Before she spoke, Riya checked the microphone.” she → Riya. | | Exophora | Reference to the situation outside the text. | “Put it there”, said while pointing. | | Coreference | Two or more mentions refer to the same entity. | “Dr Rao”, “the lecturer”, “she”. | | Pronoun resolution | Find the referent of a pronoun. | Resolve “it” in “The laptop stopped because it overheated.” | Coreference is broader than pronoun resolution: repeated names and descriptions also form a chain. Anaphora describes direction/dependence; coreference describes identity. In “I bought a car. The engine is noisy”, engine is related to the car but is not the same entity. **Yaad rakho:** Reference = kiski baat? Coreference = alag mentions, same entity. Pronoun resolution = he/she/it kis se linked? ### Cohesion, coherence and discourse markers **Cohesion** is the explicit linguistic linking of a text through reference, repeated or related words, conjunctions and other devices. In “Rita bought a book. She read it”, she and it connect the second sentence to Rita and the book. **Coherence** is the conceptual unity that makes a discourse meaningful. “The road flooded, so the match was cancelled” presents a plausible cause and consequence. A reader relates the propositions using context and knowledge. **Discourse markers** signal relationships or organize speech: however signals contrast; therefore signals a result; first and next signal sequence; for example introduces an illustration. These concepts differ: “I like apples. Therefore the moon is square” has a linking word but no sensible inference. A connector alone does not guarantee coherence. **Yaad rakho:** Cohesion = words se link; coherence = meaning se link; marker = relation ka signal. ### Definitions and examples requested in this part | Term | Definition | Example | | --- | --- | --- | | Cohesion | Explicit language links connecting text: pronouns, repetition, conjunctions, lexical ties | Rita bought a book. She read it. She/it refer back to Rita/book. | | Coherence | Conceptual unity and plausible relations making the text meaningful | The road flooded, so the match was cancelled: cause and consequence connect the propositions. | | Discourse marker | Expression signalling relation or organizing the discourse | however=contrast; therefore=result; first/next=sequence; for example=illustration | “I like apples. Therefore the moon is square” contains a connector but lacks a sensible inference: surface cohesion does not guarantee coherence. --- ## TEE · Q3c · Question answering system Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q3c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q3c Previous examination · complete worked answers **Question. Explain a question-answering system with its architecture and an example.** [10 marks] ### Answer **Question answering (QA) returns an answer to a natural-language question using available evidence or knowledge.** Information retrieval usually returns ranked documents; QA must identify the requested information and deliver the answer. A relevant document is useful evidence but is not automatically the answer. ### Question types and answer types | Question | Expected answer type | Required operation | | --- | --- | --- | | Who founded the company? | Person or organisation | Find the founder relation | | When did the museum open? | Date/time | Locate the opening date | | Where is the campus? | Location | Locate the campus-location relation | | How many books were issued? | Quantity | Read or compute the stated count | | Why was the train cancelled? | Reason/explanation | Find the causal evidence | | Is the office open on Sunday? | Yes/no with evidence | Check the relevant schedule statement | **Closed-domain QA** operates within a restricted area such as a university handbook. **Open-domain QA** addresses a wider range of subjects and usually needs broad retrieval. **Extractive QA** selects a span from a passage. **Generative QA** constructs answer text; it still must preserve the evidence and avoid unsupported details. Question analysis → Evidence retrieval → Passage/record understanding → Answer selection or composition → Support check → Present answer ### Explain each module | Module | What happens | Why it matters | | --- | --- | --- | | Question analysis | Identify topic, intent, constraints and answer type. | When asks for a time, not a place. | | Retrieval | Find a relevant passage or structured fact. | Reduces the search space. | | Evidence understanding | Parse relations, resolve references and ambiguous words. | The correct date must belong to the correct event. | | Answer formation | Extract a span or compose a short supported response. | Turns a document into an actual answer. | | Verification | Check whether every claimed detail follows from evidence. | Prevents filling gaps by guessing. | | Presentation | Return a concise answer with suitable context. | Avoids dumping unrelated text. | ### Worked application 1 · extract the correct fact **Question. Given “The City Museum opened in 1998 in Pune. It was renovated in 2016”, answer “When did the museum open?” Show the QA stages.** #### Given A two-sentence passage. Dates 1998 and 2016 refer to different events. #### To find The opening date, not the renovation date. #### Method and why it applies Use extractive QA: identify the requested event and expected date type, then choose the span linked to that event. ### Solution #### Step 1. Analyse the question When requires a date/time answer. The event is opening; the entity is the City Museum. #### Step 2. Retrieve the relevant sentence The first sentence mentions the museum and its opening. The second sentence discusses renovation. #### Step 3. Identify event–date relations | Event | Date | | --- | --- | | opened | 1998 | | renovated | 2016 | #### Step 4. Select the answer span 1998 is attached to opened, so select the four-character year span 1998. #### Step 5. Reject the distractor 2016 is also a date, but it answers a different question: when was the museum renovated? #### Step 6. Present the answer “The museum opened in 1998.” **Answer.** 1998. Matching only the answer type date would be insufficient without matching the event. ### Worked application 2 · resolve a pronoun before answering **Question. Given “Mira joined Orion Labs in 2022. She became its research lead in 2024”, answer “Who became Orion Labs' research lead, and when?”** #### Given The passage contains the name Mira, pronoun She, organisation Orion Labs and possessive its. Assume the normal reading that She refers to Mira and its to Orion Labs. #### To find The person and the date of becoming research lead. #### Method and why it applies Combine reference resolution with relation extraction. The question asks for two fields, both tied to the promotion/role event. ### Solution #### Step 1. Identify requested fields Who → person; when → date. The relation is became research lead of Orion Labs. #### Step 2. Resolve mentions | Mention | Referent | | --- | --- | | She | Mira | | its | Orion Labs | #### Step 3. Read the target event After resolution, the second sentence states that Mira became Orion Labs' research lead in 2024. #### Step 4. Distinguish the dates 2022 belongs to joining the organisation. 2024 belongs to becoming research lead. #### Step 5. Compose a supported answer “Mira became Orion Labs' research lead in 2024.” #### Step 6. Verify completeness The answer supplies both requested fields and does not add an unsupported department, qualification or reason. **Answer.** Mira, in 2024. Reference resolution connects the second sentence to the named person and organisation. ### Worked application 3 · detect insufficient evidence **Question. A handbook excerpt states “The library opens at 8 a.m. on weekdays.” A user asks “At what time does it close on Sunday?” Explain the correct QA response.** #### Given Only the opening time for weekdays is supplied. No closing time or Sunday schedule is present. #### To find Whether an answer is supported and how to handle the missing information. #### Method and why it applies Compare every question constraint against the evidence. Topic overlap with library is not sufficient: time type and day must also match. ### Solution #### Step 1. Extract the question constraints | Constraint | Requested | | --- | --- | | Entity | Library | | Event | Closing | | Day | Sunday | | Answer type | Time | #### Step 2. Extract the supplied fact Entity=library; event=opening; day=weekdays; time=8 a.m. #### Step 3. Identify mismatches Opening is not closing. Weekdays do not supply a Sunday schedule. #### Step 4. Avoid the unsupported substitution Do not answer 8 a.m. just because it is the only time in the passage. #### Step 5. Return an evidence-limited response “The excerpt does not state the library's Sunday closing time.” A system with further retrieval can search a relevant schedule. #### Step 6. State the benefit The system avoids converting a relevant-looking number into an incorrect answer. **Answer.** No Sunday closing time is supported by the excerpt. Report insufficient evidence or retrieve the missing schedule. ### Advantages, limits and evaluation | Design choice | Benefit | Limitation | | --- | --- | --- | | Extractive answer | The returned wording comes directly from evidence. | The answer may require combining sentences or may not be present as one span. | | Generative answer | Can combine supported facts into a readable response. | Can introduce unsupported information unless carefully checked. | | Closed-domain collection | Controlled terminology and relevant resources. | Cannot answer facts outside its knowledge source. | | Broad retrieval | Wider question coverage. | Irrelevant, outdated or conflicting sources can be retrieved. | Evaluate **exact match** when a canonical answer string is expected, **token overlap/F1** when wording may vary, and **factual correctness and completeness** when answers are composed. An answer can be grammatically fluent but wrong. Questions with no answer in the evidence should be included in evaluation so the system's restraint is also tested. For a longer exam answer, define QA, distinguish it from retrieval, explain answer types and system variants, draw the pipeline, work through a passage including distractors, and discuss ambiguity, missing evidence and evaluation. A list of chatbot names is not a QA design. **Yaad rakho:** Question ka entity + event + constraint match karo. Passage mein koi bhi number mil gaya matlab answer nahi. --- ## TEE · Q4a · NLP pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q4a Previous examination · complete worked answers **Question. Explain the stages of an NLP pipeline.** [5 marks] ### Answer **An NLP pipeline is an organized sequence of operations that transforms raw language into a task output and checks whether that output is useful.** Not every task needs every cleaning operation. Data acquisition → Preprocessing → Feature representation → Model → Evaluation → Deployment and monitoring 1. **Data acquisition:** collect relevant text and, when needed, labels. 2. **Cleaning and preprocessing:** handle markup, tokenize, normalize and decide which information to retain. 3. **Feature engineering:** represent text numerically using BoW, TF–IDF or embeddings. 4. **Modelling:** apply rules or train a statistical/neural model. 5. **Evaluation:** test on held-out data using task-appropriate measures. 6. **Deployment:** make the system available, log suitable feedback and monitor errors or data changes. ### Concrete spam-filter example Collect labelled emails → separate train/test data → tokenize text while preserving meaningful terms → fit vocabulary and IDF on training data → train classifier → evaluate spam detection on the held-out set → deploy an email service filter → monitor legitimate emails wrongly blocked. **Intrinsic evaluation** measures model/task quality, such as precision, recall or tagging accuracy. **Extrinsic evaluation** measures effect in the wider application, such as whether users find answers faster. A higher internal metric does not automatically guarantee a better user outcome. **Data leakage:** learning vocabulary statistics, labels or preprocessing decisions from the test set can make reported performance too optimistic. Fit learned transformations on training data, then apply them to validation/test data. Collect → clean → numbers → model → test → use. Test data ko training ki information mat banao. **Definition.** NLP is the study of computational methods for analyzing, understanding and generating human language. The pipeline diagram above turns raw language into a task-specific prediction or generated response. --- ## TEE · Q4b · Morphology and syntax Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q4b Previous examination · complete worked answers **Question. Explain the importance of morphology and syntax in NLP.** [5 marks] ### Answer **A phoneme** is a sound unit that can distinguish meanings in a language, such as the /p/ and /b/ contrast in “pat” and “bat”. A phoneme is a sound, not necessarily one alphabet letter. **A morpheme** is the smallest meaning-bearing grammatical unit: cats = cat + plural -s; unhappy = un- + happy. **A lexeme** is an abstract vocabulary item grouping its inflected forms: write, writes, wrote and writing belong to WRITE. A **phrase** is a group of words functioning together; a **clause** normally contains a predicate and its associated participants; a **sentence** expresses a complete grammatical unit. | Level | What it analyses | Concrete example | | --- | --- | --- | | Phonetics / phonology | Physical sounds / sound patterns in a language | Distinguish bat from pat | | Morphology / lexical analysis | Word structure and lexical items | un + help + ful → unhelpful | | Syntax | Grammatical arrangement and relationships | The dog chased the cat: subject, verb, object | | Semantics | Meaning of words and composed expressions | The dog is the chaser; the cat is chased | | Discourse | Connections across sentences | “Riya arrived. She smiled.” → She refers to Riya | | Pragmatics | Intended meaning in the situation | “It is cold here” may request closing the window | word forms → grammatical structure → sentence meaning → cross-sentence context → speaker intention ### Three worked interpretations **“The cats are sleeping.”** Morphology identifies cat + s as plural. Syntax connects “cats” to “are sleeping”. Semantics represents animals engaged in sleeping. **“Aman bought a laptop. It was expensive.”** Sentence-level syntax alone does not identify “it”. Discourse connects “it” to the laptop across the sentence boundary. **“Wonderful, another power cut!”** The word wonderful is positive literally, but situational context can indicate sarcasm and frustration. This is a pragmatic interpretation. The analysis levels are conceptual descriptions. Modern systems may learn several jointly rather than executing a compulsory separate module for every level. Morphology = word ke andar; syntax = sentence ka structure; semantics = meaning; discourse = sentences ka link; pragmatics = intention. Morphology analyzes internal word structure: unreadable→un+read+able, cats→cat+plural. Syntax analyzes how words combine: subject–verb–object relations distinguish “dog bites man” from “man bites dog”. Morphological normalization supports vocabulary matching; syntax preserves grammatical relations needed for parsing, extraction and translation. Translation needs morphological number/tense agreement and target-language word order. Sentiment needs syntax to distinguish “not good” from “good” and morphology to relate disappointed/disappointing to their base form without discarding the negative meaning. --- ## TEE · Q4c · Most probable bigram continuations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q4c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q4c Previous examination · complete worked answers **Question. Train on I love programming ; Programming is fun ; I love coding ; Coding is great . Predict next tokens for I love ..., programming ..., coding ... using a bigram model.** [10 marks] ### Answer ### Given I love programming Programming is fun I love coding Coding is great ### To find Most probable next token for each prefix. ### Formula: which one and why $$ P(w\mid h)=\frac{c(h,w)}{\sum_v c(h,v)} $$ Lowercase lexical tokens; include sentence-end as an observed next token. Bigram prediction uses only the last token of the prefix. ### Step 1. Count each adjacent pair separately within sentences | Bigram | Count | | --- | --- | | → coding | 1 | | → i | 2 | | → programming | 1 | | coding → | 1 | | coding → is | 1 | | fun → | 1 | | great → | 1 | | i → love | 2 | | is → fun | 1 | | is → great | 1 | | love → coding | 1 | | love → programming | 1 | | programming → | 1 | | programming → is | 1 | ### Step 2. Continue I love | Candidate | Count / history count | Probability | | --- | --- | --- | | coding | 1/2 | 0.500000 | | programming | 1/2 | 0.500000 | Highest: programming, coding. This is a tie; no tie-break rule is supplied. ### Step 3. Continue programming | Candidate | Count / history count | Probability | | --- | --- | --- | | | 1/2 | 0.500000 | | is | 1/2 | 0.500000 | Highest: , is. This is a tie; no tie-break rule is supplied. ### Step 4. Continue coding | Candidate | Count / history count | Probability | | --- | --- | --- | | | 1/2 | 0.500000 | | is | 1/2 | 0.500000 | Highest: , is. This is a tie; no tie-break rule is supplied. ### Final answer I love → programming or coding programming → or is coding → or is If an examiner explicitly restricts the answer to a lexical word, exclude END and state that restricted candidate set. Do not silently remove an END tie. --- ## TEE · Q5a · Preprocess the NLP passage Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q5a Previous examination · complete worked answers **Question. Perform the requested text preprocessing on: “Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through natural language. The ultimate objective of NLP is to read, decipher, understand, and make sense of human languages in a valuable way.” Remove the supplied stopwords {is,a,of,that,on,I,the,and,to,in}.** [6 marks] ### Answer | Stage | Result | | --- | --- | | Sentence segmentation | Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and humans through natural language. The ultimate objective of NLP is to read, decipher, understand, and make sense of human languages in a valuable way. | | Word tokens (punctuation omitted) | Natural, Language, Processing, NLP, is, a, field, of, artificial, intelligence, that, focuses, on, the, interaction, between, computers, and, humans, through, natural, language, The, ultimate, objective, of, NLP, is, to, read, decipher, understand, and, make, sense, of, human, languages, in, a, valuable, way | | Lowercase | natural, language, processing, nlp, is, a, field, of, artificial, intelligence, that, focuses, on, the, interaction, between, computers, and, humans, through, natural, language, the, ultimate, objective, of, nlp, is, to, read, decipher, understand, and, make, sense, of, human, languages, in, a, valuable, way | | Declared stoplist | a, and, i, in, is, of, on, that, the, to | | After exact stop removal | natural, language, processing, nlp, field, artificial, intelligence, focuses, interaction, between, computers, humans, through, natural, language, ultimate, objective, nlp, read, decipher, understand, make, sense, human, languages, valuable, way | | Operation | Why it is useful | | --- | --- | | Sentence segmentation | Defines separate units for parsing and prevents counts from crossing sentence boundaries. | | Lowercasing | Merges case variants when capitalization is not needed; retain original case for NER when useful. | | Stop removal | Can remove frequent low-information words for a bag-of-words retrieval task; it is task-dependent and can damage negation/grammar. | | Tokenization | Defines the units whose counts/features are computed; punctuation and abbreviation conventions must be consistent. | --- ## TEE · Q5b · Stemming and lemmatization tools Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q5b Previous examination · complete worked answers **Question. Differentiate stemming and lemmatization, naming two approaches/tools for each.** [4 marks] ### Answer **Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma. | Criterion | Stemming | Lemmatization | | --- | --- | --- | | Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis | | Output | May be nonword, e.g. studi | Dictionary form, e.g. study | | Context/POS | Usually little or none | Often important | | Irregular forms | May miss went→go | Can map went→go with verb analysis | | Cost | Generally simpler | Usually more linguistic processing | Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation. ### Three worked reductions 1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked. 2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation. 3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter. **Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice. Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan. Examples of stemmers: Porter and Lancaster (Snowball is another). Examples of lemmatization tools: WordNet lemmatizer with POS information and spaCy’s language-specific lemmatizer. Outputs depend on language and supplied POS; better→good requires lexical knowledge, unlike simple suffix cutting. --- ## TEE · Q5c · Hobbs on John and Mary Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q5c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q5c Previous examination · complete worked answers **Question. Explain Hobbs’ algorithm and resolve the pronouns in “John gave a book to Mary after he finished reading it.”** [10 marks] ### Answer **Hobbs’ algorithm is a syntax-based procedure that searches constituent parse trees in a fixed order for a compatible noun-phrase antecedent of a pronoun.** The tree determines which candidates are proposed first. Agreement and other relevant constraints determine whether a proposed candidate can be accepted. It is not simply the rule “choose the closest noun”. A nearby noun may have the wrong number, may be syntactically ineligible, or may be visited after a structurally more prominent noun. The algorithm produces a preference under the supplied parse; ambiguous language can still require context. ### The tree vocabulary you need | Term | Meaning | Example | | --- | --- | --- | | NP | Noun phrase: a phrase referring to an entity | the student; Anita; they | | S | A sentence/clause node | the node containing a subject and predicate | | VP | Verb phrase | told Ravi that she should revise | | X | The NP or S node currently being searched | An algorithm label, not a word | | Path p | The branch travelled upward from the pronoun/previous X | Used to distinguish left and right subtrees | | N-bar (N′) | The noun-centred part of an NP in a grammar using that label | The nominal-child test depends on the given tree | | Breadth-first | Visit shallower nodes before deeper ones | Level 1 before level 2 | | Left-to-right | Visit equally deep nodes in their written/tree order | Earlier left branch before right branch | Locate pronoun NP → Climb to NP/S → Propose candidates in tree order → Check compatibility → Accept or continue searching ### The complete search procedure #### Step 1. Start at the pronoun Locate the NP immediately dominating the pronoun. “Dominating” means containing it as a descendant in the tree. #### Step 2. Climb to the first NP or S Move upward to the first NP or S above that starting NP. Call this node X. Keep the upward path p. #### Step 3. Search the left branches initially Below X, traverse branches to the left of p breadth-first and left-to-right. At this initial search, propose an NP only if another NP or S lies between that candidate and X. Test proposed candidates for the required agreement and constraints. #### Step 4. If X is the highest sentence node, inspect earlier sentences Search previous sentence trees from most recent to oldest. Within each, use breadth-first, left-to-right order to propose NPs. Accept the first eligible compatible candidate, or report that no candidate was found in the available context. #### Step 5. Otherwise climb again Move from X to the next NP or S above it. Record this new X and the path followed during the climb. #### Step 6. Apply the higher-NP test If the new X is an NP, propose X when the path did not pass through its nominal N-bar child. Use the given parse labels; do not invent an N-bar branch missing from the question's structure. #### Step 7. Search its left branches Traverse branches below the new X to the left of the path, breadth-first and left-to-right, proposing NPs. The special intervening-NP/S restriction belongs to the initial search in Step 3. #### Step 8. If X is S, search the permitted right branches Search branches to the right of the path breadth-first and left-to-right. Do not descend below an encountered NP or S. Propose eligible NP candidates there. #### Step 9. Repeat or stop Return to the highest-S check and continue the climb/search procedure until a compatible candidate is accepted or the available trees are exhausted. Record the candidate order rather than only writing an unexplained name. ### Why the first left search has an extra condition In a simple clause such as “John saw him”, the subject NP John is directly under the clause S. There is no extra NP or S between John and that S. The initial-search restriction therefore prevents casually selecting John as the ordinary local antecedent of him. Compare the reflexive “John saw himself”, which is a different grammatical case; do not treat every pronoun type as interchangeable. ### Worked tree 1 · move to the previous sentence **Question. Apply Hobbs’ search to “The student arrived. She smiled.” Assume the student is compatible with she.** #### Given Previous tree: [S [NP The student] [VP arrived]]. Current tree: [S [NP She] [VP smiled]]. #### To find The selected antecedent and the search path. #### Method and why it applies Use the supplied constituent trees. The current sentence has no earlier candidate, so the highest-S rule activates previous-sentence search. ### Solution #### Step 1. Draw both sentence structures ``` Previous sentence Current sentence S S ├── NP: The student ├── NP: She ← start └── VP: arrived └── VP: smiled ``` #### Step 2. Locate and climb Start at NP She. Its first higher NP/S is the current S. Set X to that S. #### Step 3. Search left of the path She is the leftmost subject. No NP lies to its left in the current sentence. #### Step 4. Use the highest-S rule This X is the highest S. Inspect the immediately preceding sentence tree. #### Step 5. Propose its first NP Breadth-first, left-to-right finds NP The student. #### Step 6. Check compatibility and accept The question stipulates that the student agrees with she. Accept this candidate. **Answer.** She refers to the student under the supplied assumptions. ### Worked tree 2 · reject an incompatible subject **Question. Resolve they in “Riya spoke to the students. They laughed.” Use a previous-sentence search and interpret they as plural.** #### Given Previous tree has subject NP Riya, then VP spoke [PP to [NP the students]]. Current subject is NP They. #### To find The candidate order and accepted antecedent. #### Method and why it applies Breadth-first search favours a shallow subject, but agreement must still be applied; a first candidate is not automatically a valid answer. ### Solution #### Step 1. Draw the useful part of the previous tree ``` S ├── NP: Riya ← candidate 1 └── VP ├── V: spoke └── PP ├── P: to └── NP: the students ← candidate 2 ``` #### Step 2. Search the current sentence From NP They climb to its S; there is no earlier NP in “They laughed”. This is the highest S, so move to the previous tree. #### Step 3. Visit Riya first Riya is the shallower leftmost NP. Under the stated plural interpretation, this singular candidate fails number agreement. #### Step 4. Visit the students Continue breadth-first. The next relevant NP is the students. #### Step 5. Apply number agreement Students is plural and is an eligible candidate. It matches plural they. #### Step 6. State the resolution Accept the students. The method did not simply accept the first noun it saw. **Answer.** They → the students. Candidate order: Riya rejected; the students accepted. ### Worked tree 3 · climb out of an embedded clause **Question. Resolve she in “Anita told Ravi that she should revise.” Assume Anita is compatible with she and Ravi is not. Use the displayed parse.** #### Given The pronoun is the subject of an embedded S inside the main VP. #### To find The preferred antecedent found by the climb and left-branch search. #### Method and why it applies The embedded clause has no left candidate. Climbing to the main S exposes its subject Anita as the first suitable higher-level candidate. ### Solution #### Step 1. Show the clause structure ``` S (main) ├── NP: Anita ← accepted candidate └── VP ├── V: told ├── NP: Ravi └── SBAR: that └── S (embedded) ← first X ├── NP: she ← start └── VP: should revise ``` #### Step 2. Start inside the embedded clause From NP she climb to the embedded S. There is no candidate to the left of she within this embedded S. #### Step 3. Climb to the next NP/S The embedded S is not the highest S. Move upward through SBAR and VP to the main S. These intermediate nodes are not NP/S search nodes. #### Step 4. Search left of the upward path At the main S, the subject NP Anita is a shallow left candidate. The object Ravi is deeper within the VP. Breadth-first order proposes Anita first. #### Step 5. Check the stated compatibility Anita is compatible with she by the question's assumption. Accept it; no need to choose a later incompatible candidate. #### Step 6. Give the result and its limit This is the algorithm's result under the displayed parse and agreement assumptions. It does not prove the intention of every speaker using an otherwise ambiguous sentence. **Answer.** She → Anita under the stated tree and compatibility information. ### Advantages, limitations and a complete exam answer | Advantages | Limitations | | --- | --- | | Explicit candidate order makes the decision traceable. | Wrong parse structure can produce the wrong search order. | | Requires no labelled coreference-training corpus for its basic rules. | World knowledge and discourse intention may override syntactic preference. | | Uses structure rather than only surface distance. | Some ambiguous, long-distance or unusual constructions need additional treatment. | For a longer answer, write the definition, inputs, tree notation, ordered algorithm, one labelled tree with a candidate table, final antecedent, and limitations. In a worked question show separately: candidate → why proposed → agreement result → accept/reject. Do not equate the selected candidate with a logically guaranteed interpretation when the sentence is ambiguous. **Yaad rakho:** Tree order candidate deta hai; agreement usko allow/reject karta hai. Nearest noun automatically answer nahi. ### Apply to this sentence 1. Build the main clause with subject NP John; VP gave contains object NP a book and PP to Mary; the after-clause has subject he and object it under reading. 2. For he, ascend from its NP to the governing S/NP and traverse candidate NPs to the left using the Hobbs breadth-first search. Test number/person/gender and binding compatibility. 3. John is an earlier singular human candidate. Under the intended conventional name-gender reading, he→John. Mary is not chosen merely because it is nearer. Names alone do not establish real-world gender; this is the ordinary exercise reading. 4. For it, the inanimate singular a book is compatible with reading; John and Mary are human mentions. Resolve it→a book. 5. Final interpretation: John gave the book after John finished reading the book. The trees/candidate order guide the search; semantic compatibility is also needed. ### The four requested properties | Property | What it means | Apply here | | --- | --- | --- | | Binding | Syntactic restrictions on reference; ordinary personal pronouns and reflexives have different local-antecedent requirements | Do not mechanically link every nearby NP; he in an after/because clause can refer outside that local clause. | | Gender agreement | Candidate must match the pronoun under available linguistic/contextual knowledge | John/he is the exercise’s conventional reading; professor/student do not provide gender by themselves. | | Distance | Token, syntactic or sentence separation between pronoun and candidate | Closer candidates may be preferred, but cannot override incompatibility or Hobbs search order. | | Recency | Recently mentioned discourse entities are often more salient | Mary/student is later than John/professor, but recency is a preference, not a guaranteed antecedent rule. | Reference resolution preserves who performed each action; without it an extraction or QA system could wrongly attribute the reading to Mary. John is the first suitable main-clause subject under the declared parse; the inanimate book fails human he compatibility. --- ## TEE · Q6a · Corpora and acquisition Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q6a Previous examination · complete worked answers **Question. Explain types, characteristics and acquisition of text corpora.** [10 marks] ### Answer **A corpus is a collection of language data assembled for study or computational use.** Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. **Annotation** adds linguistic information such as POS tags, entities, parse structures or sense labels. | Structure | Meaning | Use | | --- | --- | --- | | Plain | Collection without category labels | Word counts, language modelling | | Categorized | Documents grouped by genre/topic | Classification | | Overlapping categories | One document has several categories | Multi-topic news | | Temporal | Documents indexed by time | Language change | | Annotated | Linguistic labels attached | Tagging, parsing, WSD | | Parallel | Aligned texts in different languages | Translation | A **tagset** is the allowed label inventory. A **treebank** is a corpus annotated with syntax trees. **Metadata** records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions. **Raw example:** “Riya works in Pune.” **POS annotation:** Riya/PROPN works/VERB in/ADP Pune/PROPN. **NER annotation:** Riya/PERSON, Pune/LOCATION. Each annotation answers a different task. A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use. Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information. | Resource | Contents | Typical use | | --- | --- | --- | | Gutenberg selection | Literary/book texts | Long-form language and vocabulary | | Web text | Informal domain-specific text | Study usage beyond edited books | | Chat corpus | Conversational posts | Dialogue, informal forms | | Brown | English texts across genres | Genre comparison | | Reuters | News with topic labels | Text classification; overlapping topics | | Inaugural | Speeches indexed by time | Vocabulary change across years | | Wordlist | Words/names/stopwords | Lexical filtering | | Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme | **Gutenberg versus web/chat:** book language does not represent every spelling or conversational pattern in messages. **Brown versus Reuters:** Brown emphasizes genres; Reuters news can have overlapping topic labels. **Inaugural addresses:** dates provide a time dimension. Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property. Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time. **Data acquisition** obtains language examples relevant to the task from existing data, public resources or appropriate new collection. Product instrumentation can gather useful feedback. Supervised learning additionally needs an annotation process when labels are absent. **Data augmentation** creates training variants intended to preserve the correct label. It is not simply changing words arbitrarily. | Method | Procedure | Caution | | --- | --- | --- | | Synonym replacement | Use suitable synonyms for selected words | Choose the right sense | | Back translation | Translate to another language and back | May change meanings/entities | | TF–IDF-informed replacement | Protect informative words while varying others | Do not remove label-bearing clues | | Bigram flipping | Swap adjacent words | Can break grammar or meaning | | Entity replacement | Replace by same-type entity | Must preserve the task label | | Noise injection | Simulate typos/keyboard errors | Match expected noise | | Active learning | Select informative cases for human labels | This acquires labels; it is not synthetic text | ### Three label decisions “The film is good”→“The film is excellent” may preserve a positive sentiment label. “The film is not good”→“The film is good” reverses the label and is not valid label-preserving augmentation. “Flights to Pune”→“Flights to Mumbai” may preserve a travel-topic label but changes the destination label. Split train/test data before creating related augmented variants. Otherwise near-duplicates can leak between sets and produce misleading evaluation. Keep a representative evaluation set of genuine examples. Sentence badal sakta hai; required correct label nahi. Label badla toh augmentation invalid. ### Corpus structure diagram ``` Corpus ├─ Documents (plain collection) │ └─ Sentences → tokens ├─ Categories → document IDs (one or overlapping labels) ├─ Dates → document IDs (temporal collection) ├─ Annotations → token spans / tags / parse trees └─ Parallel pairs → aligned source and target sentences ``` An annotated corpus attaches linguistic labels to the raw text: e.g. Rita/PROPN runs/VERB. A wordlist is a lexical inventory without full discourse context; a Gutenberg corpus contains books; web/chat corpora contain informal online language. Their structures determine which evidence can be queried. --- ## TEE · Q6b · HMM components and estimates Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q6b Previous examination · complete worked answers **Question. Explain HMM components, the number of possible tag sequences and limitations. Construct transition and emission tables for the supplied Alice/Bob/They/The-cat tagged corpus.** [10 marks] ### Answer **A Hidden Markov Model (HMM)** is a probabilistic sequence model in which the underlying state sequence is unobserved, while each state emits an observed item. For POS tagging, words are observed and their grammatical tags are hidden. The task is to infer a suitable tag sequence from the words. Hidden: Noun → Hidden: Verb → Hidden: Noun Observed: birds → Observed: eat → Observed: seeds Horizontal state-to-state links are **transitions**. The relation from each hidden state to its observed word is an **emission**. A transition P(V|N) asks how often a verb tag follows a noun tag. An emission P(eat|V) asks how often the observed word eat appears among verb-tagged tokens. | Component | Notation | Meaning | | --- | --- | --- | | State set | S={N,V,…} | Possible hidden tags. | | Observation vocabulary | O={birds,eat,seeds,…} | Visible word tokens. | | Initial distribution | πⱼ=P(t₁=j) | Probability of starting in tag j. | | Transition matrix | aᵢⱼ=P(tₖ=j|tₖ₋₁=i) | Probability of next tag j after i. | | Emission matrix | bⱼ(w)=P(w|tag j) | Probability that state j emits word w. | Two basic assumptions are used: the current hidden state depends only on the previous hidden state, and the current observation depends only on the current hidden state. A standard first-order HMM therefore does not explicitly condition an emitted word on the whole sentence. $$ P(t_{1:T},w_{1:T})=\pi_{t_1}b_{t_1}(w_1)\prod_{k=2}^{T}a_{t_{k-1},t_k}b_{t_k}(w_k) $$ Read the formula as **start × first emission × transition × emission × transition × emission…**. There are T emission factors and T−1 transitions, plus one initial factor. If the question supplies an explicit end-state probability, multiply that too. Do not invent an end factor when the model has none. **Three different questions.** (1) Scoring a specified tag path uses the joint product above. (2) Finding the probability of the observed words sums scores over all possible tag paths. (3) Finding the best tag path maximizes the path score, typically using Viterbi. A joint path score is not a normalized posterior P(tags|words). **Yaad rakho:** Transition = tag se agla tag. Emission = tag se visible word. If each of T observed words can receive any of N tags, there are N choices at each position. Thus the total number of complete tag sequences is N multiplied by itself T times. This counts possible assignments before grammar rules or zero probabilities eliminate any. $$ \text{Number of tag sequences}=N^T $$ **Question. A four-word sentence is tagged using three possible POS tags per word. How many complete tag sequences are possible?** #### Given N=3 possible tags per position; T=4 word positions. #### To find The number of tag sequences. #### Formula and why it applies $$ M=N^T $$ Every independent position contributes a factor of three choices. #### Solution ##### Step 1. Write one factor per position $$ M=3\times3\times3\times3 $$ ##### Step 2. Multiply $$ 3\times3=9,\quad9\times3=27,\quad27\times3=81 $$ **Final answer.** 81 possible tag sequences. **Question. A five-word sentence has two candidate tags per word. Count its tag sequences.** #### Given N=2; T=5. #### To find M. #### Formula and why it applies $$ M=N^T $$ The length is the exponent, not a multiplier. #### Solution ##### Step 1. Substitute $$ M=2^5 $$ ##### Step 2. Evaluate $$ 2\times2\times2\times2\times2=32 $$ **Final answer.** 32 possible sequences. **Question. Using 45 possible tags, compare the number of tag sequences for sentences of length 2 and 3.** #### Given N=45. First T=2, then T=3. #### To find Both counts and the growth factor. #### Formula and why it applies $$ M_2=45^2,\quad M_3=45^3 $$ Adding one position multiplies the possibilities by the number of tag choices. #### Solution ##### Step 1. Two words $$ 45^2=2025 $$ ##### Step 2. Three words $$ 45^3=2025\times45=91125 $$ ##### Step 3. Compare $$ 91125/2025=45 $$ **Final answer.** 2 words: 2025 paths. 3 words: 91125 paths. One additional word multiplies the count by 45. The slides introduce Viterbi to avoid enumerating every path. It reuses the best prefix score for each ending state. For a dense first-order HMM, it needs O(TN²) recurrence work instead of enumerating NT complete paths. **Yaad rakho:** Choices per word ki power = sentence length. ### Exact corpus calculation ### Given | Sentence | Given tags | | --- | --- | | Alice will eat cookies | N M V N | | Bob can drink tea | N M V N | | They will watch movies | Pronoun M V N | | The cat is sleeping | Det N V V | ### To find Emission and transition tables ### Formula: which one and why $$ P(w\mid t)=\frac{c(t,w)}{c(t)} $$ $$ P(t_j\mid t_i)=\frac{c(t_i,t_j)}{\sum_kc(t_i,t_k)} $$ The second formula is the declared no-END convention. Reset at sentence boundaries; never create a transition from the last word of one training sentence to the first of the next. ### Step 1. Count emissions | Tag | Total occurrences | Alice | will | eat | cookies | Bob | can | drink | tea | They | watch | movies | The | cat | is | sleeping | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | N | 6 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | | M | 3 | 0 | 2 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | V | 5 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | | Pronoun | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | | Det | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | ### Step 2. Divide each emission by its tag count | Tag | Alice | will | eat | cookies | Bob | can | drink | tea | They | watch | movies | The | cat | is | sleeping | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | N | 1/6 | 0 | 0 | 1/6 | 1/6 | 0 | 0 | 1/6 | 0 | 0 | 1/6 | 0 | 1/6 | 0 | 0 | | M | 0 | 2/3 | 0 | 0 | 0 | 1/3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | | V | 0 | 0 | 1/5 | 0 | 0 | 0 | 1/5 | 0 | 0 | 1/5 | 0 | 0 | 0 | 1/5 | 1/5 | | Pronoun | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | | Det | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | ### Step 3. Count transitions inside each sentence | From | Outgoing transitions | Sentence-final count | N | M | V | Pronoun | Det | | --- | --- | --- | --- | --- | --- | --- | --- | | N | 3 | 3 | 0 | 2 | 1 | 0 | 0 | | M | 3 | 0 | 0 | 0 | 3 | 0 | 0 | | V | 4 | 1 | 3 | 0 | 1 | 0 | 0 | | Pronoun | 1 | 0 | 0 | 1 | 0 | 0 | 0 | | Det | 1 | 0 | 1 | 0 | 0 | 0 | 0 | ### Step 4. Convert to a transition table | From / to | N | M | V | Pronoun | Det | | --- | --- | --- | --- | --- | --- | | N | 0 | 2/3 | 1/3 | 0 | 0 | | M | 0 | 0 | 1 | 0 | 0 | | V | 3/4 | 0 | 1/4 | 0 | 0 | | Pronoun | 0 | 1 | 0 | 0 | 0 | | Det | 1 | 0 | 0 | 0 | 0 | Here sentence-final tags are excluded from the no-END outgoing denominator. If END is explicitly modeled, divide all tag→tag and tag→END counts by the total tag count instead; the row then includes END. Do not mix denominators. ### Final answer The two displayed probability tables are the estimated HMM. Zero means unobserved without smoothing, not logically impossible. For T tokens and K possible tags per token, exhaustive assignments number K^T before constraints; the paper does not supply a single test length for a specific numerical total. HMM assumptions limit long-range context, first-order transitions and emission independence; sparsity and unknown words require smoothing or other handling. The supplied tag is for is and sleeping is V, so both count as V even though finer linguistic tagsets distinguish auxiliaries/participles. --- ## TEE · Q7a · Word classes, chunking and chinking Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q7a Previous examination · complete worked answers **Question. Explain open/closed word classes, chunking and chinking with examples.** [5 marks] ### Answer **Part-of-speech (POS) tagging** assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence. | Tag | Role | Example in context | | --- | --- | --- | | NN / NNS | Singular / plural common noun | a book / two books | | NNP | Proper noun | Mary, Mumbai | | VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads | | JJ | Adjective | the red book | | RB | Adverb | runs quickly | | DT | Determiner | the, a, this | | IN | Preposition or subordinating conjunction | on the table | | PRP | Personal pronoun | she, they | | MD | Modal auxiliary | can, will, should | **Open classes** readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. **Closed classes** contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes. Input: Book that flight → Context: an imperative request → Output: Book/VB that/DT flight/NN Compare “hand me that **book**”: book is a noun here. Similarly, “I **can** swim” has can/MD, while “the **can** is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is **POS ambiguity**. | Sentence | Tagged interpretation | Reason | | --- | --- | --- | | They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. | | The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. | | She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. | | Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. | POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning. **Yaad rakho:** Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai. **Chunking**, also called shallow parsing, groups adjacent tagged words into meaningful non-overlapping units, commonly noun phrases. It usually identifies local phrase boundaries without producing the complete hierarchical structure of a sentence. **Chinking** removes specified tokens from an existing candidate chunk, splitting or trimming it. The/DT small/JJ cat/NN sat/VBD on/IN the/DT mat/NN → NP: [The small cat] → Outside: sat on → NP: [the mat] A POS tagger labels individual words. A chunker groups words into phrases. A full parser shows how phrases relate recursively to form the entire sentence. These are related tasks with different outputs. | Operation | Input | Output | | --- | --- | --- | | POS tagging | The small cat | The/DT small/JJ cat/NN | | NP chunking | Tagged tokens | [NP The small cat] | | Chinking | A broad candidate chunk | Remove verbs/prepositions, keep noun-phrase segments | | Full parsing | Sentence + grammar | A nested tree with sentence, noun phrase and verb phrase nodes | **Rule-based chunking** uses tag patterns. Statistical or learned chunkers infer boundaries from labeled examples. A useful NP rule is an optional determiner, followed by zero or more adjectives, followed by a noun. In NLTK-style notation: `NP: {
?*}`. Here ? means zero or one occurrence; \* means zero or more; + means one or more when used. The braces mean “form a chunk around the matched sequence”. **Question. Apply the NP rule optional DT, zero or more JJ, then one NN to “The/DT quick/JJ brown/JJ fox/NN jumps/VBZ over/IN the/DT lazy/JJ dog/NN”.** #### Given Tagged input is supplied. Rule: DT? JJ\* NN. #### To find All noun-phrase chunks. #### Formula and why it applies `NP → DT? JJ* NN` The rule groups a noun with its preceding determiner and adjectives; a verb or preposition does not match this NP pattern. #### Solution ##### Step 1. Start at The The matches optional DT; quick and brown both match JJ\*; fox matches NN. ##### Step 2. Form the first NP **[NP The quick brown fox]** ##### Step 3. Skip nonmatching tokens jumps/VBZ and over/IN do not satisfy the NP pattern. ##### Step 4. Match the second NP the is DT, lazy is JJ, dog is NN: **[NP the lazy dog]**. **Final answer.** [NP The quick brown fox] jumps over [NP the lazy dog]. **Question. Using NP → DT? JJ\* NN, identify the chunks in “bright/JJ stars/NN shine/VBP and/CC a/DT moon/NN glows/VBZ”. Use the supplied tags exactly.** #### Given Input tags label stars as NN for this exercise. Rule requires optional DT, zero or more JJ, then NN. #### To find The matched NP spans. #### Formula and why it applies `NP → DT? JJ* NN` The optional determiner may be absent and the adjective repetition may be empty. #### Solution ##### Step 1. Match bright stars No determiner is present, which DT? permits. bright matches JJ; stars matches NN. ##### Step 2. Inspect shine and and VBP and CC fall outside the NP rule. ##### Step 3. Match a moon a matches DT; no adjective is present; moon matches NN. **Final answer.** Chunks are [NP bright stars] and [NP a moon]. With ordinary Penn tagging, stars is usually NNS; broaden the noun pattern to include NNS when that tag is used. **Question. Begin with one candidate chunk containing “The/DT cat/NN sat/VBD on/IN the/DT mat/NN”. Chink out verbs and prepositions. Show the remaining chunks.** #### Given Initial candidate: [The cat sat on the mat]. Remove tags VBD and IN. #### To find The chunk segments after exclusion. #### Formula and why it applies Chinking removes matched tokens from the existing chunk. Removing the middle verb/preposition region divides the original candidate into two remaining contiguous spans. #### Solution ##### Step 1. Locate the tokens to remove sat is VBD and on is IN. ##### Step 2. Remove those tokens from chunk membership The remaining token groups are “The cat” and “the mat”. sat and on still exist in the sentence; they are simply outside these chunks. ##### Step 3. Write the final chunked sentence **[The cat]** sat on **[the mat]**. **Final answer.** Two chunks remain: The cat; the mat. **Yaad rakho:** Chunking = andar group karo; chinking = unwanted tokens ko group se bahar karo. --- ## TEE · Q7b · Named entity recognition Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q7b Previous examination · complete worked answers **Question. Explain named-entity recognition and its approaches with examples.** [5 marks] ### Answer **Named Entity Recognition (NER)** identifies text spans referring to entities and assigns their types, such as PERSON, ORGANIZATION and LOCATION. Many practical schemas also include DATE, TIME, MONEY or domain-specific types. The exact allowed labels depend on the application's annotation scheme. NER has two parts: **boundary detection** finds where the entity starts and ends; **classification** assigns the appropriate type. A multiword name should normally be one entity span: “New York” is a location span, not two unrelated location decisions. Text: Mary joined Google in New York on Monday. → Mary → PERSON → Google → ORGANIZATION → New York → LOCATION → Monday → DATE Not every noun is a named entity. In “a developer works at Google”, developer is an ordinary job description while Google is a named organization. A specialized HR system may define a JOB\_TITLE category, but it must state that schema explicitly. NER identifies mentions; entity linking is the separate task of connecting a mention to a unique database identity. | Task | Question it answers | Example | | --- | --- | --- | | POS | What grammatical role does this token have? | Google/NNP | | Chunking | Which adjacent words form a phrase? | [NP a skilled developer] | | NER | Which span names an entity, and of what type? | [Google] ORGANIZATION | | Entity linking | Which real entity does this mention denote? | Connect Paris to the intended city or person record | **Applications.** NER structures search results and documents, extracts organizations/people from news, routes support tickets by product or organization, supports recommendation, and extracts specialized entities from healthcare or legal text. Context is essential: “Apple released a laptop” names an organization, while “she ate an apple” describes fruit. **Yaad rakho:** NER mein do kaam: naam kahan se kahan tak, aur kis category ka. | Approach | How it works | Example | Main limitation | | --- | --- | --- | --- | | Dictionary / gazetteer | Match text against lists of known names | A city list includes Mumbai and Pune | New names, spelling variants and ambiguous names are missed | | Rule-based | Use word shapes, regular expressions and context | A title such as Dr. followed by a name-like span suggests PERSON | Manual maintenance; exceptions and domain/language dependence | | Machine learning | Learn a labeling rule from annotated examples using features | Capitalization, suffixes, POS and surrounding words help a classifier/CRF | Requires labeled data and useful features | | Deep learning | Learn contextual word/character representations and predict labels | A contextual sequence encoder labels multiword entity spans | Training/domain dependence and resource demands | A **pattern-based rule** examines the form of text, such as an email pattern or a date format. A **context-based rule** examines neighboring words: “Dr.” can support a following PERSON span, while “headquartered in” can support a following LOCATION span. Capitalization alone is insufficient because sentence-initial common nouns are also capitalized. A dictionary may find “May” as a person's name, but “in May” may refer to a month. A learned model uses surrounding evidence to disambiguate. Hybrid systems can combine dictionaries, rules and learned models; these categories are practical tools rather than mutually exclusive restrictions. **Preprocessing for NER.** Tokenize while preserving token offsets and useful capitalization. Blindly removing capital letters or stopwords before recognition can damage names such as “Bank of India”. NER output should preserve the original span so the reader can locate it in the input. **Exam-ready 3-mark answer.** NER locates named-entity spans and classifies them into categories such as person, organization and location. A dictionary-based approach matches known name lists; a rule-based approach uses patterns or context; machine-learning and deep-learning approaches learn entity decisions from labeled data and representations. For example, in “Mary works at Google in Mumbai”, Mary is PERSON, Google is ORGANIZATION and Mumbai is LOCATION. --- ## TEE · Q7c · Co-occurrence with inconsistent requested comparisons Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q7c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r25/q7c Previous examination · complete worked answers **Question. Construct a whole-sentence co-occurrence matrix for: The bird flew across the sky. The cat watched the bird. The flowers grew in the garden. The bird rested on the tree. The cat slept near the tree. Targets={bird,cat,garden,flowers}; contexts={sky,tree,garden,slept}. The printed follow-up asks dot products boy–dog and boy–sun and a plot on kicked/field dimensions.** [10 marks] ### Answer ### Given Use the exact target/context lists above. ### To find Construct the supported matrix and identify which later requests are undefined. ### Formula: which one and why Count ordered target-token/context-token pairs in the same sentence, excluding the target token itself. A single garden occurrence cannot co-occur with itself. If a classroom convention uses sentence-level presence including diagonal, only garden→garden changes from0to1; state that choice. ### Step 1. Enumerate contributions and sum | Sentence | Target occurrences | Context occurrences | | --- | --- | --- | | 1 | bird | sky | | 2 | cat, bird | | | 3 | flowers, garden | garden | | 4 | bird | tree | | 5 | cat | slept, tree | | Target / context | sky | tree | garden | slept | | --- | --- | --- | --- | --- | | bird | 1 | 1 | 0 | 0 | | cat | 0 | 1 | 0 | 1 | | garden | 0 | 0 | 0 | 0 | | flowers | 0 | 0 | 1 | 0 | ### Step 2. Check requested names before multiplying boy,dog,sun are not target rows; kicked,field are not context columns. Therefore the requested dot products and plot cannot be calculated from the supplied matrix. Zero vectors for invented words would not repair the missing givens. ### Step 3. Clearly labelled corrected illustration If the follow-up were changed to bird–cat and bird–flowers, using the displayed rows: bird·cat=1×0+1×1+0×0+0×1=1; bird·flowers=1×0+1×0+0×1+0×0=0. A sky/tree plot would use bird(1,1),cat(0,1),garden(0,0),flowers(0,0). These substitutions are teaching examples, not printed instructions. ### Final answer **The original matrix is solved; its requested boy/dog/sun comparisons and kicked/field plot are not well-defined.** Ask for corrected target/context names rather than inventing a numerical answer. **Source / convention note.** Source defect: follow-up names belong to another corpus. Exact supported matrix is complete; unsupported parts are explicitly unresolved from the original data. --- ## TEE · Q1a · TF–IDF in retrieval Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q1a Previous examination · complete worked answers **Question. Write TF–IDF expressions and explain their role in information retrieval.** [individual allocation not printed; Q1 totals 20] ### Answer **TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus.** The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency: $$ \operatorname{TF}(t,d)=\frac{C(t,d)}{N_d} $$ $$ \operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)} $$ $$ \operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t). $$ $$ N_d $$ is the document's token count after the specified preprocessing; $$ D $$ is number of documents; $$ \operatorname{df}(t) $$ is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency. **Why the log?** It compresses the ratio's scale. If a term occurs in all documents, $$ D/df=1 $$ , so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default. **Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.** #### Given D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10. #### To find The TF–IDF weight. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ , then multiply. #### Step 1. TF $$ 3/12=0.25 $$ . #### Step 2. Document ratio $$ 10/2=5 $$ . #### Step 3. IDF $$ \log_{10}5\approx0.698970 $$ . #### Step 4. Weight $$ 0.25\times0.698970\approx0.174743 $$ . **Answer.** TF=0.25, IDF≈0.698970, TF–IDF≈0.174743. **Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.** #### Given C=4, Nd=20, D=8, df=8. #### To find Its weight under unsmoothed base-10 IDF. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ . #### Step 1. TF $$ 4/20=0.2 $$ . #### Step 2. IDF $$ \log_{10}(8/8)=\log_{10}1=0 $$ . #### Step 3. Multiply $$ 0.2\times0=0 $$ . **Answer.** Weight 0: this term does not distinguish documents under this convention. **Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.** #### Given No further preprocessing. Each term occurs in exactly two of the three documents. #### To find All three vectors. #### Formula and why it applies First count df by documents, then compute each document's TF, then multiply by its term's IDF. #### Step 1. Common IDF $$ D=3 $$ , $$ df(cat)=df(dog)=df(fish)=2 $$ . Each IDF $$ =\log_{10}(3/2)\approx0.176091 $$ . #### Step 2. D1 term frequencies Length 3 → $$ [2/3,1/3,0] $$ . #### Step 3. D1 weights $$ [(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0] $$ . #### Step 4. D2 weights Length 2; TF $$ [0,1/2,1/2] $$ . Vector $$ \approx[0,0.088046,0.088046] $$ . #### Step 5. D3 weights Length 3; TF $$ [1/3,0,2/3] $$ . Vector $$ \approx[0.058697,0,0.117394] $$ . **Answer.** D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394]. TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply. **Information retrieval (IR)** locates and ranks items relevant to an information need, usually expressed as a query. The collection may contain documents, product descriptions or scientific papers. Retrieval estimates relevance; it does not itself establish that every retrieved claim is true. Documents → cleaning → Build index → Query → same representation → Match & rank → Return relevant results ### The inverted index An inverted index maps a term to the documents containing it, often with positions or counts. For D1 “cats drink milk” and D2 “dogs drink water”, the posting list for drink is {D1,D2}, while milk maps only to D1. This avoids rescanning every document for every query. | Term | Posting list | | --- | --- | | cats | D1 | | drink | D1, D2 | | milk | D1 | | dogs | D2 | | water | D2 | ### Matching and ranking Boolean retrieval combines term membership using AND, OR and NOT. Vector-space retrieval represents documents and queries as weighted vectors, for example with TF-IDF, and compares them with cosine similarity. Learned retrieval can use dense embeddings for meaning-based matching. The representation used at query time must be compatible with the document index. ### Search engines and recommendation A search engine receives an explicit query and ranks matching indexed pages. A recommender may derive an implicit query from a user’s liked items or profile and retrieve similar candidates. Both require candidate retrieval and ranking, but the source of the information need differs. ### Evaluate the results For retrieval, TP is the number of retrieved relevant documents, FP retrieved irrelevant documents, and FN relevant documents not retrieved. Precision and recall use the same formulas as classification. Ranking measures also consider the positions of relevant items; a relevant result at rank 1 is more useful than one far below. ### Retrieval evaluation A search system retrieves 8 documents; 6 are relevant. There are 10 relevant documents in the collection. Find precision, recall and F1. **Given** Retrieved=8; relevant retrieved=6; all relevant=10. **To find** P, R and F1. ### Formula and why it applies $$ P=\frac{\text{relevant retrieved}}{\text{retrieved}},\quad R=\frac{\text{relevant retrieved}}{\text{all relevant}} $$ $$ F_1=\frac{2PR}{P+R} $$ The denominator is 8 for precision and 10 for recall because they ask different questions. ### Solution #### Step 1. Identify the confusion counts TP=6; FP=8−6=2; FN=10−6=4. #### Step 2. Precision $$ P=\frac68=0.75 $$ #### Step 3. Recall $$ R=\frac6{10}=0.6 $$ #### Step 4. F1 $$ F_1=\frac{2(0.75)(0.6)}{0.75+0.6}=\frac{0.9}{1.35}=\frac23\approx0.666667 $$ Precision=0.75, recall=0.60, F1≈0.666667. **Yaad rakho:** Precision: jo laaye, unmein kitne sahi? Recall: jo sahi the, unmein kitne laaye? --- ## TEE · Q1b · NER and approaches Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q1b Previous examination · complete worked answers **Question. Explain NER with examples and approaches.** [individual allocation not printed; Q1 totals 20] ### Answer **Named Entity Recognition (NER)** identifies text spans referring to entities and assigns their types, such as PERSON, ORGANIZATION and LOCATION. Many practical schemas also include DATE, TIME, MONEY or domain-specific types. The exact allowed labels depend on the application's annotation scheme. NER has two parts: **boundary detection** finds where the entity starts and ends; **classification** assigns the appropriate type. A multiword name should normally be one entity span: “New York” is a location span, not two unrelated location decisions. Text: Mary joined Google in New York on Monday. → Mary → PERSON → Google → ORGANIZATION → New York → LOCATION → Monday → DATE Not every noun is a named entity. In “a developer works at Google”, developer is an ordinary job description while Google is a named organization. A specialized HR system may define a JOB\_TITLE category, but it must state that schema explicitly. NER identifies mentions; entity linking is the separate task of connecting a mention to a unique database identity. | Task | Question it answers | Example | | --- | --- | --- | | POS | What grammatical role does this token have? | Google/NNP | | Chunking | Which adjacent words form a phrase? | [NP a skilled developer] | | NER | Which span names an entity, and of what type? | [Google] ORGANIZATION | | Entity linking | Which real entity does this mention denote? | Connect Paris to the intended city or person record | **Applications.** NER structures search results and documents, extracts organizations/people from news, routes support tickets by product or organization, supports recommendation, and extracts specialized entities from healthcare or legal text. Context is essential: “Apple released a laptop” names an organization, while “she ate an apple” describes fruit. **Yaad rakho:** NER mein do kaam: naam kahan se kahan tak, aur kis category ka. | Approach | How it works | Example | Main limitation | | --- | --- | --- | --- | | Dictionary / gazetteer | Match text against lists of known names | A city list includes Mumbai and Pune | New names, spelling variants and ambiguous names are missed | | Rule-based | Use word shapes, regular expressions and context | A title such as Dr. followed by a name-like span suggests PERSON | Manual maintenance; exceptions and domain/language dependence | | Machine learning | Learn a labeling rule from annotated examples using features | Capitalization, suffixes, POS and surrounding words help a classifier/CRF | Requires labeled data and useful features | | Deep learning | Learn contextual word/character representations and predict labels | A contextual sequence encoder labels multiword entity spans | Training/domain dependence and resource demands | A **pattern-based rule** examines the form of text, such as an email pattern or a date format. A **context-based rule** examines neighboring words: “Dr.” can support a following PERSON span, while “headquartered in” can support a following LOCATION span. Capitalization alone is insufficient because sentence-initial common nouns are also capitalized. A dictionary may find “May” as a person's name, but “in May” may refer to a month. A learned model uses surrounding evidence to disambiguate. Hybrid systems can combine dictionaries, rules and learned models; these categories are practical tools rather than mutually exclusive restrictions. **Preprocessing for NER.** Tokenize while preserving token offsets and useful capitalization. Blindly removing capital letters or stopwords before recognition can damage names such as “Bank of India”. NER output should preserve the original span so the reader can locate it in the input. **Exam-ready 3-mark answer.** NER locates named-entity spans and classifies them into categories such as person, organization and location. A dictionary-based approach matches known name lists; a rule-based approach uses patterns or context; machine-learning and deep-learning approaches learn entity decisions from labeled data and representations. For example, in “Mary works at Google in Mumbai”, Mary is PERSON, Google is ORGANIZATION and Mumbai is LOCATION. --- ## TEE · Q1c · Skip-gram and CBOW Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q1c Previous examination · complete worked answers **Question. Explain Skip-gram and CBOW.** [individual allocation not printed; Q1 totals 20] ### Answer **Word2Vec learns dense word embeddings by predicting words from nearby context.** Its two common architectures are CBOW and skip-gram. A vocabulary ID or one-hot vector is the input representation; the learned embedding table converts that identity into a short trainable vector. | Architecture | Input | Prediction | Memory cue | | --- | --- | --- | --- | | CBOW | Surrounding context words | Centre/target word | Context → centre | | Skip-gram | Centre/target word | Surrounding words | Centre → context | CBOW: the + sat → Average context embeddings → Predict: cat Skip-gram: cat → Embedding of cat → Predict: the and sat For vocabulary size $$ V $$ and embedding dimension $$ d $$ , an input one-hot row is $$ 1\times V $$ ; the input embedding matrix is $$ V\times d $$ ; the hidden row is $$ 1\times d $$ ; the output matrix is $$ d\times V $$ ; output scores are $$ 1\times V $$ . Multiplying a one-hot row by the embedding matrix simply selects one row. $$ P(w_o\mid w_i)=\frac{\exp(u_{w_o}^{T}v_{w_i})}{\sum_{w\in V}\exp(u_w^{T}v_{w_i})}. $$ $$ v_{w_i} $$ is the input word embedding, $$ u_w $$ an output-context embedding, and the dot product is a compatibility score. Softmax turns all scores into probabilities that sum to one. Training increases the probability of observed context words. Negative sampling or hierarchical softmax can reduce computation in larger vocabularies; use the algorithm specified by the question. **Question. For the sentence “the cat sat on mat”, generate CBOW and skip-gram training data for centre word sat with a window of one token on each side.** #### Given Tokens indexed 1 to 5: the, cat, sat, on, mat. Centre = sat. #### To find The CBOW example and the skip-gram pairs. #### Formula and why it applies CBOW predicts the centre from its neighbours; skip-gram predicts each neighbour from the centre. #### Step 1. Identify neighbours Left neighbour = cat; right neighbour = on. “the” and “mat” are outside this window. #### Step 2. CBOW example Input [cat,on] → target sat. #### Step 3. Skip-gram examples (sat,cat) and (sat,on), with first item as input and second as predicted context. **Answer.** CBOW: [cat,on]→sat. Skip-gram: sat→cat and sat→on. **Question. The embeddings of context words cat and on are [1,2] and [3,0]. Compute the CBOW hidden vector using mean aggregation. Output vectors for [sat,ran] are [1,0] and [0,1]. Compute softmax probabilities.** #### Given Context embeddings [1,2], [3,0]; mean aggregation; two candidate output vectors. #### To find The hidden vector, logits and probabilities. #### Formula and why it applies Mean context vector $$ h=(v_1+v_2)/2 $$ . Logit $$ z_w=u_w^Th $$ . Softmax $$ P(w)=e^{z_w}/\sum_j e^{z_j} $$ . #### Step 1. Add coordinates $$ [1+3,\,2+0]=[4,2] $$ . #### Step 2. Divide by number of words $$ h=[4/2,2/2]=[2,1] $$ . #### Step 3. Compute logits $$ z_{sat}=1(2)+0(1)=2 $$ . $$ z_{ran}=0(2)+1(1)=1 $$ . #### Step 4. Exponentials $$ e^2\approx7.389056 $$ , $$ e^1\approx2.718282 $$ ; sum $$ \approx10.107338 $$ . #### Step 5. Divide $$ P(sat)\approx0.731059 $$ , $$ P(ran)\approx0.268941 $$ . **Answer.** h=[2,1]; sat is more probable (0.731059). **Question. A skip-gram centre embedding is v=[1,−1]. Output embeddings for [eat,sleep,drive] are [1,1], [0,1], [1,0]. Calculate the predicted context probabilities.** #### Given One centre vector and three output vectors; full softmax over these three words. #### To find Three logits and three probabilities. #### Formula and why it applies $$ z_j=u_j^Tv $$ , then $$ P(j)=e^{z_j}/\sum_k e^{z_k} $$ . #### Step 1. Eat dot product $$ 1(1)+1(-1)=0 $$ . #### Step 2. Sleep and drive dot products $$ z_{sleep}=0(1)+1(-1)=-1 $$ . $$ z_{drive}=1(1)+0(-1)=1 $$ . #### Step 3. Exponentiate and sum $$ e^0=1 $$ , $$ e^{-1}\approx0.367879 $$ , $$ e^1\approx2.718282 $$ . Sum $$ \approx4.086161 $$ . #### Step 4. Normalize $$ P(eat)\approx0.244728 $$ ; $$ P(sleep)\approx0.090031 $$ ; $$ P(drive)\approx0.665241 $$ . **Answer.** drive has the highest probability in this toy learned model. **Limitations:** a standard static Word2Vec vector merges a word's senses; it has limited representation of subword morphology and out-of-vocabulary words; its local prediction objective does not directly fit a full global co-occurrence table. Similarity is learned from data and is not a guarantee of factual or causal understanding. CBOW = neighbours se missing centre. Skip-gram = centre se neighbours. --- ## TEE · Q1d · Pragmatics and discourse Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q1d Previous examination · complete worked answers **Question. Explain pragmatics and discourse in NLP.** [individual allocation not printed; Q1 totals 20] ### Answer **Pragmatic analysis** interprets an utterance using its situation and the speaker’s intended meaning. **Discourse analysis** studies how utterances connect across a text or conversation. Syntax and literal word meanings alone cannot settle every interpretation. | Text | Literal content | Contextual interpretation | | --- | --- | --- | | “Can you close the window?” | A question about ability. | Usually a polite request. | | “Meera bought a laptop. It was expensive.” | Two sentences. | “It” connects the second sentence to the laptop. | | “Meet me here tomorrow.” | An instruction. | Me, here and tomorrow depend on speaker, place and date. | ### Reference, mention and antecedent A **referring expression** points to an entity; a **mention** is its occurrence in the text. Its **referent** is the entity being discussed. An **antecedent** is a preceding expression used to interpret a later referring expression. Reference resolution identifies the intended referent. Coreference resolution groups mentions that refer to the same entity. “Meera”name mention → Person entitysame referent → “she” / “the engineer”other mentions | Concept | Definition | Example | | --- | --- | --- | | Anaphora | Reference back to something introduced earlier. | “Riya arrived. She sat down.” She → Riya. | | Cataphora | Reference forward to a later expression. | “Before she spoke, Riya checked the microphone.” she → Riya. | | Exophora | Reference to the situation outside the text. | “Put it there”, said while pointing. | | Coreference | Two or more mentions refer to the same entity. | “Dr Rao”, “the lecturer”, “she”. | | Pronoun resolution | Find the referent of a pronoun. | Resolve “it” in “The laptop stopped because it overheated.” | Coreference is broader than pronoun resolution: repeated names and descriptions also form a chain. Anaphora describes direction/dependence; coreference describes identity. In “I bought a car. The engine is noisy”, engine is related to the car but is not the same entity. **Yaad rakho:** Reference = kiski baat? Coreference = alag mentions, same entity. Pronoun resolution = he/she/it kis se linked? ### Cohesion, coherence and discourse markers **Cohesion** is the explicit linguistic linking of a text through reference, repeated or related words, conjunctions and other devices. In “Rita bought a book. She read it”, she and it connect the second sentence to Rita and the book. **Coherence** is the conceptual unity that makes a discourse meaningful. “The road flooded, so the match was cancelled” presents a plausible cause and consequence. A reader relates the propositions using context and knowledge. **Discourse markers** signal relationships or organize speech: however signals contrast; therefore signals a result; first and next signal sequence; for example introduces an illustration. These concepts differ: “I like apples. Therefore the moon is square” has a linking word but no sensible inference. A connector alone does not guarantee coherence. **Yaad rakho:** Cohesion = words se link; coherence = meaning se link; marker = relation ka signal. --- ## TEE · Q2a · Six preprocessing operations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q2a Previous examination · complete worked answers **Question. Explain lowercasing, tokenization, stopword removal, stemming, lemmatization and POS tagging with examples.** [6 marks] ### Answer **Tokenization divides text into units called tokens for further processing.** A token may be a word, punctuation mark, character or subword, depending on the tokenizer. **Sentence segmentation** identifies sentence boundaries. Tokenization is not necessarily just splitting on spaces. | Method | Example | Trade-off | | --- | --- | --- | | Word | “cats sleep” → [cats,sleep] | Easy to interpret; unfamiliar words can be unseen tokens | | Character | cat → [c,a,t] | Handles spelling units; creates longer sequences | | Subword | unhappiness → pieces such as [un,happi,ness] in an illustrative vocabulary | Balances vocabulary size and rare-word representation | | Sentence | “Birds fly. Fish swim.” → two sentences | Abbreviations and punctuation complicate boundaries | Actual subword splits depend on the learned vocabulary; the example is illustrative, not a universal tokenizer output. Word segmentation is especially important where spaces do not reliably separate words or in compound-rich languages. **Question. Tokenize “Dr. Rao can't come today.” into words and punctuation under this rule: preserve Dr. and can't as whole tokens, and split only the final full stop.** #### Given The tokenization rule is explicitly supplied. #### To find The resulting token sequence. #### Formula and why it applies Apply the stated boundary rule; do not assume every period or apostrophe is a split point. #### Step 1. Handle abbreviation Dr. remains one token. #### Step 2. Handle contraction can't remains one token. #### Step 3. Write sequence [Dr., Rao, can't, come, today, .] **Answer.** 6 tokens under the specified rule. Another tokenizer may legitimately produce a different sequence. ### Three difficult cases **New York** may be two tokens but one named entity. **3.14** should not normally end a sentence at the decimal point. **don't** may remain whole or split into do + n't, depending on the tokenizer. Always state the convention when calculating counts. Token = chosen unit. Space split ko universal rule mat samajh. **Normalization reduces superficial variation so comparable forms can be processed consistently.** Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization. **Stopwords** are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us. **Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.** #### Given Operations and stopword set are fixed. Do not remove not. #### To find The final token list. #### Formula and why it applies Apply the requested operations in order, keeping a visible intermediate result after each transformation. #### Step 1. Lowercase the cats are not sleeping!!! #### Step 2. Tokenize and remove punctuation [the,cats,are,not,sleeping] #### Step 3. Remove specified stopwords Remove the and are → [cats,not,sleeping]. #### Step 4. Lemmatize cats→cat; sleeping→sleep → [cat,not,sleep]. **Answer.** [cat,not,sleep]. The negative meaning is retained. ### Task decides preprocessing For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them. Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi. **Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma. | Criterion | Stemming | Lemmatization | | --- | --- | --- | | Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis | | Output | May be nonword, e.g. studi | Dictionary form, e.g. study | | Context/POS | Usually little or none | Often important | | Irregular forms | May miss went→go | Can map went→go with verb analysis | | Cost | Generally simpler | Usually more linguistic processing | Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation. ### Three worked reductions 1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked. 2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation. 3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter. **Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice. Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan. **Part-of-speech (POS) tagging** assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence. | Tag | Role | Example in context | | --- | --- | --- | | NN / NNS | Singular / plural common noun | a book / two books | | NNP | Proper noun | Mary, Mumbai | | VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads | | JJ | Adjective | the red book | | RB | Adverb | runs quickly | | DT | Determiner | the, a, this | | IN | Preposition or subordinating conjunction | on the table | | PRP | Personal pronoun | she, they | | MD | Modal auxiliary | can, will, should | **Open classes** readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. **Closed classes** contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes. Input: Book that flight → Context: an imperative request → Output: Book/VB that/DT flight/NN Compare “hand me that **book**”: book is a noun here. Similarly, “I **can** swim” has can/MD, while “the **can** is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is **POS ambiguity**. | Sentence | Tagged interpretation | Reason | | --- | --- | --- | | They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. | | The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. | | She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. | | Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. | POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning. **Yaad rakho:** Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai. --- ## TEE · Q2b · Lexical similarity and cosine Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q2b Previous examination · complete worked answers **Question. Explain the need for lexical similarity and illustrate cosine similarity.** [4 marks] ### Answer Similarity supports retrieval, clustering, duplicate detection and semantic matching. Choose a representation first: character overlap measures spelling; context embeddings approximate semantic relatedness. An illustrative cosine computation follows; these vectors are chosen to explain the method, not given exam data. ### Given u=[1, 1, 0]; v=[1, 0, 1] ### To find Cosine similarity in the given coordinate order. ### Formula: which one and why $$ \cos(u,v)=\frac{u\cdot v}{\|u\|\|v\|} $$ ### Step 1. Multiply matching coordinates 1×1 + 1×0 + 0×1 = 1 ### Step 2. Calculate both squared lengths u: 1² + 1² + 0² = 2; v: 1² + 0² + 1² = 2 ### Step 3. Take square roots and divide 1 / (√2 × √2) = 0.500000000 ### Final answer **Cosine=0.500000000.** This measures direction, not a probability. --- ## TEE · Q2c · Animal categorical Naive Bayes Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q2c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q2c Previous examination · complete worked answers **Question. Using the printed20-animal training table, classify an animal with Give Birth=Yes, Can Fly=No, Live in Water=Yes, Have Legs=No. Treat Name only as an identifier.** [10 marks] ### Answer Rows below follow the source order: Human, Python, Salmon, Whale, Frog, Komodo, Bat, Pigeon, Cat, Leopard Shark, Turtle, Penguin, Porcupine, Eel, Salamander, Gila Monster, Platypus, Owl, Dolphin, Eagle. The source assigns Leopard Shark to Non-mammals and Platypus to Mammals; retain those labels. ### Given | Give Birth | Can Fly | Live in Water | Have Legs | Class | | --- | --- | --- | --- | --- | | Yes | No | No | Yes | Mammals | | No | No | No | No | Non-mammals | | No | No | Yes | No | Non-mammals | | Yes | No | Yes | No | Mammals | | No | No | Sometimes | Yes | Non-mammals | | No | No | No | Yes | Non-mammals | | Yes | Yes | No | Yes | Mammals | | No | Yes | No | Yes | Non-mammals | | Yes | No | No | Yes | Mammals | | Yes | No | Yes | No | Non-mammals | | No | No | Sometimes | Yes | Non-mammals | | No | No | Sometimes | Yes | Non-mammals | | Yes | No | No | Yes | Mammals | | No | No | Yes | No | Non-mammals | | No | No | Sometimes | Yes | Non-mammals | | No | No | No | Yes | Non-mammals | | No | No | No | Yes | Mammals | | No | Yes | No | Yes | Non-mammals | | Yes | No | Yes | No | Mammals | | No | Yes | No | Yes | Non-mammals | Test: {'Give Birth': 'Yes', 'Can Fly': 'No', 'Live in Water': 'Yes', 'Have Legs': 'No'} ### To find Class under categorical Naive Bayes. ### Formula: which one and why $$ \mathrm{score}(c)=P(c)\prod_jP(x_j\mid c) $$ Use empirical categorical frequencies, not a multinomial word model. No likelihood is zero for the requested test, so unsmoothed estimates are sufficient. Each feature contributes once; ignore names/identifiers as predictors. ### Step 1. Count examples in each class | Class | Count | Prior | | --- | --- | --- | | Mammals | 7 | 7/20 | | Non-mammals | 13 | 13/20 | ### Step 2. Count each requested feature within each class and multiply | Class | n | Prior | Give Birth | Can Fly | Live in Water | Have Legs | Exact product | Decimal product | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Mammals | 7 | 7/20 | 6/7 | 6/7 | 2/7 | 2/7 | 36/1715 | 0.020991253644 | | Non-mammals | 13 | 13/20 | 1/13 | 10/13 | 3/13 | 4/13 | 6/2197 | 0.002730996814 | ### Step 3. Normalize for a posterior if wanted | Class | Score / score sum | | --- | --- | | Mammals | 0.884876150 | | Non-mammals | 0.115123850 | ### Final answer **Predicted class: Mammals.** The independence assumption is conditional on class, not a claim that these biological/weather features are truly independent. --- ## TEE · Q3a · Hobbs and ambiguous professor–student reference Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q3a Previous examination · complete worked answers **Question. Explain Hobbs’ algorithm, including binding, gender, distance and recency. Resolve “The professor advised the student because she was struggling.”** [10 marks] ### Answer **Hobbs’ algorithm is a syntax-based procedure that searches constituent parse trees in a fixed order for a compatible noun-phrase antecedent of a pronoun.** The tree determines which candidates are proposed first. Agreement and other relevant constraints determine whether a proposed candidate can be accepted. It is not simply the rule “choose the closest noun”. A nearby noun may have the wrong number, may be syntactically ineligible, or may be visited after a structurally more prominent noun. The algorithm produces a preference under the supplied parse; ambiguous language can still require context. ### The tree vocabulary you need | Term | Meaning | Example | | --- | --- | --- | | NP | Noun phrase: a phrase referring to an entity | the student; Anita; they | | S | A sentence/clause node | the node containing a subject and predicate | | VP | Verb phrase | told Ravi that she should revise | | X | The NP or S node currently being searched | An algorithm label, not a word | | Path p | The branch travelled upward from the pronoun/previous X | Used to distinguish left and right subtrees | | N-bar (N′) | The noun-centred part of an NP in a grammar using that label | The nominal-child test depends on the given tree | | Breadth-first | Visit shallower nodes before deeper ones | Level 1 before level 2 | | Left-to-right | Visit equally deep nodes in their written/tree order | Earlier left branch before right branch | Locate pronoun NP → Climb to NP/S → Propose candidates in tree order → Check compatibility → Accept or continue searching ### The complete search procedure #### Step 1. Start at the pronoun Locate the NP immediately dominating the pronoun. “Dominating” means containing it as a descendant in the tree. #### Step 2. Climb to the first NP or S Move upward to the first NP or S above that starting NP. Call this node X. Keep the upward path p. #### Step 3. Search the left branches initially Below X, traverse branches to the left of p breadth-first and left-to-right. At this initial search, propose an NP only if another NP or S lies between that candidate and X. Test proposed candidates for the required agreement and constraints. #### Step 4. If X is the highest sentence node, inspect earlier sentences Search previous sentence trees from most recent to oldest. Within each, use breadth-first, left-to-right order to propose NPs. Accept the first eligible compatible candidate, or report that no candidate was found in the available context. #### Step 5. Otherwise climb again Move from X to the next NP or S above it. Record this new X and the path followed during the climb. #### Step 6. Apply the higher-NP test If the new X is an NP, propose X when the path did not pass through its nominal N-bar child. Use the given parse labels; do not invent an N-bar branch missing from the question's structure. #### Step 7. Search its left branches Traverse branches below the new X to the left of the path, breadth-first and left-to-right, proposing NPs. The special intervening-NP/S restriction belongs to the initial search in Step 3. #### Step 8. If X is S, search the permitted right branches Search branches to the right of the path breadth-first and left-to-right. Do not descend below an encountered NP or S. Propose eligible NP candidates there. #### Step 9. Repeat or stop Return to the highest-S check and continue the climb/search procedure until a compatible candidate is accepted or the available trees are exhausted. Record the candidate order rather than only writing an unexplained name. ### Why the first left search has an extra condition In a simple clause such as “John saw him”, the subject NP John is directly under the clause S. There is no extra NP or S between John and that S. The initial-search restriction therefore prevents casually selecting John as the ordinary local antecedent of him. Compare the reflexive “John saw himself”, which is a different grammatical case; do not treat every pronoun type as interchangeable. ### Worked tree 1 · move to the previous sentence **Question. Apply Hobbs’ search to “The student arrived. She smiled.” Assume the student is compatible with she.** #### Given Previous tree: [S [NP The student] [VP arrived]]. Current tree: [S [NP She] [VP smiled]]. #### To find The selected antecedent and the search path. #### Method and why it applies Use the supplied constituent trees. The current sentence has no earlier candidate, so the highest-S rule activates previous-sentence search. ### Solution #### Step 1. Draw both sentence structures ``` Previous sentence Current sentence S S ├── NP: The student ├── NP: She ← start └── VP: arrived └── VP: smiled ``` #### Step 2. Locate and climb Start at NP She. Its first higher NP/S is the current S. Set X to that S. #### Step 3. Search left of the path She is the leftmost subject. No NP lies to its left in the current sentence. #### Step 4. Use the highest-S rule This X is the highest S. Inspect the immediately preceding sentence tree. #### Step 5. Propose its first NP Breadth-first, left-to-right finds NP The student. #### Step 6. Check compatibility and accept The question stipulates that the student agrees with she. Accept this candidate. **Answer.** She refers to the student under the supplied assumptions. ### Worked tree 2 · reject an incompatible subject **Question. Resolve they in “Riya spoke to the students. They laughed.” Use a previous-sentence search and interpret they as plural.** #### Given Previous tree has subject NP Riya, then VP spoke [PP to [NP the students]]. Current subject is NP They. #### To find The candidate order and accepted antecedent. #### Method and why it applies Breadth-first search favours a shallow subject, but agreement must still be applied; a first candidate is not automatically a valid answer. ### Solution #### Step 1. Draw the useful part of the previous tree ``` S ├── NP: Riya ← candidate 1 └── VP ├── V: spoke └── PP ├── P: to └── NP: the students ← candidate 2 ``` #### Step 2. Search the current sentence From NP They climb to its S; there is no earlier NP in “They laughed”. This is the highest S, so move to the previous tree. #### Step 3. Visit Riya first Riya is the shallower leftmost NP. Under the stated plural interpretation, this singular candidate fails number agreement. #### Step 4. Visit the students Continue breadth-first. The next relevant NP is the students. #### Step 5. Apply number agreement Students is plural and is an eligible candidate. It matches plural they. #### Step 6. State the resolution Accept the students. The method did not simply accept the first noun it saw. **Answer.** They → the students. Candidate order: Riya rejected; the students accepted. ### Worked tree 3 · climb out of an embedded clause **Question. Resolve she in “Anita told Ravi that she should revise.” Assume Anita is compatible with she and Ravi is not. Use the displayed parse.** #### Given The pronoun is the subject of an embedded S inside the main VP. #### To find The preferred antecedent found by the climb and left-branch search. #### Method and why it applies The embedded clause has no left candidate. Climbing to the main S exposes its subject Anita as the first suitable higher-level candidate. ### Solution #### Step 1. Show the clause structure ``` S (main) ├── NP: Anita ← accepted candidate └── VP ├── V: told ├── NP: Ravi └── SBAR: that └── S (embedded) ← first X ├── NP: she ← start └── VP: should revise ``` #### Step 2. Start inside the embedded clause From NP she climb to the embedded S. There is no candidate to the left of she within this embedded S. #### Step 3. Climb to the next NP/S The embedded S is not the highest S. Move upward through SBAR and VP to the main S. These intermediate nodes are not NP/S search nodes. #### Step 4. Search left of the upward path At the main S, the subject NP Anita is a shallow left candidate. The object Ravi is deeper within the VP. Breadth-first order proposes Anita first. #### Step 5. Check the stated compatibility Anita is compatible with she by the question's assumption. Accept it; no need to choose a later incompatible candidate. #### Step 6. Give the result and its limit This is the algorithm's result under the displayed parse and agreement assumptions. It does not prove the intention of every speaker using an otherwise ambiguous sentence. **Answer.** She → Anita under the stated tree and compatibility information. ### Advantages, limitations and a complete exam answer | Advantages | Limitations | | --- | --- | | Explicit candidate order makes the decision traceable. | Wrong parse structure can produce the wrong search order. | | Requires no labelled coreference-training corpus for its basic rules. | World knowledge and discourse intention may override syntactic preference. | | Uses structure rather than only surface distance. | Some ambiguous, long-distance or unusual constructions need additional treatment. | For a longer answer, write the definition, inputs, tree notation, ordered algorithm, one labelled tree with a candidate table, final antecedent, and limitations. In a worked question show separately: candidate → why proposed → agreement result → accept/reject. Do not equate the selected candidate with a logically guaranteed interpretation when the sentence is ambiguous. **Yaad rakho:** Tree order candidate deta hai; agreement usko allow/reject karta hai. Nearest noun automatically answer nahi. **The intended antecedent is not uniquely determined by the printed sentence.** Both professor and student can be singular female people; neither noun supplies gender. Structural Hobbs search can propose the matrix subject professor early, while the causal meaning of advice because of struggling often supports student. Distance/recency favouring student is a preference, not proof. 1. Parse subject NP the professor, object NP the student, subordinate-clause subject she. 2. Generate structurally accessible earlier NPs through the Hobbs ascent and breadth-first leftward search. 3. Check grammatical number/person and binding constraints. Both candidates can pass here; she is not a reflexive. 4. Gender/world knowledge is unavailable, so do not invent either person’s gender. 5. Use broader discourse to select. Without it, report professor/student ambiguity; explain that contextual interpretation often prefers student but the syntax-only algorithm does not guarantee that intent. **Source / convention note.** Pronoun referent is ambiguous without extra gender/discourse information; structural algorithm preference is separated from semantic interpretation. ### The four requested properties | Property | What it means | Apply here | | --- | --- | --- | | Binding | Syntactic restrictions on reference; ordinary personal pronouns and reflexives have different local-antecedent requirements | Do not mechanically link every nearby NP; he in an after/because clause can refer outside that local clause. | | Gender agreement | Candidate must match the pronoun under available linguistic/contextual knowledge | John/he is the exercise’s conventional reading; professor/student do not provide gender by themselves. | | Distance | Token, syntactic or sentence separation between pronoun and candidate | Closer candidates may be preferred, but cannot override incompatibility or Hobbs search order. | | Recency | Recently mentioned discourse entities are often more salient | Mary/student is later than John/professor, but recency is a preference, not a guaranteed antecedent rule. | --- ## TEE · Q3b · Classification and sentiment in industry Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q3b Previous examination · complete worked answers **Question. Discuss real-world text classification and sentiment analysis, with examples showing decision-making and customer engagement.** [10 marks] ### Answer **Text classification** assigns a document to one or more predefined categories. Spam detection usually uses two classes; news classification may use several topics. In single-label multiclass classification, choose one of several categories. In multilabel classification, several labels may apply to the same document. Define labels → Prepare labelled examples → Train representation + model → Evaluate on unseen text → Predict new labels ### An exam-ready pipeline | Stage | What you do | Why it matters | | --- | --- | --- | | Task definition | Decide what counts as spam or which news topics exist. | Unclear labels create inconsistent training data. | | Data collection and annotation | Collect representative examples and assign correct labels. | The model learns from these examples. | | Split data | Separate training, validation and test data. | Test examples must not influence training choices. | | Preprocess | Tokenize and normalise where suitable; retain meaningful negation. | Reduce irrelevant variation without deleting signal. | | Represent | Use BoW, TF-IDF or learned embeddings. | Models need numerical features. | | Train | Fit Naive Bayes, logistic regression or another appropriate model. | Learn how features predict categories. | | Evaluate | Measure relevant precision, recall and F1 on held-out examples. | Accuracy alone may hide poor minority-class performance. | | Deploy and monitor | Apply the same preprocessing and monitor changing language. | Incoming data can differ from old examples. | Naive Bayes is especially useful for hand calculations. With a bag-of-words representation, its class score is the prior multiplied by the conditional probability of each token; repeated tokens repeat the factor. The “naive” assumption is conditional independence of features given the class, not that words are independent in real language. $$ s(c)=P(c)\prod_{w\in V}P(w\mid c)^{C(w,d)} $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ C(w,d) counts word w in document d. C(w,c) counts it in training documents belonging to class c. Nc is the total token count for class c. Add-one smoothing adds one to every vocabulary count and therefore adds |V| to the denominator. This is the same arithmetic taught for WSD, with document classes instead of senses. ### Worked classification · two-token message A spam classifier has vocabulary {win, prize, meeting}. Priors are P(spam)=P(ham)=0.5. Spam token counts are [3,2,0]; ham counts are [0,0,5]. With add-one smoothing, classify “win prize”. **Given** |V|=3; N(spam)=5; N(ham)=5; message has win once and prize once. **To find** Class scores and predicted label. ### Formula and why it applies $$ s(c)=P(c)P(\text{win}\mid c)P(\text{prize}\mid c) $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ The model is multinomial Naive Bayes; multiply the two token likelihoods and the class prior. ### Solution #### Step 1. Compute the smoothed denominators $$ 5+3=8 $$ #### Step 2. Compute spam likelihoods $$ P(\text{win}\mid S)=\frac{3+1}{8}=\frac12 $$ $$ P(\text{prize}\mid S)=\frac{2+1}{8}=\frac38 $$ #### Step 3. Compute ham likelihoods $$ P(\text{win}\mid H)=P(\text{prize}\mid H)=\frac{0+1}{8}=\frac18 $$ #### Step 4. Multiply the spam score $$ s(S)=\frac12\times\frac12\times\frac38=\frac{3}{32}=0.09375 $$ #### Step 5. Multiply the ham score $$ s(H)=\frac12\times\frac18\times\frac18=\frac{1}{128}=0.0078125 $$ #### Step 6. Compare 0.09375 > 0.0078125. These are unnormalised scores; the label is chosen by the larger score. Predicted label: spam. **Yaad rakho:** Class prior × har word ki class-likelihood; sabse bada score wins. **Sentiment analysis** identifies expressed opinion or attitude, often as positive, negative or neutral. Document-level analysis labels a complete text; sentence-level analysis labels a sentence; aspect-level analysis attaches sentiment to a specific feature or entity. | Review | Useful interpretation | | --- | --- | | “The display is excellent, but the battery is poor.” | Display: positive; battery: negative. One overall label loses this distinction. | | “The service is not good.” | Negative: negation changes good. Removing not can reverse the intended meaning. | | “Great, another two-hour delay.” | Likely sarcasm: literal positive word alone is insufficient. | ### Approaches **Lexicon based:** look up sentiment scores, combine them, and apply stated handling for negation/intensifiers. It is interpretable but fragile with context and sarcasm. **Supervised:** learn a mapping from labelled text features to sentiment labels. **Neural/contextual:** learn or use contextual representations to capture word interactions, then fine-tune a sentiment classifier. None guarantees correct interpretation of every review. ### A specified lexicon calculation Use good = +2, excellent = +3 and slow = −2. A “not” immediately before a sentiment word reverses its score. Score “The support was not good but the delivery was excellent and slow.” **Given** Three sentiment words; not modifies good only; sum word scores under the stated rule. **To find** Total sentiment score and label, taking positive totals as positive. ### Formula and why it applies $$ S=\sum_i s_i $$ Apply the given negation rule before summing. This is a simplified specified model, not a universal linguistic rule. ### Solution #### Step 1. Identify sentiment words good, excellent and slow. #### Step 2. Apply negation $$ s(\text{not good})=-2 $$ #### Step 3. Add the contributions $$ S=-2+3-2=-1 $$ #### Step 4. Map score to label The sum is negative. Total score = −1; negative under this rule. | Industry | Concrete input and output | Useful decision / limitation | | --- | --- | --- | | Retail | Review→topic and sentiment | Route delivery complaints; sarcasm/negation may mislead | | Banking | Customer message→card/fraud/loan intent | Route to the correct team; a model flag is not proof of fraud | | Healthcare | Feedback→service-quality topics | Detect recurring appointment issues; protect patient identifiers | | Media | Article→topic; public responses→sentiment | Organize feeds and monitor reaction; sampling bias prevents treating posts as a census | --- ## TEE · Q4a · Top-down syntax tree Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q4a Previous examination · complete worked answers **Question. Explain the need for syntax parsing; compare top-down and bottom-up parsing. Parse “the cat chased the dog” using S→NP VP, NP→Det N, VP→V NP, Det→the, N→cat|dog, V→chased|saw.** [10 marks] ### Answer **Top-down parsing** starts at the grammar's start symbol S and expands nonterminals until the leaves match the input words in order. **Recursive descent** commonly implements each nonterminal as a procedure that tries its productions. It builds from the root toward the leaves. Start with S → Choose an allowed S production → Expand its next nonterminal → Match the resulting terminal to the input → Continue, or backtrack if the choice fails **Algorithm.** Start with S as the current goal. Choose a production for the leftmost unexpanded nonterminal. Replace that nonterminal by the production's right side. If a terminal appears, compare it with the next input token. If it matches, consume the token; if it does not, reject that branch and try another production. Accept only when the complete input is consumed and all grammar goals are satisfied. A **leftmost derivation** always expands the leftmost remaining nonterminal first. Top-down parsing can make guesses that the actual words cannot support, so it may need backtracking. A naive recursive-descent parser can loop on left-recursive rules such as NP→NP PP unless the grammar or algorithm is changed. Predictive LL(1) parsing chooses productions using one lookahead token when the grammar permits; not every English CFG is LL(1). | Approach | Begins with | Direction | Common difficulty | | --- | --- | --- | --- | | Top-down | Start symbol S | Root to word leaves | Predictions that do not match input; left recursion | | Bottom-up | Actual input tokens | Leaves to root S | Reductions that cannot contribute to a complete sentence | **Yaad rakho:** Top-down: sentence ka structure pehle, words tak neeche aao. **Bottom-up parsing** starts from the input tokens and combines them into larger constituents until the start symbol is reached. A **shift-reduce parser** maintains a stack of recognized material and a list of remaining input tokens. | Action | Exact operation | Example | | --- | --- | --- | | Shift | Move the next input token onto the stack | Stack: the; remaining input: dog runs | | Reduce | Replace a suffix of the stack matching a rule's RHS with its LHS | Det N becomes NP using NP→Det N | | Accept | Stack is S and input is exhausted | Stack: S; input: empty | | Fail/backtrack | Chosen reductions cannot complete a parse | A noun reading of book blocks an imperative parse | Reduction uses a production in reverse: the grammar NP→Det N expands an NP top-down, but reduces Det N into NP bottom-up. The stack's **rightmost item is its top** in the traces below. Each row shows the state **after** the listed action, avoiding an off-by-one mismatch between actions and stack contents. A lexical reduction such as N→dog still counts as a reduction. Do not skip the sequence from word to category to phrase when the question asks for a complete trace. Backtracking means returning to an earlier choice and trying another applicable rule, not changing the original input. **Yaad rakho:** Shift = input se stack. Reduce = chhote pieces se bada grammatical unit. ### Exact top-down derivation 1. S 2. NP VP 3. Det N VP 4. the N VP 5. the cat VP 6. the cat V NP 7. the cat chased NP 8. the cat chased Det N 9. the cat chased the N 10. the cat chased the dog SNPDettheNcatVPVchasedNPDettheNdog [S [NP [Det the] [N cat]] [VP [V chased] [NP [Det the] [N dog]]]] Parsing assigns constituent structure and grammatical relationships for extraction, translation and ambiguity resolution. Top-down expands from S toward input; bottom-up combines observed tokens toward S. This derivation uses only supplied rules. --- ## TEE · Q4b · Data acquisition and four augmentation methods Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q4b Previous examination · complete worked answers **Question. Explain NLP data acquisition, why augmentation is needed, and any four augmentation methods.** [10 marks] ### Answer **Data acquisition** obtains language examples relevant to the task from existing data, public resources or appropriate new collection. Product instrumentation can gather useful feedback. Supervised learning additionally needs an annotation process when labels are absent. **Data augmentation** creates training variants intended to preserve the correct label. It is not simply changing words arbitrarily. | Method | Procedure | Caution | | --- | --- | --- | | Synonym replacement | Use suitable synonyms for selected words | Choose the right sense | | Back translation | Translate to another language and back | May change meanings/entities | | TF–IDF-informed replacement | Protect informative words while varying others | Do not remove label-bearing clues | | Bigram flipping | Swap adjacent words | Can break grammar or meaning | | Entity replacement | Replace by same-type entity | Must preserve the task label | | Noise injection | Simulate typos/keyboard errors | Match expected noise | | Active learning | Select informative cases for human labels | This acquires labels; it is not synthetic text | ### Three label decisions “The film is good”→“The film is excellent” may preserve a positive sentiment label. “The film is not good”→“The film is good” reverses the label and is not valid label-preserving augmentation. “Flights to Pune”→“Flights to Mumbai” may preserve a travel-topic label but changes the destination label. Split train/test data before creating related augmented variants. Otherwise near-duplicates can leak between sets and produce misleading evaluation. Keep a representative evaluation set of genuine examples. Sentence badal sakta hai; required correct label nahi. Label badla toh augmentation invalid. | Method | Example | Required label check | | --- | --- | --- | | Synonym replacement | good film→excellent film | Replacement must preserve sense and sentiment | | Back-translation | English→French→English paraphrase | Check negation, entity and factual preservation | | Contextual paraphrasing | The delivery was late→The package arrived late | Do not change class or named facts | | Controlled noise / perturbation | Small realistic spelling error or punctuation variation | Appropriate only if target should remain unchanged | Acquire licensed/public datasets, commissioned annotation, surveys/user-consented logs or permitted scraping. Augment training data only after splitting to avoid near-duplicate leakage into validation/test. More synthetic examples do not guarantee better generalization. --- ## TEE · Q5a · Feature engineering, BoW and Euclidean Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q5a Previous examination · complete worked answers **Question. Explain text feature engineering and list vectorization approaches. For the three sentences below remove stopwords {is,that,are,to,and}, create vocabulary and BoW, then compute Euclidean distance between sentences1and2: “Natural Language Processing is fascinating”; “I love Natural Language processing and like to study Natural Language”; “I like things that are fascinating”.** [10 marks] ### Answer Feature engineering converts text into informative model inputs. Common approaches: one-hot, BoW, bag of n-grams, TF–IDF and dense embeddings. This question specifically requests count-based BoW, so do not use TF–IDF in its distance. ### Given Use lowercase lexical tokens and the exact printed stoplist; I remains because it is not a stopword here. ### To find Vocabulary, count matrix and distance D1–D2. ### Formula: which one and why $$ d_E(u,v)=\sqrt{\sum_i(u_i-v_i)^2} $$ ### Step 1. Tokenize and remove the exact stoplist | Document | Retained tokens | | --- | --- | | 1 | natural, language, processing, fascinating | | 2 | i, love, natural, language, processing, like, study, natural, language | | 3 | i, like, things, fascinating | ### Step 2. Fix vocabulary and count repeats | Document | fascinating | i | language | like | love | natural | processing | study | things | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | 0 | | 2 | 0 | 1 | 2 | 1 | 1 | 2 | 1 | 1 | 0 | | 3 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | ### Step 3. Subtract coordinates [1, -1, -1, -1, -1, -1, 0, -1, 0] ### Step 4. Square, sum and take the root 1 + 1 + 1 + 1 + 1 + 1 + 0 + 1 + 0 = 7; √7 = 2.645751311 ### Final answer **Euclidean distance=2.645751311.** Natural and Language each occur twice in sentence2; preserve both counts. --- ## TEE · Q5b · WSD approaches and uses Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q5b Previous examination · complete worked answers **Question. Explain WSD, its approaches and applications.** [5 marks] ### Answer **Word sense disambiguation (WSD) is the task of selecting the intended sense of an ambiguous word in its context from a set of possible senses.** The output is a sense label, not a different replacement word. Input: “He grilled the bass.” → Candidate senses: fish / low sound → Context: grilled → Output: fish sense The **target word** is the ambiguous word. The **context** is the surrounding evidence. The **sense inventory** lists candidate meanings, often using WordNet. A **feature** is a usable clue: nearby word, position, part-of-speech tag or a phrase. | Approach | What it needs | How it decides | Limitation | | --- | --- | --- | --- | | Knowledge based | Dictionary, thesaurus or semantic graph | Definition overlap, category evidence or graph scores | Short definitions and missing knowledge | | Supervised | Examples already labelled with senses | Learns context → sense mapping | Labelling cost, limited rare-sense examples | | Semi-supervised | Small labelled set plus unlabelled data | Bootstraps from reliable seed examples | Early errors can spread | | Unsupervised sense induction | Unlabelled contextual occurrences | Clusters similar contexts | Clusters are not automatically named dictionary senses | | Neural/contextual | Learned word/context representations | Predicts or compares a context representation | Needs suitable data/model and evaluation | **Bag-of-words features** record nearby words while ignoring exact positions. **Collocational features** preserve a positional phrase such as “river bank”, “bank account”, “chocolate bar” or “bar exam”. ### Why WSD matters Machine translation needs the correct target-language meaning; information retrieval should distinguish river-bank travel from banking services; text-to-speech may need different pronunciations for “bass”; information extraction should avoid connecting entities through the wrong relation. WSD: same spelling ke options mein context dekh ke sahi meaning select. --- ## TEE · Q5c · Discourse analysis of a conversation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q5c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q5c Previous examination · complete worked answers **Question. Analyze coherence, cohesion, topic structure, information flow and discourse markers: Speaker1 “I went to the store to buy some milk.” Speaker2 “Was it crowded?” Speaker1 “Yes, I had to wait in line for10minutes.” Speaker3 “I was going to attend the meeting, but I got stuck in traffic.”** [5 marks] ### Answer | Feature | Evidence and explanation | | --- | --- | | Coherence | First3turns form a meaningful store/crowding/waiting exchange. The meeting/traffic turn shifts topic without an explicit bridge; local relevance is weaker. | | Cohesion | it refers to the store; Yes answers the preceding question; first-person I shifts referent with each speaker. | | Topic structure | Store visit→crowding→wait; then new topic of meeting attendance and traffic. | | Information flow | Known store is followed by new question crowding, then new evidence10-minute wait. The last turn introduces separate plans and obstacle. | | Discourse markers | Yes signals affirmation; but signals contrast between intention and obstructing traffic. “To buy” states purpose, not merely temporal order. | Cohesion is explicit linguistic linkage; coherence is meaningful conceptual connection. A conversation can contain grammatical connectors and still have a poorly connected topic shift. --- ## TEE · Q6a · Henry bigram continuations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q6a Previous examination · complete worked answers **Question. From the seven supplied training sentences below, predict the most probable next token for Do ..., I like ..., and I ... using bigrams.** [10 marks] ### Answer ### Given I am Henry I like college Do Henry like college Henry I am Do I like Henry Do I like college I do like Henry ### To find Most probable next token for each prefix. ### Formula: which one and why $$ P(w\mid h)=\frac{c(h,w)}{\sum_v c(h,v)} $$ Retain case exactly; include sentence-end as an observed next token. Bigram prediction uses only the last token of the prefix. ### Step 1. Count each adjacent pair separately within sentences | Bigram | Count | | --- | --- | | → Do | 3 | | → Henry | 1 | | → I | 3 | | Do → Henry | 1 | | Do → I | 2 | | Henry → | 3 | | Henry → I | 1 | | Henry → like | 1 | | I → am | 2 | | I → do | 1 | | I → like | 3 | | am → | 1 | | am → Henry | 1 | | college → | 3 | | do → like | 1 | | like → Henry | 2 | | like → college | 3 | ### Step 2. Continue Do | Candidate | Count / history count | Probability | | --- | --- | --- | | Henry | 1/3 | 0.333333 | | I | 2/3 | 0.666667 | Highest: I. ### Step 3. Continue I like | Candidate | Count / history count | Probability | | --- | --- | --- | | Henry | 2/5 | 0.400000 | | college | 3/5 | 0.600000 | Highest: college. ### Step 4. Continue I | Candidate | Count / history count | Probability | | --- | --- | --- | | am | 2/6 | 0.333333 | | do | 1/6 | 0.166667 | | like | 3/6 | 0.500000 | Highest: like. ### Final answer Do → I I like → college I → like If an examiner explicitly restricts the answer to a lexical word, exclude END and state that restricted candidate set. Do not silently remove an END tie. **Source / convention note.** Case is retained as printed: Do and do are distinct. The corpus is not silently grammatically corrected. --- ## TEE · Q6b · Five linguistic levels Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q6b Previous examination · complete worked answers **Question. Explain the contribution of phonology, morphology, syntax, semantics and pragmatics to language understanding.** [5 marks] ### Answer **A phoneme** is a sound unit that can distinguish meanings in a language, such as the /p/ and /b/ contrast in “pat” and “bat”. A phoneme is a sound, not necessarily one alphabet letter. **A morpheme** is the smallest meaning-bearing grammatical unit: cats = cat + plural -s; unhappy = un- + happy. **A lexeme** is an abstract vocabulary item grouping its inflected forms: write, writes, wrote and writing belong to WRITE. A **phrase** is a group of words functioning together; a **clause** normally contains a predicate and its associated participants; a **sentence** expresses a complete grammatical unit. | Level | What it analyses | Concrete example | | --- | --- | --- | | Phonetics / phonology | Physical sounds / sound patterns in a language | Distinguish bat from pat | | Morphology / lexical analysis | Word structure and lexical items | un + help + ful → unhelpful | | Syntax | Grammatical arrangement and relationships | The dog chased the cat: subject, verb, object | | Semantics | Meaning of words and composed expressions | The dog is the chaser; the cat is chased | | Discourse | Connections across sentences | “Riya arrived. She smiled.” → She refers to Riya | | Pragmatics | Intended meaning in the situation | “It is cold here” may request closing the window | word forms → grammatical structure → sentence meaning → cross-sentence context → speaker intention ### Three worked interpretations **“The cats are sleeping.”** Morphology identifies cat + s as plural. Syntax connects “cats” to “are sleeping”. Semantics represents animals engaged in sleeping. **“Aman bought a laptop. It was expensive.”** Sentence-level syntax alone does not identify “it”. Discourse connects “it” to the laptop across the sentence boundary. **“Wonderful, another power cut!”** The word wonderful is positive literally, but situational context can indicate sarcasm and frustration. This is a pragmatic interpretation. The analysis levels are conceptual descriptions. Modern systems may learn several jointly rather than executing a compulsory separate module for every level. Morphology = word ke andar; syntax = sentence ka structure; semantics = meaning; discourse = sentences ka link; pragmatics = intention. --- ## TEE · Q6c · NLU and NLG working together Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q6c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q6c Previous examination · complete worked answers **Question. Explain how NLU and NLG work together for meaningful human–machine communication.** [5 marks] ### Answer **Natural language understanding (NLU)** maps language into an interpretation useful for a task: intents, entities, semantic relations or another meaning representation. **Natural language generation (NLG)** produces human-language output from data, a representation or another source. “Book a ticket to Pune tomorrow.” → NLU: intent=book; destination=Pune; date=tomorrow → Application: search available journeys → NLG: “Here are the available journeys to Pune.” | Feature | NLU | NLG | | --- | --- | --- | | Direction | Language → interpretation | Data/representation → language | | Main concern | Meaning, intent, ambiguity | Content choice, organization, fluent wording | | Example | Find destination in a request | Write the booking confirmation | | Common failure | Wrong destination or sense | Unclear, unsupported or ungrammatical output | ### NLG stages in a classical pipeline 1. **Content determination:** choose facts to communicate. 2. **Data interpretation:** identify relationships or trends. 3. **Document planning:** organize facts in a useful order. 4. **Sentence aggregation:** combine related facts without repetition. 5. **Lexical and grammatical realization:** choose words and construct grammatical sentences. 6. **Presentation:** format the generated message for its audience. For a weather report, facts {city=Pune, temperature=28°C, rain=yes} can become “Pune will have rain with a temperature of 28°C.” NLU would instead extract facts or intent from an incoming language message. NLU = language se meaning andar. NLG = data se language bahar. --- ## TEE · Q7a · POS, chunks and boundary-aware HMM Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q7a Previous examination · complete worked answers **Question. Define function/content words, chunking/chinking and POS applications. Construct transition and emission tables including start/end for: Book/Verb a/Det flight/Noun; Read/Verb the/Det book/Noun; The/Det book/Noun is/Modal-Verb green/Noun.** [10 marks] ### Answer **Part-of-speech (POS) tagging** assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence. | Tag | Role | Example in context | | --- | --- | --- | | NN / NNS | Singular / plural common noun | a book / two books | | NNP | Proper noun | Mary, Mumbai | | VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads | | JJ | Adjective | the red book | | RB | Adverb | runs quickly | | DT | Determiner | the, a, this | | IN | Preposition or subordinating conjunction | on the table | | PRP | Personal pronoun | she, they | | MD | Modal auxiliary | can, will, should | **Open classes** readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. **Closed classes** contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes. Input: Book that flight → Context: an imperative request → Output: Book/VB that/DT flight/NN Compare “hand me that **book**”: book is a noun here. Similarly, “I **can** swim” has can/MD, while “the **can** is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is **POS ambiguity**. | Sentence | Tagged interpretation | Reason | | --- | --- | --- | | They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. | | The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. | | She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. | | Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. | POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning. **Yaad rakho:** Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai. **Chunking**, also called shallow parsing, groups adjacent tagged words into meaningful non-overlapping units, commonly noun phrases. It usually identifies local phrase boundaries without producing the complete hierarchical structure of a sentence. **Chinking** removes specified tokens from an existing candidate chunk, splitting or trimming it. The/DT small/JJ cat/NN sat/VBD on/IN the/DT mat/NN → NP: [The small cat] → Outside: sat on → NP: [the mat] A POS tagger labels individual words. A chunker groups words into phrases. A full parser shows how phrases relate recursively to form the entire sentence. These are related tasks with different outputs. | Operation | Input | Output | | --- | --- | --- | | POS tagging | The small cat | The/DT small/JJ cat/NN | | NP chunking | Tagged tokens | [NP The small cat] | | Chinking | A broad candidate chunk | Remove verbs/prepositions, keep noun-phrase segments | | Full parsing | Sentence + grammar | A nested tree with sentence, noun phrase and verb phrase nodes | **Rule-based chunking** uses tag patterns. Statistical or learned chunkers infer boundaries from labeled examples. A useful NP rule is an optional determiner, followed by zero or more adjectives, followed by a noun. In NLTK-style notation: `NP: {
?*}`. Here ? means zero or one occurrence; \* means zero or more; + means one or more when used. The braces mean “form a chunk around the matched sequence”. **Question. Apply the NP rule optional DT, zero or more JJ, then one NN to “The/DT quick/JJ brown/JJ fox/NN jumps/VBZ over/IN the/DT lazy/JJ dog/NN”.** #### Given Tagged input is supplied. Rule: DT? JJ\* NN. #### To find All noun-phrase chunks. #### Formula and why it applies `NP → DT? JJ* NN` The rule groups a noun with its preceding determiner and adjectives; a verb or preposition does not match this NP pattern. #### Solution ##### Step 1. Start at The The matches optional DT; quick and brown both match JJ\*; fox matches NN. ##### Step 2. Form the first NP **[NP The quick brown fox]** ##### Step 3. Skip nonmatching tokens jumps/VBZ and over/IN do not satisfy the NP pattern. ##### Step 4. Match the second NP the is DT, lazy is JJ, dog is NN: **[NP the lazy dog]**. **Final answer.** [NP The quick brown fox] jumps over [NP the lazy dog]. **Question. Using NP → DT? JJ\* NN, identify the chunks in “bright/JJ stars/NN shine/VBP and/CC a/DT moon/NN glows/VBZ”. Use the supplied tags exactly.** #### Given Input tags label stars as NN for this exercise. Rule requires optional DT, zero or more JJ, then NN. #### To find The matched NP spans. #### Formula and why it applies `NP → DT? JJ* NN` The optional determiner may be absent and the adjective repetition may be empty. #### Solution ##### Step 1. Match bright stars No determiner is present, which DT? permits. bright matches JJ; stars matches NN. ##### Step 2. Inspect shine and and VBP and CC fall outside the NP rule. ##### Step 3. Match a moon a matches DT; no adjective is present; moon matches NN. **Final answer.** Chunks are [NP bright stars] and [NP a moon]. With ordinary Penn tagging, stars is usually NNS; broaden the noun pattern to include NNS when that tag is used. **Question. Begin with one candidate chunk containing “The/DT cat/NN sat/VBD on/IN the/DT mat/NN”. Chink out verbs and prepositions. Show the remaining chunks.** #### Given Initial candidate: [The cat sat on the mat]. Remove tags VBD and IN. #### To find The chunk segments after exclusion. #### Formula and why it applies Chinking removes matched tokens from the existing chunk. Removing the middle verb/preposition region divides the original candidate into two remaining contiguous spans. #### Solution ##### Step 1. Locate the tokens to remove sat is VBD and on is IN. ##### Step 2. Remove those tokens from chunk membership The remaining token groups are “The cat” and “the mat”. sat and on still exist in the sentence; they are simply outside these chunks. ##### Step 3. Write the final chunked sentence **[The cat]** sat on **[the mat]**. **Final answer.** Two chunks remain: The cat; the mat. **Yaad rakho:** Chunking = andar group karo; chinking = unwanted tokens ko group se bahar karo. ### Exact HMM table ### Given | Sentence | Supplied tags | | --- | --- | | Book a flight | Verb Det Noun | | Read the book | Verb Det Noun | | The book is green | Det Noun Modal-Verb Noun | ### To find Transition and emission probability tables including START and END. ### Formula: which one and why $$ a_{ij}=\frac{c(t_i,t_j)}{c(t_i)},\qquad b_i(w)=\frac{c(t_i,w)}{c(t_i)} $$ For START divide by sentence count. END has no outgoing or emission distribution. Keep the supplied tags even when their linguistic choice is unusual, and retain case-sensitive Book/book and The/the as printed. ### Step 1. Add boundary tags to every sentence START Verb Det Noun END START Verb Det Noun END START Det Noun Modal-Verb Noun END ### Step 2. Count transitions | From → to | Count | | --- | --- | | START → Verb | 2 | | Verb → Det | 2 | | Det → Noun | 3 | | Noun → END | 3 | | START → Det | 1 | | Noun → Modal-Verb | 1 | | Modal-Verb → Noun | 1 | ### Step 3. Divide by each source tag count | From | Denominator | Verb | Det | Noun | Modal-Verb | END | | --- | --- | --- | --- | --- | --- | --- | | START | 3 | 2/3 | 1/3 | 0 | 0 | 0 | | Verb | 2 | 0 | 1 | 0 | 0 | 0 | | Det | 3 | 0 | 0 | 1 | 0 | 0 | | Noun | 4 | 0 | 0 | 0 | 1/4 | 3/4 | | Modal-Verb | 1 | 0 | 0 | 1 | 0 | 0 | ### Step 4. Count and normalize emissions | Tag | Count | Book | a | flight | Read | the | book | The | is | green | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Verb | 2 | 1/2 | 0 | 0 | 1/2 | 0 | 0 | 0 | 0 | 0 | | Det | 3 | 0 | 1/3 | 0 | 0 | 1/3 | 0 | 1/3 | 0 | 0 | | Noun | 4 | 0 | 0 | 1/4 | 0 | 0 | 1/2 | 0 | 0 | 1/4 | | Modal-Verb | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | ### Final answer Every nonterminal transition row sums to1 when END is included; every emission row sums to1. Boundary symbols do not emit lexical words. POS applications include syntax parsing, disambiguation, lemmatization, chunking and extraction. For this numerical retain printed green=Noun and is=Modal-Verb even though ordinary linguistic tagging would differ; the task is estimation from supplied labels. **Source / convention note.** Printed tags are used as training data; START and END are explicitly required and included. --- ## TEE · Q7b · Co-occurrence, dot products and plot Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f24/q7b Previous examination · complete worked answers **Question. Define target words, context window, context words and co-occurrence matrix. For the printed boy/ball/dog/sun corpus below, use one whole sentence as the window; contexts={kicked,field,tree,ran}. Find boy·dog and boy·sun, plot on kicked/field dimensions and comment.** [10 marks] ### Answer **Target** is the word whose row we represent. **Context** is an observed neighbouring word whose column we count. A **window** defines which tokens count as neighbours; here all other tokens in the same sentence. A **co-occurrence matrix** records those counts. ### Given The boy kicked the ball into the field. The dog barked at the boy. The sun was shining over the field. The boy ran after the dog. The ball rolled to the tree. ### To find Matrix, two dot products and the requested 2D coordinates. ### Formula: which one and why $$ u\cdot v=\sum_i u_iv_i $$ Use lowercase token-pair counts within each sentence; do not cross sentence boundaries. ### Step 1. List contributions and build the matrix | Sentence | Target occurrences | Context occurrences | | --- | --- | --- | | 1 | boy, ball | kicked, field | | 2 | dog, boy | | | 3 | sun | field | | 4 | boy, dog | ran | | 5 | ball | tree | | Target / context | kicked | field | tree | ran | | --- | --- | --- | --- | --- | | boy | 1 | 1 | 0 | 1 | | ball | 1 | 1 | 1 | 0 | | dog | 0 | 0 | 0 | 1 | | sun | 0 | 1 | 0 | 0 | ### Step 2. Dot boy and dog [1, 1, 0, 1] · [0, 0, 0, 1] = 1×0 + 1×0 + 0×0 + 1×1 = 1 ### Step 3. Dot boy and sun [1, 1, 0, 1] · [0, 1, 0, 0] = 1×0 + 1×1 + 0×0 + 1×0 = 1 ### Step 4. Plot coordinates: kicked on x, field on y | Target | (x,y) | | --- | --- | | boy | (1, 1) | | ball | (1, 1) | | dog | (0, 0) | | sun | (0, 1) | kickedfielddog (0,0)sun (0,1)boy, ball (1,1) Two coordinates omit tree and ran, so boy and ball overlap here despite different full vectors. ### Final answer **boy·dog=1; boy·sun=1.** Both share one counted context contribution. Dot-product similarity reflects this corpus and magnitude, not a claim that boy is equally semantically related to dog and sun in general. --- ## TEE · Q1a · Anaphora versus cataphora Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q1a Previous examination · complete worked answers **Question. Compare anaphoric and cataphoric reference with examples.** [5 marks] ### Answer **Pragmatic analysis** interprets an utterance using its situation and the speaker’s intended meaning. **Discourse analysis** studies how utterances connect across a text or conversation. Syntax and literal word meanings alone cannot settle every interpretation. | Text | Literal content | Contextual interpretation | | --- | --- | --- | | “Can you close the window?” | A question about ability. | Usually a polite request. | | “Meera bought a laptop. It was expensive.” | Two sentences. | “It” connects the second sentence to the laptop. | | “Meet me here tomorrow.” | An instruction. | Me, here and tomorrow depend on speaker, place and date. | ### Reference, mention and antecedent A **referring expression** points to an entity; a **mention** is its occurrence in the text. Its **referent** is the entity being discussed. An **antecedent** is a preceding expression used to interpret a later referring expression. Reference resolution identifies the intended referent. Coreference resolution groups mentions that refer to the same entity. “Meera”name mention → Person entitysame referent → “she” / “the engineer”other mentions | Concept | Definition | Example | | --- | --- | --- | | Anaphora | Reference back to something introduced earlier. | “Riya arrived. She sat down.” She → Riya. | | Cataphora | Reference forward to a later expression. | “Before she spoke, Riya checked the microphone.” she → Riya. | | Exophora | Reference to the situation outside the text. | “Put it there”, said while pointing. | | Coreference | Two or more mentions refer to the same entity. | “Dr Rao”, “the lecturer”, “she”. | | Pronoun resolution | Find the referent of a pronoun. | Resolve “it” in “The laptop stopped because it overheated.” | Coreference is broader than pronoun resolution: repeated names and descriptions also form a chain. Anaphora describes direction/dependence; coreference describes identity. In “I bought a car. The engine is noisy”, engine is related to the car but is not the same entity. **Yaad rakho:** Reference = kiski baat? Coreference = alag mentions, same entity. Pronoun resolution = he/she/it kis se linked? ### Cohesion, coherence and discourse markers **Cohesion** is the explicit linguistic linking of a text through reference, repeated or related words, conjunctions and other devices. In “Rita bought a book. She read it”, she and it connect the second sentence to Rita and the book. **Coherence** is the conceptual unity that makes a discourse meaningful. “The road flooded, so the match was cancelled” presents a plausible cause and consequence. A reader relates the propositions using context and knowledge. **Discourse markers** signal relationships or organize speech: however signals contrast; therefore signals a result; first and next signal sequence; for example introduces an illustration. These concepts differ: “I like apples. Therefore the moon is square” has a linking word but no sensible inference. A connector alone does not guarantee coherence. **Yaad rakho:** Cohesion = words se link; coherence = meaning se link; marker = relation ka signal. --- ## TEE · Q1b · Five corpora Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q1b Previous examination · complete worked answers **Question. Define corpora and discuss any five corpora used in NLP.** [5 marks] ### Answer **A corpus is a collection of language data assembled for study or computational use.** Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. **Annotation** adds linguistic information such as POS tags, entities, parse structures or sense labels. | Structure | Meaning | Use | | --- | --- | --- | | Plain | Collection without category labels | Word counts, language modelling | | Categorized | Documents grouped by genre/topic | Classification | | Overlapping categories | One document has several categories | Multi-topic news | | Temporal | Documents indexed by time | Language change | | Annotated | Linguistic labels attached | Tagging, parsing, WSD | | Parallel | Aligned texts in different languages | Translation | A **tagset** is the allowed label inventory. A **treebank** is a corpus annotated with syntax trees. **Metadata** records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions. **Raw example:** “Riya works in Pune.” **POS annotation:** Riya/PROPN works/VERB in/ADP Pune/PROPN. **NER annotation:** Riya/PERSON, Pune/LOCATION. Each annotation answers a different task. A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use. Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information. | Resource | Contents | Typical use | | --- | --- | --- | | Gutenberg selection | Literary/book texts | Long-form language and vocabulary | | Web text | Informal domain-specific text | Study usage beyond edited books | | Chat corpus | Conversational posts | Dialogue, informal forms | | Brown | English texts across genres | Genre comparison | | Reuters | News with topic labels | Text classification; overlapping topics | | Inaugural | Speeches indexed by time | Vocabulary change across years | | Wordlist | Words/names/stopwords | Lexical filtering | | Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme | **Gutenberg versus web/chat:** book language does not represent every spelling or conversational pattern in messages. **Brown versus Reuters:** Brown emphasizes genres; Reuters news can have overlapping topic labels. **Inaugural addresses:** dates provide a time dimension. Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property. Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time. --- ## TEE · Q1c · Linguistic foundations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q1c Previous examination · complete worked answers **Question. Define linguistic foundation and explain its importance in NLP.** [5 marks] ### Answer **A phoneme** is a sound unit that can distinguish meanings in a language, such as the /p/ and /b/ contrast in “pat” and “bat”. A phoneme is a sound, not necessarily one alphabet letter. **A morpheme** is the smallest meaning-bearing grammatical unit: cats = cat + plural -s; unhappy = un- + happy. **A lexeme** is an abstract vocabulary item grouping its inflected forms: write, writes, wrote and writing belong to WRITE. A **phrase** is a group of words functioning together; a **clause** normally contains a predicate and its associated participants; a **sentence** expresses a complete grammatical unit. | Level | What it analyses | Concrete example | | --- | --- | --- | | Phonetics / phonology | Physical sounds / sound patterns in a language | Distinguish bat from pat | | Morphology / lexical analysis | Word structure and lexical items | un + help + ful → unhelpful | | Syntax | Grammatical arrangement and relationships | The dog chased the cat: subject, verb, object | | Semantics | Meaning of words and composed expressions | The dog is the chaser; the cat is chased | | Discourse | Connections across sentences | “Riya arrived. She smiled.” → She refers to Riya | | Pragmatics | Intended meaning in the situation | “It is cold here” may request closing the window | word forms → grammatical structure → sentence meaning → cross-sentence context → speaker intention ### Three worked interpretations **“The cats are sleeping.”** Morphology identifies cat + s as plural. Syntax connects “cats” to “are sleeping”. Semantics represents animals engaged in sleeping. **“Aman bought a laptop. It was expensive.”** Sentence-level syntax alone does not identify “it”. Discourse connects “it” to the laptop across the sentence boundary. **“Wonderful, another power cut!”** The word wonderful is positive literally, but situational context can indicate sarcasm and frustration. This is a pragmatic interpretation. The analysis levels are conceptual descriptions. Modern systems may learn several jointly rather than executing a compulsory separate module for every level. Morphology = word ke andar; syntax = sentence ka structure; semantics = meaning; discourse = sentences ka link; pragmatics = intention. --- ## TEE · Q1d · Five regex operators Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q1d Previous examination · complete worked answers **Question. For word-pattern detection, explain: ^abc, abc$, [A-Z0-9], ed|ing|s, [abc].** [5 marks] ### Answer | Pattern | Exact meaning | Positive example | Important distinction | | --- | --- | --- | --- | | ^abc | abc at start of the input/line under chosen mode | abcdef | Not abc anywhere | | abc$ | abc at end | xxabc | Not starts with abc | | [A-Z0-9] | One uppercase ASCII letter or digit | Q or 7 | One character, not the whole string | | ed|ing|s | One of the alternatives ed, ing or s | matched ed | Without anchors this is not necessarily a suffix | | [abc] | One a, b or c | b | Not the three-character string abc | Use full-string matching when the entire string must satisfy a pattern; a search may find only a substring. --- ## TEE · Q2a · Wordlist puzzle and lexical resource Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q2a Previous examination · complete worked answers **Question. Explain lexical resources. Solve the wordlist puzzle with letters E,G,I / V,R,V / O,N,L. Words must have at least four letters, contain centre R, use each tile at most once, and include at least one nine-letter word.** [10 marks] ### Answer ### Given | | Column1 | Column2 | Column3 | | --- | --- | --- | --- | | Row1 | E | G | I | | Row2 | V | R | V | | Row3 | O | N | L | ### To find Find valid words and explain how a wordlist corpus supplies the candidates. ### Formula: which one and why A lexical resource stores words and linguistic information. Filter an explicitly chosen dictionary by length, mandatory letter and a multiset of tile counts; duplicate V tiles permit two Vs. ### Step 1. Count available tiles E1,G1,I1,V2,R1,O1,N1,L1. ### Step 2. Apply all predicates For each lowercase dictionary word w: length(w)≥4; r occurs; for every letter c, count\_w(c)≤count\_tiles(c). Deduplicate dictionary entries. ### Step 3. Verify examples | Word | Reason | | --- | --- | | revolving | 9 letters: r e v o l v i n g; uses both V tiles and all others once | | roving | 6 letters, includes R, within tile counts | | lover | 5 letters, includes R | | giver | 5 letters, includes R | | groin | 5 letters, includes R | | iron | 4 letters, includes R | | ring | 4 letters, includes R | | rove | 4 letters, includes R | ### Step 4. Explain the requested count The count is the number of entries surviving those checks in the chosen wordlist. No wordlist/version is supplied, so a unique total cannot be stated. The examples above are verified examples, not an exhaustive dictionary. ### Final answer **REVOLVING** is a valid nine-letter answer. Enumerate all other valid entries using the declared wordlist. **Source / convention note.** The dictionary is unspecified; an exact number of valid words would be fabricated. --- ## TEE · Q2b · NLP applications and limitations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q2b Previous examination · complete worked answers **Question. List NLP applications and explain limitations of NLP.** [10 marks] ### Answer **Natural language processing (NLP) is the field concerned with computational methods for analysing, interpreting and generating human language.** Natural languages such as English, Hindi and Marathi evolved through human use; their meanings are not specified by the rigid rules of a programming language. An NLP system may receive text, speech converted to text, or another linguistic representation. Its output depends on the task: a category, a tag for every word, a translated sentence, a ranked document list or a generated answer. It does not always “understand” in the human sense; its performance must be evaluated for the specific task. | Task | Input | Output | | --- | --- | --- | | Sentiment analysis | “The service was excellent.” | Positive sentiment | | Text classification | An email | Spam / not spam | | POS tagging | “Birds fly.” | Birds/NOUN fly/VERB | | Named entity recognition | “Asha joined Google in Mumbai.” | Asha: person; Google: organisation; Mumbai: location | | Machine translation | A Hindi sentence | A corresponding English sentence | | Information retrieval | A search query | Ranked relevant documents | | Question answering | Question plus knowledge/context | An answer | | Summarization | A long article | A shorter text preserving key information | ### How NLP relates to AI, ML and DL **AI** is the broader study of systems performing tasks associated with intelligent behaviour. **Machine learning** learns patterns from data. **Deep learning** uses multiple trainable neural layers. NLP is an application/research field that can use hand-written rules, statistical ML or deep learning. Therefore, not every NLP system is a neural network. Linguistics supplies knowledge about language structure. Statistics supplies uncertainty models. Computer science supplies algorithms, storage and efficient implementation. A practical NLP system combines these ideas. NLP = human language ko computational task mein badlo: label, meaning, search ya generated sentence. **Ambiguity** means an expression supports more than one interpretation. Good exam answers name the ambiguity and explain both readings. | Challenge | Example | Why a system can fail | | --- | --- | --- | | Lexical/sense ambiguity | bank: finance or river edge | One written word has several senses | | Syntactic ambiguity | I saw the man with a telescope. | The telescope may modify saw or the man | | Reference ambiguity | Riya told Neha that she won. | She may refer to either person without more context | | Sarcasm | Great, another delay. | Literal positive word conveys frustration | | Negation | The film was not good. | Dropping not reverses meaning | | Domain shift | “Positive” in a medical report | Meaning/use differs by domain | | Code mixing | Kal meeting reschedule karna. | Several languages and spelling conventions | | Noisy text | tmrw, repeated letters, typos | Vocabulary/token boundaries become uncertain | | Low-resource setting | Few labelled examples in a language | Models lack representative training evidence | | Bias and privacy | Skewed data or personal details | Errors and harms can persist in deployed systems | **Homophones** sound alike but may differ in spelling (pair/pear). **Homographs** share spelling but can differ in meaning or pronunciation (lead the team / lead metal). These are different dimensions from syntactic ambiguity. Long-distance dependencies, cultural knowledge, idioms, morphology and multilingual writing systems also complicate processing. For “kick the bucket”, literal word-by-word interpretation can miss the idiomatic meaning. For “The book that the students recommended is useful”, the verb is agrees with book across an intervening phrase. Exam mein challenge ka naam + ek sentence + dono meanings/actual failure likho. --- ## TEE · Q3a · Top-down restaurant parse Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q3a Previous examination · complete worked answers **Question. Draw a top-down parse of “A restaurant serves dosas” using the printed grammar: S→NP VP | Aux NP VP | VP; NP→Pronoun | Proper-Noun | Det Nominal; Nominal→Noun | Nominal Noun | Nominal PP; VP→Verb | Verb NP | VP PP; PP→Prep NP. Lexicon: Det=A; Noun=restaurant,dosas; Verb=serves.** [10 marks] ### Answer **As printed, this grammar cannot derive the complete sentence.** The object dosas is a Noun, but NP has no rule for a bare Nominal/Noun. NP→Det Nominal would require a determiner before dosas that is absent. ### Clearly stated minimal repair Add NP→Nominal. The repaired top-down derivation is: | Step | Sentential form | | --- | --- | | 1 | S | | 2 | NP VP | | 3 | Det Nominal VP | | 4 | A Noun VP | | 5 | A restaurant Verb NP | | 6 | A restaurant serves Nominal | | 7 | A restaurant serves Noun | | 8 | A restaurant serves dosas | SNPDetANominalNounrestaurantVPVerbservesNPNominalNoundosas [S [NP [Det A] [Nominal [Noun restaurant]]] [VP [Verb serves] [NP [Nominal [Noun dosas]]]]] This tree uses the added rule only for the object. Do not silently pretend it is derivable under the original rule set. Left-recursive Nominal/VP alternatives require a terminating parser strategy; the displayed successful derivation chooses the needed finite branches. **Source / convention note.** Original CFG is incomplete for the given bare-plural object. Exact failure and explicit repaired derivation are both shown. --- ## TEE · Q3b · Preprocessing pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q3b Previous examination · complete worked answers **Question. Explain basic preprocessing steps on a blob of text with an example and pipeline diagram.** [10 marks] ### Answer **An NLP pipeline is an organized sequence of operations that transforms raw language into a task output and checks whether that output is useful.** Not every task needs every cleaning operation. Data acquisition → Preprocessing → Feature representation → Model → Evaluation → Deployment and monitoring 1. **Data acquisition:** collect relevant text and, when needed, labels. 2. **Cleaning and preprocessing:** handle markup, tokenize, normalize and decide which information to retain. 3. **Feature engineering:** represent text numerically using BoW, TF–IDF or embeddings. 4. **Modelling:** apply rules or train a statistical/neural model. 5. **Evaluation:** test on held-out data using task-appropriate measures. 6. **Deployment:** make the system available, log suitable feedback and monitor errors or data changes. ### Concrete spam-filter example Collect labelled emails → separate train/test data → tokenize text while preserving meaningful terms → fit vocabulary and IDF on training data → train classifier → evaluate spam detection on the held-out set → deploy an email service filter → monitor legitimate emails wrongly blocked. **Intrinsic evaluation** measures model/task quality, such as precision, recall or tagging accuracy. **Extrinsic evaluation** measures effect in the wider application, such as whether users find answers faster. A higher internal metric does not automatically guarantee a better user outcome. **Data leakage:** learning vocabulary statistics, labels or preprocessing decisions from the test set can make reported performance too optimistic. Fit learned transformations on training data, then apply them to validation/test data. Collect → clean → numbers → model → test → use. Test data ko training ki information mat banao. **Normalization reduces superficial variation so comparable forms can be processed consistently.** Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization. **Stopwords** are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us. **Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.** #### Given Operations and stopword set are fixed. Do not remove not. #### To find The final token list. #### Formula and why it applies Apply the requested operations in order, keeping a visible intermediate result after each transformation. #### Step 1. Lowercase the cats are not sleeping!!! #### Step 2. Tokenize and remove punctuation [the,cats,are,not,sleeping] #### Step 3. Remove specified stopwords Remove the and are → [cats,not,sleeping]. #### Step 4. Lemmatize cats→cat; sleeping→sleep → [cat,not,sleep]. **Answer.** [cat,not,sleep]. The negative meaning is retained. ### Task decides preprocessing For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them. Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi. **Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma. | Criterion | Stemming | Lemmatization | | --- | --- | --- | | Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis | | Output | May be nonword, e.g. studi | Dictionary form, e.g. study | | Context/POS | Usually little or none | Often important | | Irregular forms | May miss went→go | Can map went→go with verb analysis | | Cost | Generally simpler | Usually more linguistic processing | Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation. ### Three worked reductions 1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked. 2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation. 3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter. **Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice. Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan. --- ## TEE · Q4a · Embeddings and Skip-gram Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q4a Previous examination · complete worked answers **Question. Explain word embeddings and the steps in Skip-gram with a suitable diagram.** [10 marks] ### Answer **A sparse representation** has many zero entries and often one coordinate per vocabulary item. BoW and TF–IDF are common sparse document representations. **A dense embedding** has comparatively few learned real-valued coordinates; one word's properties are spread across many coordinates. | Representation | What coordinates mean | Strength | Limitation | | --- | --- | --- | --- | | One-hot | One identity position per vocabulary word | Unambiguous ID | Different words are orthogonal; no learned similarity | | Count-based context vector | Context-word co-occurrence counts | Transparent corpus evidence | Large, sparse; frequent words may dominate | | TF–IDF document vector | Weighted vocabulary terms | Useful retrieval weighting | Word order and contextual meaning are limited | | Dense word embedding | Learned latent features | Compact semantic/syntactic relationships | Coordinates are not generally human-labelled features | **Distributional** describes learning from patterns of linguistic context. **Distributed** describes representing an item across multiple coordinates rather than a single active identity bit. These are related but different ideas. A one-hot ID alone does not learn distributional similarity. ### Model design choices Choose contexts (document, sentence, fixed window), weighting (raw counts, normalization, TF–IDF-like weighting or association scores), dimensionality reduction or learned embeddings, and a similarity measure. Cosine compares direction and is less affected by length than a raw dot product; Euclidean distance also reflects magnitude. Sparse = bahut zero. Dense = chhoti learned number-list. Similar context se useful similarity seekhte hain. **Word2Vec learns dense word embeddings by predicting words from nearby context.** Its two common architectures are CBOW and skip-gram. A vocabulary ID or one-hot vector is the input representation; the learned embedding table converts that identity into a short trainable vector. | Architecture | Input | Prediction | Memory cue | | --- | --- | --- | --- | | CBOW | Surrounding context words | Centre/target word | Context → centre | | Skip-gram | Centre/target word | Surrounding words | Centre → context | CBOW: the + sat → Average context embeddings → Predict: cat Skip-gram: cat → Embedding of cat → Predict: the and sat For vocabulary size $$ V $$ and embedding dimension $$ d $$ , an input one-hot row is $$ 1\times V $$ ; the input embedding matrix is $$ V\times d $$ ; the hidden row is $$ 1\times d $$ ; the output matrix is $$ d\times V $$ ; output scores are $$ 1\times V $$ . Multiplying a one-hot row by the embedding matrix simply selects one row. $$ P(w_o\mid w_i)=\frac{\exp(u_{w_o}^{T}v_{w_i})}{\sum_{w\in V}\exp(u_w^{T}v_{w_i})}. $$ $$ v_{w_i} $$ is the input word embedding, $$ u_w $$ an output-context embedding, and the dot product is a compatibility score. Softmax turns all scores into probabilities that sum to one. Training increases the probability of observed context words. Negative sampling or hierarchical softmax can reduce computation in larger vocabularies; use the algorithm specified by the question. **Question. For the sentence “the cat sat on mat”, generate CBOW and skip-gram training data for centre word sat with a window of one token on each side.** #### Given Tokens indexed 1 to 5: the, cat, sat, on, mat. Centre = sat. #### To find The CBOW example and the skip-gram pairs. #### Formula and why it applies CBOW predicts the centre from its neighbours; skip-gram predicts each neighbour from the centre. #### Step 1. Identify neighbours Left neighbour = cat; right neighbour = on. “the” and “mat” are outside this window. #### Step 2. CBOW example Input [cat,on] → target sat. #### Step 3. Skip-gram examples (sat,cat) and (sat,on), with first item as input and second as predicted context. **Answer.** CBOW: [cat,on]→sat. Skip-gram: sat→cat and sat→on. **Question. The embeddings of context words cat and on are [1,2] and [3,0]. Compute the CBOW hidden vector using mean aggregation. Output vectors for [sat,ran] are [1,0] and [0,1]. Compute softmax probabilities.** #### Given Context embeddings [1,2], [3,0]; mean aggregation; two candidate output vectors. #### To find The hidden vector, logits and probabilities. #### Formula and why it applies Mean context vector $$ h=(v_1+v_2)/2 $$ . Logit $$ z_w=u_w^Th $$ . Softmax $$ P(w)=e^{z_w}/\sum_j e^{z_j} $$ . #### Step 1. Add coordinates $$ [1+3,\,2+0]=[4,2] $$ . #### Step 2. Divide by number of words $$ h=[4/2,2/2]=[2,1] $$ . #### Step 3. Compute logits $$ z_{sat}=1(2)+0(1)=2 $$ . $$ z_{ran}=0(2)+1(1)=1 $$ . #### Step 4. Exponentials $$ e^2\approx7.389056 $$ , $$ e^1\approx2.718282 $$ ; sum $$ \approx10.107338 $$ . #### Step 5. Divide $$ P(sat)\approx0.731059 $$ , $$ P(ran)\approx0.268941 $$ . **Answer.** h=[2,1]; sat is more probable (0.731059). **Question. A skip-gram centre embedding is v=[1,−1]. Output embeddings for [eat,sleep,drive] are [1,1], [0,1], [1,0]. Calculate the predicted context probabilities.** #### Given One centre vector and three output vectors; full softmax over these three words. #### To find Three logits and three probabilities. #### Formula and why it applies $$ z_j=u_j^Tv $$ , then $$ P(j)=e^{z_j}/\sum_k e^{z_k} $$ . #### Step 1. Eat dot product $$ 1(1)+1(-1)=0 $$ . #### Step 2. Sleep and drive dot products $$ z_{sleep}=0(1)+1(-1)=-1 $$ . $$ z_{drive}=1(1)+0(-1)=1 $$ . #### Step 3. Exponentiate and sum $$ e^0=1 $$ , $$ e^{-1}\approx0.367879 $$ , $$ e^1\approx2.718282 $$ . Sum $$ \approx4.086161 $$ . #### Step 4. Normalize $$ P(eat)\approx0.244728 $$ ; $$ P(sleep)\approx0.090031 $$ ; $$ P(drive)\approx0.665241 $$ . **Answer.** drive has the highest probability in this toy learned model. **Limitations:** a standard static Word2Vec vector merges a word's senses; it has limited representation of subword morphology and out-of-vocabulary words; its local prediction objective does not directly fit a full global co-occurrence table. Similarity is learned from data and is not a guarantee of factual or causal understanding. CBOW = neighbours se missing centre. Skip-gram = centre se neighbours. --- ## TEE · Q4b · POS, NER, chunking and TTR Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q4b Previous examination · complete worked answers **Question. Define POS tagging, entity recognition, chunking and type-token ratio with examples.** [10 marks] ### Answer **Part-of-speech (POS) tagging** assigns a grammatical category to every word token in context, such as noun, verb, adjective or determiner. It is a sequence-labeling task: the input is a sequence of words, and the output is a tag for each word. A tag describes the word's role in this sentence, not its meaning in every possible sentence. | Tag | Role | Example in context | | --- | --- | --- | | NN / NNS | Singular / plural common noun | a book / two books | | NNP | Proper noun | Mary, Mumbai | | VB / VBD / VBZ | Base / past / third-person singular verb | read / read / reads | | JJ | Adjective | the red book | | RB | Adverb | runs quickly | | DT | Determiner | the, a, this | | IN | Preposition or subordinating conjunction | on the table | | PRP | Personal pronoun | she, they | | MD | Modal auxiliary | can, will, should | **Open classes** readily accept new content words: nouns, verbs, adjectives and adverbs. Examples include chatbot and livestream. **Closed classes** contain comparatively stable grammatical/function words: determiners, pronouns, prepositions and conjunctions. These are tendencies, not a claim that language never changes. Input: Book that flight → Context: an imperative request → Output: Book/VB that/DT flight/NN Compare “hand me that **book**”: book is a noun here. Similarly, “I **can** swim” has can/MD, while “the **can** is empty” has can/NN. The spelling stays the same; neighboring words and sentence structure change the correct tag. This is **POS ambiguity**. | Sentence | Tagged interpretation | Reason | | --- | --- | --- | | They watch birds. | They/PRP watch/VBP birds/NNS | watch is the action. | | The watch stopped. | The/DT watch/NN stopped/VBD | watch is the object that stopped. | | She has a light bag. | She/PRP has/VBZ a/DT light/JJ bag/NN | light describes bag. | | Light the lamp. | Light/VB the/DT lamp/NN | Light is an imperative verb. | POS tags support parsing, named entity recognition, question answering, sentiment analysis and word-sense disambiguation. They do not by themselves identify every entity or fully resolve meaning. **Yaad rakho:** Word ka naam same ho sakta hai; sentence mein uska kaam tag decide karta hai. **Named Entity Recognition (NER)** identifies text spans referring to entities and assigns their types, such as PERSON, ORGANIZATION and LOCATION. Many practical schemas also include DATE, TIME, MONEY or domain-specific types. The exact allowed labels depend on the application's annotation scheme. NER has two parts: **boundary detection** finds where the entity starts and ends; **classification** assigns the appropriate type. A multiword name should normally be one entity span: “New York” is a location span, not two unrelated location decisions. Text: Mary joined Google in New York on Monday. → Mary → PERSON → Google → ORGANIZATION → New York → LOCATION → Monday → DATE Not every noun is a named entity. In “a developer works at Google”, developer is an ordinary job description while Google is a named organization. A specialized HR system may define a JOB\_TITLE category, but it must state that schema explicitly. NER identifies mentions; entity linking is the separate task of connecting a mention to a unique database identity. | Task | Question it answers | Example | | --- | --- | --- | | POS | What grammatical role does this token have? | Google/NNP | | Chunking | Which adjacent words form a phrase? | [NP a skilled developer] | | NER | Which span names an entity, and of what type? | [Google] ORGANIZATION | | Entity linking | Which real entity does this mention denote? | Connect Paris to the intended city or person record | **Applications.** NER structures search results and documents, extracts organizations/people from news, routes support tickets by product or organization, supports recommendation, and extracts specialized entities from healthcare or legal text. Context is essential: “Apple released a laptop” names an organization, while “she ate an apple” describes fruit. **Yaad rakho:** NER mein do kaam: naam kahan se kahan tak, aur kis category ka. **Chunking**, also called shallow parsing, groups adjacent tagged words into meaningful non-overlapping units, commonly noun phrases. It usually identifies local phrase boundaries without producing the complete hierarchical structure of a sentence. **Chinking** removes specified tokens from an existing candidate chunk, splitting or trimming it. The/DT small/JJ cat/NN sat/VBD on/IN the/DT mat/NN → NP: [The small cat] → Outside: sat on → NP: [the mat] A POS tagger labels individual words. A chunker groups words into phrases. A full parser shows how phrases relate recursively to form the entire sentence. These are related tasks with different outputs. | Operation | Input | Output | | --- | --- | --- | | POS tagging | The small cat | The/DT small/JJ cat/NN | | NP chunking | Tagged tokens | [NP The small cat] | | Chinking | A broad candidate chunk | Remove verbs/prepositions, keep noun-phrase segments | | Full parsing | Sentence + grammar | A nested tree with sentence, noun phrase and verb phrase nodes | **Rule-based chunking** uses tag patterns. Statistical or learned chunkers infer boundaries from labeled examples. A useful NP rule is an optional determiner, followed by zero or more adjectives, followed by a noun. In NLTK-style notation: `NP: {
?*}`. Here ? means zero or one occurrence; \* means zero or more; + means one or more when used. The braces mean “form a chunk around the matched sequence”. **Question. Apply the NP rule optional DT, zero or more JJ, then one NN to “The/DT quick/JJ brown/JJ fox/NN jumps/VBZ over/IN the/DT lazy/JJ dog/NN”.** #### Given Tagged input is supplied. Rule: DT? JJ\* NN. #### To find All noun-phrase chunks. #### Formula and why it applies `NP → DT? JJ* NN` The rule groups a noun with its preceding determiner and adjectives; a verb or preposition does not match this NP pattern. #### Solution ##### Step 1. Start at The The matches optional DT; quick and brown both match JJ\*; fox matches NN. ##### Step 2. Form the first NP **[NP The quick brown fox]** ##### Step 3. Skip nonmatching tokens jumps/VBZ and over/IN do not satisfy the NP pattern. ##### Step 4. Match the second NP the is DT, lazy is JJ, dog is NN: **[NP the lazy dog]**. **Final answer.** [NP The quick brown fox] jumps over [NP the lazy dog]. **Question. Using NP → DT? JJ\* NN, identify the chunks in “bright/JJ stars/NN shine/VBP and/CC a/DT moon/NN glows/VBZ”. Use the supplied tags exactly.** #### Given Input tags label stars as NN for this exercise. Rule requires optional DT, zero or more JJ, then NN. #### To find The matched NP spans. #### Formula and why it applies `NP → DT? JJ* NN` The optional determiner may be absent and the adjective repetition may be empty. #### Solution ##### Step 1. Match bright stars No determiner is present, which DT? permits. bright matches JJ; stars matches NN. ##### Step 2. Inspect shine and and VBP and CC fall outside the NP rule. ##### Step 3. Match a moon a matches DT; no adjective is present; moon matches NN. **Final answer.** Chunks are [NP bright stars] and [NP a moon]. With ordinary Penn tagging, stars is usually NNS; broaden the noun pattern to include NNS when that tag is used. **Question. Begin with one candidate chunk containing “The/DT cat/NN sat/VBD on/IN the/DT mat/NN”. Chink out verbs and prepositions. Show the remaining chunks.** #### Given Initial candidate: [The cat sat on the mat]. Remove tags VBD and IN. #### To find The chunk segments after exclusion. #### Formula and why it applies Chinking removes matched tokens from the existing chunk. Removing the middle verb/preposition region divides the original candidate into two remaining contiguous spans. #### Solution ##### Step 1. Locate the tokens to remove sat is VBD and on is IN. ##### Step 2. Remove those tokens from chunk membership The remaining token groups are “The cat” and “the mat”. sat and on still exist in the sentence; they are simply outside these chunks. ##### Step 3. Write the final chunked sentence **[The cat]** sat on **[the mat]**. **Final answer.** Two chunks remain: The cat; the mat. **Yaad rakho:** Chunking = andar group karo; chinking = unwanted tokens ko group se bahar karo. **Type-token ratio** = number of distinct tokens / total tokens. In “cats chase cats”, types={cats,chase}=2 and tokens=3, so TTR=2/3. This is an authored illustration of the definition, not extra paper data. --- ## TEE · Q5a · Lexical and vector semantics Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q5a Previous examination · complete worked answers **Question. Explain lexical semantics and vector semantics with examples.** [10 marks] ### Answer **A lexical relation connects words or senses by a meaning relationship.** State the relation and its direction before giving the example. | Relation | Exam definition | Examples and direction | | --- | --- | --- | | Homonymy | The same form has unrelated meanings. | bank: financial institution / river edge; bat: animal / sports implement | | Polysemy | One lexical item has multiple related senses. | head: body part / head of a department; paper: material / article | | Synonymy | Different words have identical or closely similar meanings in a specified sense. | sofa / couch; automobile / car | | Antonymy | Words have opposite meanings. | hot / cold; alive / dead; buy / sell | | Hyponymy | A more specific concept is a type of a more general one. | rose is a flower; car is a vehicle | | Hypernymy | A broader concept includes a more specific one. | flower is a hypernym of rose | | Meronymy | A concept is a part of another concept. | wheel is part of car; finger is part of hand | | Holonymy | A concept is the whole containing a part. | car is a holonym of wheel | ### How to distinguish the commonly confused pairs For homonymy versus polysemy, ask whether the senses share an identifiable meaning. A river bank and a financial bank are unrelated. A financial bank and a blood bank share a repository/storage idea, illustrating polysemy in the supplied notes. For hyponymy versus meronymy, use the sentence test: **“X is a kind of Y”** indicates hyponymy; **“X is part of Y”** indicates meronymy. A wheel is not a type of car. | Expression | Correct relation | Reason | | --- | --- | --- | | sparrow → bird | Hyponymy | A sparrow is a kind of bird | | bird → sparrow | Hypernymy | Bird is the broader category | | keyboard → computer | Meronymy | Keyboard is a component | | computer → keyboard | Holonymy | Computer is the containing whole | | mouth of a person / mouth of a river | Polysemy | Both senses share the idea of an opening; the river use is a related extension | Synonyms are not always interchangeable in every context or register. “Big mistake” and “large mistake” can sound different even though big and large overlap in meaning. Antonyms can be gradable (hot/cold), complementary (alive/dead) or relational converses (buy/sell). HYPO = specific type; HYPER = bigger category. MERO = part; HOLO = whole. **The distributional hypothesis states that words occurring in similar contexts tend to have similar meanings or semantic behaviour.** “Cat” and “dog” often occur near pet, feed and animal. Their context patterns can therefore be represented and compared numerically. A **vector** is an ordered list of numbers. Each position must have a fixed meaning. For context order [pet, food, road], cat=[4,3,0] means four co-occurrences with pet, three with food, none with road. A **matrix** stacks vectors as rows in a table. A **word–context co-occurrence matrix** records how often a target word occurs with each context word under a specified context definition. A window may be a sentence, document, or a fixed number of neighbouring tokens. State its width, direction and whether targets count themselves. Choose target words → Choose context/window → Count co-occurrences → Store each row as a vector → Compare vectors **Question. Build the symmetric co-occurrence matrix for “cats eat fish” using a window of one token on either side, no self-counts and vocabulary [cats,eat,fish].** #### Given One sentence, one occurrence of each token. Left and right adjacent tokens are contexts. #### To find A 3×3 target-by-context count matrix. #### Formula and why it applies $$ X_{ij} $$ counts the number of times context word $$ j $$ occurs within the chosen window of target $$ i $$ . #### Step 1. Target cats Only eat is adjacent. Row cats = [0,1,0]. #### Step 2. Target eat cats is left and fish is right. Row eat = [1,0,1]. #### Step 3. Target fish Only eat is adjacent. Row fish = [0,1,0]. #### Step 4. Assemble the matrix | Target / context | cats | eat | fish | | --- | --- | --- | --- | | cats | 0 | 1 | 0 | | eat | 1 | 0 | 1 | | fish | 0 | 1 | 0 | **Answer.** Rows are [0,1,0], [1,0,1], [0,1,0]. The symmetry follows from the symmetric window. **Question. Build a target–context matrix for sentences “cats eat fish” and “dogs eat fish”. Target words are cats and dogs; context columns [eat,fish]; use the entire sentence excluding the target.** #### Given Each sentence contains one target. Count context token occurrences. #### To find Two context vectors. #### Formula and why it applies Count each context word in every sentence containing the target. #### Step 1. Count cats contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 2. Count dogs contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 3. Compare Both targets have identical observed context patterns in this very small corpus. **Answer.** cats=[1,1], dogs=[1,1]. This corpus supplies no context evidence to distinguish them. **Question. For “a b a”, use a right-only window of one token. Build the matrix with rows/columns [a,b].** #### Given Count directed target→next-word events, not both directions. #### To find The co-occurrence matrix. #### Formula and why it applies $$ X_{ij}=C(\text{target }i\text{ immediately followed by }j) $$ . #### Step 1. First target a Next token is b → add 1 to X(a,b). #### Step 2. Target b Next token is a → add 1 to X(b,a). #### Step 3. Final a No following token → no addition. #### Step 4. Write matrix | Target / next | a | b | | --- | --- | --- | | a | 0 | 1 | | b | 1 | 0 | **Answer.** a=[0,1], b=[1,0]. A different context window would produce a different matrix. Word ka meaning: uske aas-paas kaunse words kitni baar aaye, uski number-list. --- ## TEE · Q5b · Classification label structures Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q5b Previous examination · complete worked answers **Question. Compare binary, multiclass and multilabel supervised text classification with examples.** [10 marks] ### Answer | Type | Allowed output | Example | Typical output rule | | --- | --- | --- | --- | | Binary | One of two labels | positive OR negative review | One sigmoid or two-class softmax | | Multiclass | Exactly one of several labels | sport OR business OR science | Softmax over mutually exclusive classes | | Multilabel | Zero, one or several labels | Article tagged technology AND finance | Independent sigmoid per label | Multiclass and multilabel differ in whether labels may coexist, not merely in the number of labels. A loss and evaluation measure must match the task. --- ## TEE · Q6a · TF–IDF in document representation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q6a Previous examination · complete worked answers **Question. Explain TF–IDF in document representation and information retrieval with an example.** [10 marks] ### Answer **TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus.** The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency: $$ \operatorname{TF}(t,d)=\frac{C(t,d)}{N_d} $$ $$ \operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)} $$ $$ \operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t). $$ $$ N_d $$ is the document's token count after the specified preprocessing; $$ D $$ is number of documents; $$ \operatorname{df}(t) $$ is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency. **Why the log?** It compresses the ratio's scale. If a term occurs in all documents, $$ D/df=1 $$ , so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default. **Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.** #### Given D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10. #### To find The TF–IDF weight. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ , then multiply. #### Step 1. TF $$ 3/12=0.25 $$ . #### Step 2. Document ratio $$ 10/2=5 $$ . #### Step 3. IDF $$ \log_{10}5\approx0.698970 $$ . #### Step 4. Weight $$ 0.25\times0.698970\approx0.174743 $$ . **Answer.** TF=0.25, IDF≈0.698970, TF–IDF≈0.174743. **Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.** #### Given C=4, Nd=20, D=8, df=8. #### To find Its weight under unsmoothed base-10 IDF. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ . #### Step 1. TF $$ 4/20=0.2 $$ . #### Step 2. IDF $$ \log_{10}(8/8)=\log_{10}1=0 $$ . #### Step 3. Multiply $$ 0.2\times0=0 $$ . **Answer.** Weight 0: this term does not distinguish documents under this convention. **Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.** #### Given No further preprocessing. Each term occurs in exactly two of the three documents. #### To find All three vectors. #### Formula and why it applies First count df by documents, then compute each document's TF, then multiply by its term's IDF. #### Step 1. Common IDF $$ D=3 $$ , $$ df(cat)=df(dog)=df(fish)=2 $$ . Each IDF $$ =\log_{10}(3/2)\approx0.176091 $$ . #### Step 2. D1 term frequencies Length 3 → $$ [2/3,1/3,0] $$ . #### Step 3. D1 weights $$ [(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0] $$ . #### Step 4. D2 weights Length 2; TF $$ [0,1/2,1/2] $$ . Vector $$ \approx[0,0.088046,0.088046] $$ . #### Step 5. D3 weights Length 3; TF $$ [1/3,0,2/3] $$ . Vector $$ \approx[0.058697,0,0.117394] $$ . **Answer.** D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394]. TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply. **Information retrieval (IR)** locates and ranks items relevant to an information need, usually expressed as a query. The collection may contain documents, product descriptions or scientific papers. Retrieval estimates relevance; it does not itself establish that every retrieved claim is true. Documents → cleaning → Build index → Query → same representation → Match & rank → Return relevant results ### The inverted index An inverted index maps a term to the documents containing it, often with positions or counts. For D1 “cats drink milk” and D2 “dogs drink water”, the posting list for drink is {D1,D2}, while milk maps only to D1. This avoids rescanning every document for every query. | Term | Posting list | | --- | --- | | cats | D1 | | drink | D1, D2 | | milk | D1 | | dogs | D2 | | water | D2 | ### Matching and ranking Boolean retrieval combines term membership using AND, OR and NOT. Vector-space retrieval represents documents and queries as weighted vectors, for example with TF-IDF, and compares them with cosine similarity. Learned retrieval can use dense embeddings for meaning-based matching. The representation used at query time must be compatible with the document index. ### Search engines and recommendation A search engine receives an explicit query and ranks matching indexed pages. A recommender may derive an implicit query from a user’s liked items or profile and retrieve similar candidates. Both require candidate retrieval and ranking, but the source of the information need differs. ### Evaluate the results For retrieval, TP is the number of retrieved relevant documents, FP retrieved irrelevant documents, and FN relevant documents not retrieved. Precision and recall use the same formulas as classification. Ranking measures also consider the positions of relevant items; a relevant result at rank 1 is more useful than one far below. ### Retrieval evaluation A search system retrieves 8 documents; 6 are relevant. There are 10 relevant documents in the collection. Find precision, recall and F1. **Given** Retrieved=8; relevant retrieved=6; all relevant=10. **To find** P, R and F1. ### Formula and why it applies $$ P=\frac{\text{relevant retrieved}}{\text{retrieved}},\quad R=\frac{\text{relevant retrieved}}{\text{all relevant}} $$ $$ F_1=\frac{2PR}{P+R} $$ The denominator is 8 for precision and 10 for recall because they ask different questions. ### Solution #### Step 1. Identify the confusion counts TP=6; FP=8−6=2; FN=10−6=4. #### Step 2. Precision $$ P=\frac68=0.75 $$ #### Step 3. Recall $$ R=\frac6{10}=0.6 $$ #### Step 4. F1 $$ F_1=\frac{2(0.75)(0.6)}{0.75+0.6}=\frac{0.9}{1.35}=\frac23\approx0.666667 $$ Precision=0.75, recall=0.60, F1≈0.666667. **Yaad rakho:** Precision: jo laaye, unmein kitne sahi? Recall: jo sahi the, unmein kitne laaye? **Source / convention note.** The prompt requests an example but supplies no corpus. Numbers in the explanation are authored illustrations, not original numerical givens. --- ## TEE · Q6b · HMM tables and path Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q6b Previous examination · complete worked answers **Question. For the four tagged sentences below construct transition and emission tables (3 marks each), then calculate Geeta/N will/M spot/V Rita/N using initial P(N)=0.5, P(M)=0.25, P(V)=0.25.** [10 marks] ### Answer ### Given | Sentence | Given tags | | --- | --- | | Rita can teach Will | N M V N | | Spot will see Will | N M V N | | Will Rita spot Geeta | N N V N | | Geeta will pat Spot | N M V N | Required path: Geeta will spot Rita / N M V N. Given initial probabilities: {'N': Fraction(1, 2), 'M': Fraction(1, 4), 'V': Fraction(1, 4)} ### To find Emission and transition tables and the specified tag-path joint probability ### Formula: which one and why $$ P(w\mid t)=\frac{c(t,w)}{c(t)} $$ $$ P(t_j\mid t_i)=\frac{c(t_i,t_j)}{\sum_kc(t_i,t_k)} $$ The second formula is the declared no-END convention. Reset at sentence boundaries; never create a transition from the last word of one training sentence to the first of the next. ### Step 1. Count emissions | Tag | Total occurrences | Rita | can | teach | Will | Spot | will | see | spot | Geeta | pat | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | N | 9 | 2 | 0 | 0 | 3 | 2 | 0 | 0 | 0 | 2 | 0 | | M | 3 | 0 | 1 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | | V | 4 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | ### Step 2. Divide each emission by its tag count | Tag | Rita | can | teach | Will | Spot | will | see | spot | Geeta | pat | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | N | 2/9 | 0 | 0 | 1/3 | 2/9 | 0 | 0 | 0 | 2/9 | 0 | | M | 0 | 1/3 | 0 | 0 | 0 | 2/3 | 0 | 0 | 0 | 0 | | V | 0 | 0 | 1/4 | 0 | 0 | 0 | 1/4 | 1/4 | 0 | 1/4 | ### Step 3. Count transitions inside each sentence | From | Outgoing transitions | Sentence-final count | N | M | V | | --- | --- | --- | --- | --- | --- | | N | 5 | 4 | 1 | 3 | 1 | | M | 3 | 0 | 0 | 0 | 3 | | V | 4 | 0 | 4 | 0 | 0 | ### Step 4. Convert to a transition table | From / to | N | M | V | | --- | --- | --- | --- | | N | 1/5 | 3/5 | 1/5 | | M | 0 | 0 | 1 | | V | 1 | 0 | 0 | Here sentence-final tags are excluded from the no-END outgoing denominator. If END is explicitly modeled, divide all tag→tag and tag→END counts by the total tag count instead; the row then includes END. Do not mix denominators. ### Step 5. Multiply the given path, in order | Factor | Probability | Running product | | --- | --- | --- | | Initial N | 1/2 | 1 × 1/2 = 1/2 | | Emit Geeta | 2/9 | 1/2 × 2/9 = 1/9 | | Transition N→M | 3/5 | 1/9 × 3/5 = 1/15 | | Emit will | 2/3 | 1/15 × 2/3 = 2/45 | | Transition M→V | 1 | 2/45 × 1 = 2/45 | | Emit spot | 1/4 | 2/45 × 1/4 = 1/90 | | Transition V→N | 1 | 1/90 × 1 = 1/90 | | Emit Rita | 2/9 | 1/90 × 2/9 = 1/405 | ### Final answer The two displayed probability tables are the estimated HMM. Zero means unobserved without smoothing, not logically impossible. **No-END path joint probability = 1/405 = 0.00246913580247.** If a normalized END-state model is explicitly adopted instead, this complete terminated sequence has probability 4/6561 = 0.000609663161103. These are different model conventions, not contradictory answers. --- ## TEE · Q7a · Healthcare and finance applications Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q7a Previous examination · complete worked answers **Question. Explain NLP use cases in healthcare and finance.** [10 marks] ### Answer | Domain / task | Input → method → useful output | Example and limit | | --- | --- | --- | | Healthcare: clinical extraction | Notes → entity/negation/relation extraction → diagnoses, medicines and doses | Distinguish no fever from fever; clinician review is needed | | Healthcare: coding and search | Clinical text → classify/retrieve → candidate billing codes or evidence | Coding suggestions are not autonomous diagnoses | | Healthcare: patient messaging | Patient question → intent + retrieval → approved explanation | Escalate uncertainty and emergencies; protect identifiers | | Finance: sentiment/event analysis | News/filings → entities/events/sentiment → issuer-specific signals | Distinguish rumor from confirmed earnings; not guaranteed returns | | Finance: document extraction | Reports/contracts → fields/relations → searchable structured records | Keep source spans and verify amounts/dates | | Finance: customer support/compliance triage | Messages → intent/classification → routed cases or flagged text | Human investigation is required; a language flag is not proof of wrongdoing | Acquire permitted text → Clean and preserve context → Extract / classify / retrieve → Evaluate on domain data → Human-reviewed use --- ## TEE · Q7b · Naive Bayes sentiment Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r24/q7b Previous examination · complete worked answers **Question. Train Naive Bayes on: positive “amaze wonderful great”; “happy happy great”; “amaze great happy disgust”; “great happy wonderful”. Negative “disgust sad terrible great”; “sad terrible sad happy”. Predict sentence 7: “great happy disgust great”.** [10 marks] ### Answer ### Given | Training text | Class | | --- | --- | | amaze wonderful great | positive | | happy happy great | positive | | amaze great happy disgust | positive | | great happy wonderful | positive | | disgust sad terrible great | negative | | sad terrible sad happy | negative | Test: great happy disgust great ### To find Classify the test using multinomial Naive Bayes. ### Formula: which one and why Assume add-one smoothing unless the question supplies probabilities. Keep repeated words as repeated factors. Use training vocabulary only; ignore out-of-vocabulary words unless an unknown-token model is defined. $$ \mathrm{score}(c)=P(c)\prod_{w\in V}\left(\frac{c(w,c)+1}{N_c+|V|}\right)^{c(w,d)} $$ ### Step 1. Vocabulary and class priors V = {amaze, disgust, great, happy, sad, terrible, wonderful}; |V|=7. | Class | Documents | Prior | | --- | --- | --- | | negative | 2 | 1/3 | | positive | 4 | 2/3 | ### Step 2. Count every training word within each class | Class | Total tokens | amaze | disgust | great | happy | sad | terrible | wonderful | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | negative | 8 | 0 | 1 | 1 | 1 | 3 | 2 | 0 | | positive | 13 | 2 | 1 | 4 | 4 | 0 | 0 | 2 | ### Step 3. Count the test words | Word | Count | | --- | --- | | great | 2 | | happy | 1 | | disgust | 1 | OOV words: none ### Step 4. Calculate likelihoods and multiply with the prior | Class | Prior | Individual factors | Exact score | Decimal score | | --- | --- | --- | --- | --- | | negative | 1/3 | great: (1+1)/(8+1×7) = 2/15, used 2 time(s); happy: (1+1)/(8+1×7) = 2/15, used 1 time(s); disgust: (1+1)/(8+1×7) = 2/15, used 1 time(s) | 16/151875 | 0.000105349794239 | | positive | 2/3 | great: (4+1)/(13+1×7) = 1/4, used 2 time(s); happy: (4+1)/(13+1×7) = 1/4, used 1 time(s); disgust: (1+1)/(13+1×7) = 1/10, used 1 time(s) | 1/960 | 0.00104166666667 | ### Step 5. Compare and optionally normalize Normalized scores: negative = 0.091846802, positive = 0.908153198 Division by the same total does not change the winning class. ### Final answer **Predicted class: positive.** **Yaad rakho:** Count → prior → smoothed word chances → repeated factors → compare. --- ## TEE · Q1a · WSD and pronoun resolution Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q1a Previous examination · complete worked answers **Question. Define Word Sense Disambiguation and Pronoun Resolution. Give an example of each.** [5 marks] ### Answer **Word sense disambiguation (WSD) is the task of selecting the intended sense of an ambiguous word in its context from a set of possible senses.** The output is a sense label, not a different replacement word. Input: “He grilled the bass.” → Candidate senses: fish / low sound → Context: grilled → Output: fish sense The **target word** is the ambiguous word. The **context** is the surrounding evidence. The **sense inventory** lists candidate meanings, often using WordNet. A **feature** is a usable clue: nearby word, position, part-of-speech tag or a phrase. | Approach | What it needs | How it decides | Limitation | | --- | --- | --- | --- | | Knowledge based | Dictionary, thesaurus or semantic graph | Definition overlap, category evidence or graph scores | Short definitions and missing knowledge | | Supervised | Examples already labelled with senses | Learns context → sense mapping | Labelling cost, limited rare-sense examples | | Semi-supervised | Small labelled set plus unlabelled data | Bootstraps from reliable seed examples | Early errors can spread | | Unsupervised sense induction | Unlabelled contextual occurrences | Clusters similar contexts | Clusters are not automatically named dictionary senses | | Neural/contextual | Learned word/context representations | Predicts or compares a context representation | Needs suitable data/model and evaluation | **Bag-of-words features** record nearby words while ignoring exact positions. **Collocational features** preserve a positional phrase such as “river bank”, “bank account”, “chocolate bar” or “bar exam”. ### Why WSD matters Machine translation needs the correct target-language meaning; information retrieval should distinguish river-bank travel from banking services; text-to-speech may need different pronunciations for “bass”; information extraction should avoid connecting entities through the wrong relation. WSD: same spelling ke options mein context dekh ke sahi meaning select. **Pronoun resolution** identifies the antecedent referred to by a pronoun. In “Rita arrived. She smiled”, she refers to Rita. WSD chooses a word meaning (bank as finance or river edge); pronoun resolution chooses a discourse entity. --- ## TEE · Q1b · WordNet and its hierarchy Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q1b Previous examination · complete worked answers **Question. Define WordNet. Describe the WordNet hierarchy.** [5 marks] ### Answer **WordNet is a lexical database that groups word senses into synonym sets called synsets and links them through semantic and lexical relations.** It is a network of meanings, not merely a list of similar spellings. A synset represents one concept. Its **lemmas** are the words that can express that concept. A **gloss** is a short definition, often accompanied by an example. An ambiguous word can occur in several synsets. vehicle (general) → car / automobile (same concept) → sports car (specific) This is a simplified teaching hierarchy, not a claim about every intermediate WordNet node. Going upward reaches hypernyms; going downward reaches hyponyms. Multiple parents can exist, so the structure need not be a single simple tree. | Operation | Meaning | Illustration | | --- | --- | --- | | Get senses of a word | Find candidate synsets | bank → financial sense, river-edge sense, … | | Get lemmas | Find synonyms within one synset | car, automobile, motorcar | | Get definition/examples | Read the gloss and uses | Choose the intended concept | | Get hypernyms | Move to broader categories | car → vehicle | | Get hyponyms | Move to narrower categories | vehicle → particular vehicle categories | | Get meronyms/holonyms | Find part/whole links | car ↔ wheel | ### WordNet versus a thesaurus A thesaurus mainly groups related expressions for vocabulary lookup. WordNet explicitly distinguishes senses and labels relations such as hypernymy, hyponymy and meronymy. This supports WSD, query expansion, lexical similarity and semantic search. ### Three uses explained 1. **Search expansion:** a query for “automobile” can retrieve relevant documents using “car”. Expansion should use the right sense. 2. **WSD:** definitions of the senses of “bank” provide words that can be compared with the sentence context. 3. **Similarity:** concepts near one another in an “is-a” hierarchy tend to be semantically related. Path-based measures depend on which senses were selected. WordNet mein spelling ka group nahi—meaning ka synset, aur meanings ke beech labelled links. --- ## TEE · Q1c · Sentiment feature extraction Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q1c Previous examination · complete worked answers **Question. Extract x₁ = log₁₀(word count), x₂ = count of positive lexicon words and x₃ = count of negative lexicon words from: “I was amazed at how the product quickly it arrived. The customer service was exceptional. The support team responded promptly and professionally. I was not sad with my decision. Overall, my experience with them was outstanding.” Positive lexicon={amazed,promptly,outstanding,great}; negative={horrible,sad,disgusting}.** [5 marks] ### Answer ### Given I was amazed at how the product quickly it arrived. The customer service was exceptional. The support team responded promptly and professionally. I was not sad with my decision. Overall, my experience with them was outstanding. ### To find The three numerical features. ### Formula: which one and why Use word tokens after stripping punctuation, keep stopwords, and match only the supplied lexicons. Raw lexicon counting does not automatically reverse a word under negation. ### Step 1. List and count every word | Position | Word | | --- | --- | | 1 | i | | 2 | was | | 3 | amazed | | 4 | at | | 5 | how | | 6 | the | | 7 | product | | 8 | quickly | | 9 | it | | 10 | arrived | | 11 | the | | 12 | customer | | 13 | service | | 14 | was | | 15 | exceptional | | 16 | the | | 17 | support | | 18 | team | | 19 | responded | | 20 | promptly | | 21 | and | | 22 | professionally | | 23 | i | | 24 | was | | 25 | not | | 26 | sad | | 27 | with | | 28 | my | | 29 | decision | | 30 | overall | | 31 | my | | 32 | experience | | 33 | with | | 34 | them | | 35 | was | | 36 | outstanding | ### Step 2. Apply the logarithm $$ x_1=\log_{10}36=1.556302501 $$ ### Step 3. Count positive matches amazed, promptly, outstanding each occur once. great does not occur. x₂=3. exceptional is not in the supplied lexicon, so it is not counted. ### Step 4. Count negative matches sad occurs once; horrible and disgusting do not. x₃=1. The phrase “not sad” remains one raw negative-lexicon occurrence under the specified feature definition. ### Final answer **(x₁,x₂,x₃)=(1.556302501,3,1).** --- ## TEE · Q1d · Binary, multiclass and multilabel Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q1d Previous examination · complete worked answers **Question. Compare binary, multiclass and multilabel supervised text classification with examples.** [5 marks] ### Answer | Type | Allowed output | Example | Typical output rule | | --- | --- | --- | --- | | Binary | One of two labels | positive OR negative review | One sigmoid or two-class softmax | | Multiclass | Exactly one of several labels | sport OR business OR science | Softmax over mutually exclusive classes | | Multilabel | Zero, one or several labels | Article tagged technology AND finance | Independent sigmoid per label | Multiclass and multilabel differ in whether labels may coexist, not merely in the number of labels. A loss and evaluation measure must match the task. --- ## TEE · Q2a · Text corpus structure and applications Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q2a Previous examination · complete worked answers **Question. Explain text-corpus structure and the applications of text corpora in NLP.** [10 marks] ### Answer **A corpus is a collection of language data assembled for study or computational use.** Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. **Annotation** adds linguistic information such as POS tags, entities, parse structures or sense labels. | Structure | Meaning | Use | | --- | --- | --- | | Plain | Collection without category labels | Word counts, language modelling | | Categorized | Documents grouped by genre/topic | Classification | | Overlapping categories | One document has several categories | Multi-topic news | | Temporal | Documents indexed by time | Language change | | Annotated | Linguistic labels attached | Tagging, parsing, WSD | | Parallel | Aligned texts in different languages | Translation | A **tagset** is the allowed label inventory. A **treebank** is a corpus annotated with syntax trees. **Metadata** records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions. **Raw example:** “Riya works in Pune.” **POS annotation:** Riya/PROPN works/VERB in/ADP Pune/PROPN. **NER annotation:** Riya/PERSON, Pune/LOCATION. Each annotation answers a different task. A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use. Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information. | Resource | Contents | Typical use | | --- | --- | --- | | Gutenberg selection | Literary/book texts | Long-form language and vocabulary | | Web text | Informal domain-specific text | Study usage beyond edited books | | Chat corpus | Conversational posts | Dialogue, informal forms | | Brown | English texts across genres | Genre comparison | | Reuters | News with topic labels | Text classification; overlapping topics | | Inaugural | Speeches indexed by time | Vocabulary change across years | | Wordlist | Words/names/stopwords | Lexical filtering | | Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme | **Gutenberg versus web/chat:** book language does not represent every spelling or conversational pattern in messages. **Brown versus Reuters:** Brown emphasizes genres; Reuters news can have overlapping topic labels. **Inaugural addresses:** dates provide a time dimension. Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property. Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time. Word-sequence counts support n-gram language models and smoothing. POS-annotated sentences support supervised taggers. Sense-labelled contexts support Naive Bayes WSD. Word-context counts support distributional semantics and GloVe. Parallel aligned sentences support translation. For a question asking the importance of corpora, give three points: empirical language evidence, examples for training/evaluation, and reproducible comparison. For an annotated-corpus question, define the added labels and give a concrete example. For resource comparisons, distinguish running text, lexical lists, pronunciation entries and semantic relations. Same text se task ke hisaab se counts, tags, senses ya context patterns nikaalte hain. --- ## TEE · Q2b · Lesk versus Walker and bank disambiguation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q2b Previous examination · complete worked answers **Question. Differentiate Lesk and Walker WSD. Calculate sense scores and disambiguate “bank” in “Money in this bank fetches an interest of 8% per annum.”** [10 marks] ### Answer **The Walker approach uses thesaurus categories as semantic evidence.** Associate each candidate sense with a category. Count or weight context words that support that category. Choose the largest total. $$ \operatorname{score}(s)=\sum_{w\in C} I(w,\operatorname{category}(s)) $$ $$ I=1 $$ if the supplied category mapping says the context word supports the sense, otherwise $$ I=0 $$ . The summation symbol means add all those contributions. Weighted questions may replace 1 with a supplied weight. Category membership must be supplied or justified; it is not guessed from word frequency. **Question. Disambiguate bank using Walker scoring. Context words are money, interest, annum, fetch. The finance category supports money, interest and annum; the location category supports none of these words.** #### Given Two categories: Finance and Location. Each supported word contributes 1. #### To find The two totals and selected bank sense. #### Formula and why it applies Sum the category indicators for each sense. #### Step 1. Make a contribution table | Word | Finance | Location | | --- | --- | --- | | money | 1 | 0 | | interest | 1 | 0 | | annum | 1 | 0 | | fetch | 0 | 0 | #### Step 2. Add Finance $$ 1+1+1+0=3 $$ . #### Step 3. Add Location $$ 0+0+0+0=0 $$ . #### Step 4. Select 3 > 0, so select the financial institution. **Answer.** Finance, with score 3 versus 0. **Question. Use Walker scoring for “The fisherman caught bass beside the lake.” Context {fisherman,caught,lake}. The Fish category contains all three clues; the Music category contains none.** #### Given Fish and Music senses, binary category membership given. #### To find The correct bass sense. #### Formula and why it applies Score = sum of context-category matches. #### Step 1. Record Fish matches fisherman → 1; caught → 1; lake → 1. #### Step 2. Add Fish score $$ 1+1+1=3 $$ . #### Step 3. Add Music score $$ 0+0+0=0 $$ . #### Step 4. Choose Fish has the larger score. **Answer.** Bass refers to a fish. **Question. Score “The player used a bat in the match.” Context {player,match}. Sport supports both words with weights 2 and 1; Animal supports neither. Use the supplied weighted Walker scheme.** #### Given Sport weights {player:2, match:1}; Animal weights {player:0, match:0}. #### To find Weighted sense scores. #### Formula and why it applies Use the sum of supplied weights, not an unweighted count, because the question explicitly gives weights. #### Step 1. Sport contribution of player Add 2. #### Step 2. Sport contribution of match Add 1. Sport total = 2 + 1 = 3. #### Step 3. Animal total 0 + 0 = 0. #### Step 4. Compare 3 > 0 selects the sporting implement. **Answer.** Bat is the sports implement. Walker: dictionary ke exact words nahi, context kis topic/category ko support karta hai woh count. | Aspect | Lesk | Walker/thesaurus method | | --- | --- | --- | | Evidence | Overlap of context with dictionary glosses | Context-word membership in semantic subject categories | | Score | Number of shared normalized words | Number/weight of context words supporting each sense category | | Dependency | Gloss definitions and preprocessing | Thesaurus categories and their assignments | ### Given Context is the printed sentence. No thesaurus category list is supplied. ### To find State the sense and give a transparent illustrative score. ### Formula: which one and why Declare the categories first; count only declared evidence. A numerical Walker score is not uniquely determined without the thesaurus assignments. ### Step 1. Choose explicit illustrative categories Assume money and interest belong to FINANCE; river, water and shore belong to RIVER. Ignore all other tokens as unassigned. ### Step 2. Count support | Candidate bank sense | Matching clues | Score | | --- | --- | --- | | Financial institution | money, interest | 2 | | River edge | none | 0 | ### Step 3. Compare 2 > 0, so choose the financial-institution sense under this stated assignment. ### Final answer **bank = financial institution.** The score 2 versus 0 is conditional on the declared illustrative thesaurus, not supplied numerical data. **Source / convention note.** The original gives a sentence but no thesaurus/category mapping, so a unique numerical score cannot be recovered. --- ## TEE · Q3a · Hobbs algorithm and limitations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q3a Previous examination · complete worked answers **Question. Explain Hobbs’ algorithm with an example and state its limitations.** [10 marks] ### Answer **Hobbs’ algorithm is a syntax-based procedure that searches constituent parse trees in a fixed order for a compatible noun-phrase antecedent of a pronoun.** The tree determines which candidates are proposed first. Agreement and other relevant constraints determine whether a proposed candidate can be accepted. It is not simply the rule “choose the closest noun”. A nearby noun may have the wrong number, may be syntactically ineligible, or may be visited after a structurally more prominent noun. The algorithm produces a preference under the supplied parse; ambiguous language can still require context. ### The tree vocabulary you need | Term | Meaning | Example | | --- | --- | --- | | NP | Noun phrase: a phrase referring to an entity | the student; Anita; they | | S | A sentence/clause node | the node containing a subject and predicate | | VP | Verb phrase | told Ravi that she should revise | | X | The NP or S node currently being searched | An algorithm label, not a word | | Path p | The branch travelled upward from the pronoun/previous X | Used to distinguish left and right subtrees | | N-bar (N′) | The noun-centred part of an NP in a grammar using that label | The nominal-child test depends on the given tree | | Breadth-first | Visit shallower nodes before deeper ones | Level 1 before level 2 | | Left-to-right | Visit equally deep nodes in their written/tree order | Earlier left branch before right branch | Locate pronoun NP → Climb to NP/S → Propose candidates in tree order → Check compatibility → Accept or continue searching ### The complete search procedure #### Step 1. Start at the pronoun Locate the NP immediately dominating the pronoun. “Dominating” means containing it as a descendant in the tree. #### Step 2. Climb to the first NP or S Move upward to the first NP or S above that starting NP. Call this node X. Keep the upward path p. #### Step 3. Search the left branches initially Below X, traverse branches to the left of p breadth-first and left-to-right. At this initial search, propose an NP only if another NP or S lies between that candidate and X. Test proposed candidates for the required agreement and constraints. #### Step 4. If X is the highest sentence node, inspect earlier sentences Search previous sentence trees from most recent to oldest. Within each, use breadth-first, left-to-right order to propose NPs. Accept the first eligible compatible candidate, or report that no candidate was found in the available context. #### Step 5. Otherwise climb again Move from X to the next NP or S above it. Record this new X and the path followed during the climb. #### Step 6. Apply the higher-NP test If the new X is an NP, propose X when the path did not pass through its nominal N-bar child. Use the given parse labels; do not invent an N-bar branch missing from the question's structure. #### Step 7. Search its left branches Traverse branches below the new X to the left of the path, breadth-first and left-to-right, proposing NPs. The special intervening-NP/S restriction belongs to the initial search in Step 3. #### Step 8. If X is S, search the permitted right branches Search branches to the right of the path breadth-first and left-to-right. Do not descend below an encountered NP or S. Propose eligible NP candidates there. #### Step 9. Repeat or stop Return to the highest-S check and continue the climb/search procedure until a compatible candidate is accepted or the available trees are exhausted. Record the candidate order rather than only writing an unexplained name. ### Why the first left search has an extra condition In a simple clause such as “John saw him”, the subject NP John is directly under the clause S. There is no extra NP or S between John and that S. The initial-search restriction therefore prevents casually selecting John as the ordinary local antecedent of him. Compare the reflexive “John saw himself”, which is a different grammatical case; do not treat every pronoun type as interchangeable. ### Worked tree 1 · move to the previous sentence **Question. Apply Hobbs’ search to “The student arrived. She smiled.” Assume the student is compatible with she.** #### Given Previous tree: [S [NP The student] [VP arrived]]. Current tree: [S [NP She] [VP smiled]]. #### To find The selected antecedent and the search path. #### Method and why it applies Use the supplied constituent trees. The current sentence has no earlier candidate, so the highest-S rule activates previous-sentence search. ### Solution #### Step 1. Draw both sentence structures ``` Previous sentence Current sentence S S ├── NP: The student ├── NP: She ← start └── VP: arrived └── VP: smiled ``` #### Step 2. Locate and climb Start at NP She. Its first higher NP/S is the current S. Set X to that S. #### Step 3. Search left of the path She is the leftmost subject. No NP lies to its left in the current sentence. #### Step 4. Use the highest-S rule This X is the highest S. Inspect the immediately preceding sentence tree. #### Step 5. Propose its first NP Breadth-first, left-to-right finds NP The student. #### Step 6. Check compatibility and accept The question stipulates that the student agrees with she. Accept this candidate. **Answer.** She refers to the student under the supplied assumptions. ### Worked tree 2 · reject an incompatible subject **Question. Resolve they in “Riya spoke to the students. They laughed.” Use a previous-sentence search and interpret they as plural.** #### Given Previous tree has subject NP Riya, then VP spoke [PP to [NP the students]]. Current subject is NP They. #### To find The candidate order and accepted antecedent. #### Method and why it applies Breadth-first search favours a shallow subject, but agreement must still be applied; a first candidate is not automatically a valid answer. ### Solution #### Step 1. Draw the useful part of the previous tree ``` S ├── NP: Riya ← candidate 1 └── VP ├── V: spoke └── PP ├── P: to └── NP: the students ← candidate 2 ``` #### Step 2. Search the current sentence From NP They climb to its S; there is no earlier NP in “They laughed”. This is the highest S, so move to the previous tree. #### Step 3. Visit Riya first Riya is the shallower leftmost NP. Under the stated plural interpretation, this singular candidate fails number agreement. #### Step 4. Visit the students Continue breadth-first. The next relevant NP is the students. #### Step 5. Apply number agreement Students is plural and is an eligible candidate. It matches plural they. #### Step 6. State the resolution Accept the students. The method did not simply accept the first noun it saw. **Answer.** They → the students. Candidate order: Riya rejected; the students accepted. ### Worked tree 3 · climb out of an embedded clause **Question. Resolve she in “Anita told Ravi that she should revise.” Assume Anita is compatible with she and Ravi is not. Use the displayed parse.** #### Given The pronoun is the subject of an embedded S inside the main VP. #### To find The preferred antecedent found by the climb and left-branch search. #### Method and why it applies The embedded clause has no left candidate. Climbing to the main S exposes its subject Anita as the first suitable higher-level candidate. ### Solution #### Step 1. Show the clause structure ``` S (main) ├── NP: Anita ← accepted candidate └── VP ├── V: told ├── NP: Ravi └── SBAR: that └── S (embedded) ← first X ├── NP: she ← start └── VP: should revise ``` #### Step 2. Start inside the embedded clause From NP she climb to the embedded S. There is no candidate to the left of she within this embedded S. #### Step 3. Climb to the next NP/S The embedded S is not the highest S. Move upward through SBAR and VP to the main S. These intermediate nodes are not NP/S search nodes. #### Step 4. Search left of the upward path At the main S, the subject NP Anita is a shallow left candidate. The object Ravi is deeper within the VP. Breadth-first order proposes Anita first. #### Step 5. Check the stated compatibility Anita is compatible with she by the question's assumption. Accept it; no need to choose a later incompatible candidate. #### Step 6. Give the result and its limit This is the algorithm's result under the displayed parse and agreement assumptions. It does not prove the intention of every speaker using an otherwise ambiguous sentence. **Answer.** She → Anita under the stated tree and compatibility information. ### Advantages, limitations and a complete exam answer | Advantages | Limitations | | --- | --- | | Explicit candidate order makes the decision traceable. | Wrong parse structure can produce the wrong search order. | | Requires no labelled coreference-training corpus for its basic rules. | World knowledge and discourse intention may override syntactic preference. | | Uses structure rather than only surface distance. | Some ambiguous, long-distance or unusual constructions need additional treatment. | For a longer answer, write the definition, inputs, tree notation, ordered algorithm, one labelled tree with a candidate table, final antecedent, and limitations. In a worked question show separately: candidate → why proposed → agreement result → accept/reject. Do not equate the selected candidate with a logically guaranteed interpretation when the sentence is ambiguous. **Yaad rakho:** Tree order candidate deta hai; agreement usko allow/reject karta hai. Nearest noun automatically answer nahi. --- ## TEE · Q3b · Stemming and lemmatization Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q3b Previous examination · complete worked answers **Question. Discuss the need for stemming, list different types of stemming, and differentiate stemming from lemmatization.** [10 marks] ### Answer Stemming merges inflected/derived surface variants for matching and reduces vocabulary size. Types include rule/affix-removal stemmers (Porter, Snowball, Lancaster), dictionary/lookup stemmers, statistical stemmers and hybrid methods. More aggressive reduction increases both merging and overstemming risk. **Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma. | Criterion | Stemming | Lemmatization | | --- | --- | --- | | Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis | | Output | May be nonword, e.g. studi | Dictionary form, e.g. study | | Context/POS | Usually little or none | Often important | | Irregular forms | May miss went→go | Can map went→go with verb analysis | | Cost | Generally simpler | Usually more linguistic processing | Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation. ### Three worked reductions 1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked. 2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation. 3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter. **Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice. Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan. --- ## TEE · Q4a · Lexical and vector semantics Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q4a Previous examination · complete worked answers **Question. Explain lexical semantics and vector semantics with suitable examples.** [10 marks] ### Answer **A lexical relation connects words or senses by a meaning relationship.** State the relation and its direction before giving the example. | Relation | Exam definition | Examples and direction | | --- | --- | --- | | Homonymy | The same form has unrelated meanings. | bank: financial institution / river edge; bat: animal / sports implement | | Polysemy | One lexical item has multiple related senses. | head: body part / head of a department; paper: material / article | | Synonymy | Different words have identical or closely similar meanings in a specified sense. | sofa / couch; automobile / car | | Antonymy | Words have opposite meanings. | hot / cold; alive / dead; buy / sell | | Hyponymy | A more specific concept is a type of a more general one. | rose is a flower; car is a vehicle | | Hypernymy | A broader concept includes a more specific one. | flower is a hypernym of rose | | Meronymy | A concept is a part of another concept. | wheel is part of car; finger is part of hand | | Holonymy | A concept is the whole containing a part. | car is a holonym of wheel | ### How to distinguish the commonly confused pairs For homonymy versus polysemy, ask whether the senses share an identifiable meaning. A river bank and a financial bank are unrelated. A financial bank and a blood bank share a repository/storage idea, illustrating polysemy in the supplied notes. For hyponymy versus meronymy, use the sentence test: **“X is a kind of Y”** indicates hyponymy; **“X is part of Y”** indicates meronymy. A wheel is not a type of car. | Expression | Correct relation | Reason | | --- | --- | --- | | sparrow → bird | Hyponymy | A sparrow is a kind of bird | | bird → sparrow | Hypernymy | Bird is the broader category | | keyboard → computer | Meronymy | Keyboard is a component | | computer → keyboard | Holonymy | Computer is the containing whole | | mouth of a person / mouth of a river | Polysemy | Both senses share the idea of an opening; the river use is a related extension | Synonyms are not always interchangeable in every context or register. “Big mistake” and “large mistake” can sound different even though big and large overlap in meaning. Antonyms can be gradable (hot/cold), complementary (alive/dead) or relational converses (buy/sell). HYPO = specific type; HYPER = bigger category. MERO = part; HOLO = whole. **The distributional hypothesis states that words occurring in similar contexts tend to have similar meanings or semantic behaviour.** “Cat” and “dog” often occur near pet, feed and animal. Their context patterns can therefore be represented and compared numerically. A **vector** is an ordered list of numbers. Each position must have a fixed meaning. For context order [pet, food, road], cat=[4,3,0] means four co-occurrences with pet, three with food, none with road. A **matrix** stacks vectors as rows in a table. A **word–context co-occurrence matrix** records how often a target word occurs with each context word under a specified context definition. A window may be a sentence, document, or a fixed number of neighbouring tokens. State its width, direction and whether targets count themselves. Choose target words → Choose context/window → Count co-occurrences → Store each row as a vector → Compare vectors **Question. Build the symmetric co-occurrence matrix for “cats eat fish” using a window of one token on either side, no self-counts and vocabulary [cats,eat,fish].** #### Given One sentence, one occurrence of each token. Left and right adjacent tokens are contexts. #### To find A 3×3 target-by-context count matrix. #### Formula and why it applies $$ X_{ij} $$ counts the number of times context word $$ j $$ occurs within the chosen window of target $$ i $$ . #### Step 1. Target cats Only eat is adjacent. Row cats = [0,1,0]. #### Step 2. Target eat cats is left and fish is right. Row eat = [1,0,1]. #### Step 3. Target fish Only eat is adjacent. Row fish = [0,1,0]. #### Step 4. Assemble the matrix | Target / context | cats | eat | fish | | --- | --- | --- | --- | | cats | 0 | 1 | 0 | | eat | 1 | 0 | 1 | | fish | 0 | 1 | 0 | **Answer.** Rows are [0,1,0], [1,0,1], [0,1,0]. The symmetry follows from the symmetric window. **Question. Build a target–context matrix for sentences “cats eat fish” and “dogs eat fish”. Target words are cats and dogs; context columns [eat,fish]; use the entire sentence excluding the target.** #### Given Each sentence contains one target. Count context token occurrences. #### To find Two context vectors. #### Formula and why it applies Count each context word in every sentence containing the target. #### Step 1. Count cats contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 2. Count dogs contexts In its sentence: eat occurs once, fish once → [1,1]. #### Step 3. Compare Both targets have identical observed context patterns in this very small corpus. **Answer.** cats=[1,1], dogs=[1,1]. This corpus supplies no context evidence to distinguish them. **Question. For “a b a”, use a right-only window of one token. Build the matrix with rows/columns [a,b].** #### Given Count directed target→next-word events, not both directions. #### To find The co-occurrence matrix. #### Formula and why it applies $$ X_{ij}=C(\text{target }i\text{ immediately followed by }j) $$ . #### Step 1. First target a Next token is b → add 1 to X(a,b). #### Step 2. Target b Next token is a → add 1 to X(b,a). #### Step 3. Final a No following token → no addition. #### Step 4. Write matrix | Target / next | a | b | | --- | --- | --- | | a | 0 | 1 | | b | 1 | 0 | **Answer.** a=[0,1], b=[1,0]. A different context window would produce a different matrix. Word ka meaning: uske aas-paas kaunse words kitni baar aaye, uski number-list. --- ## TEE · Q4b · Three-document TF–IDF Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q4b Previous examination · complete worked answers **Question. Calculate TF–IDF for: (1) Inflation increased unemployment (2) Company increased sales (3) Fear increased pulse.** [10 marks] ### Answer ### Given | Document | Original text | | --- | --- | | 1 | Inflation increased unemployment | | 2 | Company increased sales | | 3 | Fear increased pulse | Stopwords: none supplied ### To find BoW, IDF and TF–IDF ### Formula: which one and why Use lowercase alphabetic tokens and exactly the stated stoplist. If no stoplist is supplied, retain function words. Adopt raw-count TF and natural-log IDF; the paper does not prescribe another TF/log convention. $$ \mathrm{TFIDF}(w,d)=c(w,d)\ln\frac{N}{\mathrm{df}(w)} $$ A term shared by all N documents has IDF ln(1)=0. Counts and TF–IDF weights are different representations. ### Step 1. Tokenize each document | Document | Tokens | | --- | --- | | 1 | inflation, increased, unemployment | | 2 | company, increased, sales | | 3 | fear, increased, pulse | ### Step 2. Fix the vocabulary order company, fear, increased, inflation, pulse, sales, unemployment ### Step 3. Count occurrences (Bag of Words) | Document | company | fear | increased | inflation | pulse | sales | unemployment | | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | | 2 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | | 3 | 0 | 1 | 1 | 0 | 1 | 0 | 0 | ### Step 4. Calculate document frequency and IDF | Term | df | N / df | ln(N/df) | | --- | --- | --- | --- | | company | 1 | 3 | 1.098612289 | | fear | 1 | 3 | 1.098612289 | | increased | 3 | 1 | 0.000000000 | | inflation | 1 | 3 | 1.098612289 | | pulse | 1 | 3 | 1.098612289 | | sales | 1 | 3 | 1.098612289 | | unemployment | 1 | 3 | 1.098612289 | ### Step 5. Multiply each TF by its IDF | Document | company | fear | increased | inflation | pulse | sales | unemployment | | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | 1.098612 | | 2 | 1.098612 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 0.000000 | | 3 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | ### Final answer The final table is the TF–IDF matrix in the declared vocabulary order. A different explicitly stated log base rescales all IDFs; length-normalized TF changes each document scale. --- ## TEE · Q5a · Word embeddings and CBOW Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q5a Previous examination · complete worked answers **Question. Elaborate word embeddings. Explain the steps in CBOW with a suitable diagram.** [10 marks] ### Answer **A sparse representation** has many zero entries and often one coordinate per vocabulary item. BoW and TF–IDF are common sparse document representations. **A dense embedding** has comparatively few learned real-valued coordinates; one word's properties are spread across many coordinates. | Representation | What coordinates mean | Strength | Limitation | | --- | --- | --- | --- | | One-hot | One identity position per vocabulary word | Unambiguous ID | Different words are orthogonal; no learned similarity | | Count-based context vector | Context-word co-occurrence counts | Transparent corpus evidence | Large, sparse; frequent words may dominate | | TF–IDF document vector | Weighted vocabulary terms | Useful retrieval weighting | Word order and contextual meaning are limited | | Dense word embedding | Learned latent features | Compact semantic/syntactic relationships | Coordinates are not generally human-labelled features | **Distributional** describes learning from patterns of linguistic context. **Distributed** describes representing an item across multiple coordinates rather than a single active identity bit. These are related but different ideas. A one-hot ID alone does not learn distributional similarity. ### Model design choices Choose contexts (document, sentence, fixed window), weighting (raw counts, normalization, TF–IDF-like weighting or association scores), dimensionality reduction or learned embeddings, and a similarity measure. Cosine compares direction and is less affected by length than a raw dot product; Euclidean distance also reflects magnitude. Sparse = bahut zero. Dense = chhoti learned number-list. Similar context se useful similarity seekhte hain. **Word2Vec learns dense word embeddings by predicting words from nearby context.** Its two common architectures are CBOW and skip-gram. A vocabulary ID or one-hot vector is the input representation; the learned embedding table converts that identity into a short trainable vector. | Architecture | Input | Prediction | Memory cue | | --- | --- | --- | --- | | CBOW | Surrounding context words | Centre/target word | Context → centre | | Skip-gram | Centre/target word | Surrounding words | Centre → context | CBOW: the + sat → Average context embeddings → Predict: cat Skip-gram: cat → Embedding of cat → Predict: the and sat For vocabulary size $$ V $$ and embedding dimension $$ d $$ , an input one-hot row is $$ 1\times V $$ ; the input embedding matrix is $$ V\times d $$ ; the hidden row is $$ 1\times d $$ ; the output matrix is $$ d\times V $$ ; output scores are $$ 1\times V $$ . Multiplying a one-hot row by the embedding matrix simply selects one row. $$ P(w_o\mid w_i)=\frac{\exp(u_{w_o}^{T}v_{w_i})}{\sum_{w\in V}\exp(u_w^{T}v_{w_i})}. $$ $$ v_{w_i} $$ is the input word embedding, $$ u_w $$ an output-context embedding, and the dot product is a compatibility score. Softmax turns all scores into probabilities that sum to one. Training increases the probability of observed context words. Negative sampling or hierarchical softmax can reduce computation in larger vocabularies; use the algorithm specified by the question. **Question. For the sentence “the cat sat on mat”, generate CBOW and skip-gram training data for centre word sat with a window of one token on each side.** #### Given Tokens indexed 1 to 5: the, cat, sat, on, mat. Centre = sat. #### To find The CBOW example and the skip-gram pairs. #### Formula and why it applies CBOW predicts the centre from its neighbours; skip-gram predicts each neighbour from the centre. #### Step 1. Identify neighbours Left neighbour = cat; right neighbour = on. “the” and “mat” are outside this window. #### Step 2. CBOW example Input [cat,on] → target sat. #### Step 3. Skip-gram examples (sat,cat) and (sat,on), with first item as input and second as predicted context. **Answer.** CBOW: [cat,on]→sat. Skip-gram: sat→cat and sat→on. **Question. The embeddings of context words cat and on are [1,2] and [3,0]. Compute the CBOW hidden vector using mean aggregation. Output vectors for [sat,ran] are [1,0] and [0,1]. Compute softmax probabilities.** #### Given Context embeddings [1,2], [3,0]; mean aggregation; two candidate output vectors. #### To find The hidden vector, logits and probabilities. #### Formula and why it applies Mean context vector $$ h=(v_1+v_2)/2 $$ . Logit $$ z_w=u_w^Th $$ . Softmax $$ P(w)=e^{z_w}/\sum_j e^{z_j} $$ . #### Step 1. Add coordinates $$ [1+3,\,2+0]=[4,2] $$ . #### Step 2. Divide by number of words $$ h=[4/2,2/2]=[2,1] $$ . #### Step 3. Compute logits $$ z_{sat}=1(2)+0(1)=2 $$ . $$ z_{ran}=0(2)+1(1)=1 $$ . #### Step 4. Exponentials $$ e^2\approx7.389056 $$ , $$ e^1\approx2.718282 $$ ; sum $$ \approx10.107338 $$ . #### Step 5. Divide $$ P(sat)\approx0.731059 $$ , $$ P(ran)\approx0.268941 $$ . **Answer.** h=[2,1]; sat is more probable (0.731059). **Question. A skip-gram centre embedding is v=[1,−1]. Output embeddings for [eat,sleep,drive] are [1,1], [0,1], [1,0]. Calculate the predicted context probabilities.** #### Given One centre vector and three output vectors; full softmax over these three words. #### To find Three logits and three probabilities. #### Formula and why it applies $$ z_j=u_j^Tv $$ , then $$ P(j)=e^{z_j}/\sum_k e^{z_k} $$ . #### Step 1. Eat dot product $$ 1(1)+1(-1)=0 $$ . #### Step 2. Sleep and drive dot products $$ z_{sleep}=0(1)+1(-1)=-1 $$ . $$ z_{drive}=1(1)+0(-1)=1 $$ . #### Step 3. Exponentiate and sum $$ e^0=1 $$ , $$ e^{-1}\approx0.367879 $$ , $$ e^1\approx2.718282 $$ . Sum $$ \approx4.086161 $$ . #### Step 4. Normalize $$ P(eat)\approx0.244728 $$ ; $$ P(sleep)\approx0.090031 $$ ; $$ P(drive)\approx0.665241 $$ . **Answer.** drive has the highest probability in this toy learned model. **Limitations:** a standard static Word2Vec vector merges a word's senses; it has limited representation of subword morphology and out-of-vocabulary words; its local prediction objective does not directly fit a full global co-occurrence table. Similarity is learned from data and is not a guarantee of factual or causal understanding. CBOW = neighbours se missing centre. Skip-gram = centre se neighbours. --- ## TEE · Q5b · HMM tables and given path Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q5b Previous examination · complete worked answers **Question. For the four tagged sentences below, construct transition (3 marks) and emission (3 marks) tables. Calculate the path Geeta/N will/M spot/V Rita/N, given initial P(N)=0.5, P(M)=0.25, P(V)=0.25 (4 marks).** [10 marks] ### Answer ### Given | Sentence | Given tags | | --- | --- | | Rita can teach Will | N M V N | | Spot will see Will | N M V N | | Will Rita spot Geeta | N N V N | | Geeta will pat Spot | N M V N | Required path: Geeta will spot Rita / N M V N. Given initial probabilities: {'N': Fraction(1, 2), 'M': Fraction(1, 4), 'V': Fraction(1, 4)} ### To find Emission and transition tables and the specified tag-path joint probability ### Formula: which one and why $$ P(w\mid t)=\frac{c(t,w)}{c(t)} $$ $$ P(t_j\mid t_i)=\frac{c(t_i,t_j)}{\sum_kc(t_i,t_k)} $$ The second formula is the declared no-END convention. Reset at sentence boundaries; never create a transition from the last word of one training sentence to the first of the next. ### Step 1. Count emissions | Tag | Total occurrences | Rita | can | teach | Will | Spot | will | see | spot | Geeta | pat | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | N | 9 | 2 | 0 | 0 | 3 | 2 | 0 | 0 | 0 | 2 | 0 | | M | 3 | 0 | 1 | 0 | 0 | 0 | 2 | 0 | 0 | 0 | 0 | | V | 4 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 1 | ### Step 2. Divide each emission by its tag count | Tag | Rita | can | teach | Will | Spot | will | see | spot | Geeta | pat | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | N | 2/9 | 0 | 0 | 1/3 | 2/9 | 0 | 0 | 0 | 2/9 | 0 | | M | 0 | 1/3 | 0 | 0 | 0 | 2/3 | 0 | 0 | 0 | 0 | | V | 0 | 0 | 1/4 | 0 | 0 | 0 | 1/4 | 1/4 | 0 | 1/4 | ### Step 3. Count transitions inside each sentence | From | Outgoing transitions | Sentence-final count | N | M | V | | --- | --- | --- | --- | --- | --- | | N | 5 | 4 | 1 | 3 | 1 | | M | 3 | 0 | 0 | 0 | 3 | | V | 4 | 0 | 4 | 0 | 0 | ### Step 4. Convert to a transition table | From / to | N | M | V | | --- | --- | --- | --- | | N | 1/5 | 3/5 | 1/5 | | M | 0 | 0 | 1 | | V | 1 | 0 | 0 | Here sentence-final tags are excluded from the no-END outgoing denominator. If END is explicitly modeled, divide all tag→tag and tag→END counts by the total tag count instead; the row then includes END. Do not mix denominators. ### Step 5. Multiply the given path, in order | Factor | Probability | Running product | | --- | --- | --- | | Initial N | 1/2 | 1 × 1/2 = 1/2 | | Emit Geeta | 2/9 | 1/2 × 2/9 = 1/9 | | Transition N→M | 3/5 | 1/9 × 3/5 = 1/15 | | Emit will | 2/3 | 1/15 × 2/3 = 2/45 | | Transition M→V | 1 | 2/45 × 1 = 2/45 | | Emit spot | 1/4 | 2/45 × 1/4 = 1/90 | | Transition V→N | 1 | 1/90 × 1 = 1/90 | | Emit Rita | 2/9 | 1/90 × 2/9 = 1/405 | ### Final answer The two displayed probability tables are the estimated HMM. Zero means unobserved without smoothing, not logically impossible. **No-END path joint probability = 1/405 = 0.00246913580247.** If a normalized END-state model is explicitly adopted instead, this complete terminated sequence has probability 4/6561 = 0.000609663161103. These are different model conventions, not contradictory answers. --- ## TEE · Q6a · Text classification pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q6a Previous examination · complete worked answers **Question. Explain the text-classification pipeline in detail with a suitable block diagram.** [10 marks] ### Answer **Text classification** assigns a document to one or more predefined categories. Spam detection usually uses two classes; news classification may use several topics. In single-label multiclass classification, choose one of several categories. In multilabel classification, several labels may apply to the same document. Define labels → Prepare labelled examples → Train representation + model → Evaluate on unseen text → Predict new labels ### An exam-ready pipeline | Stage | What you do | Why it matters | | --- | --- | --- | | Task definition | Decide what counts as spam or which news topics exist. | Unclear labels create inconsistent training data. | | Data collection and annotation | Collect representative examples and assign correct labels. | The model learns from these examples. | | Split data | Separate training, validation and test data. | Test examples must not influence training choices. | | Preprocess | Tokenize and normalise where suitable; retain meaningful negation. | Reduce irrelevant variation without deleting signal. | | Represent | Use BoW, TF-IDF or learned embeddings. | Models need numerical features. | | Train | Fit Naive Bayes, logistic regression or another appropriate model. | Learn how features predict categories. | | Evaluate | Measure relevant precision, recall and F1 on held-out examples. | Accuracy alone may hide poor minority-class performance. | | Deploy and monitor | Apply the same preprocessing and monitor changing language. | Incoming data can differ from old examples. | Naive Bayes is especially useful for hand calculations. With a bag-of-words representation, its class score is the prior multiplied by the conditional probability of each token; repeated tokens repeat the factor. The “naive” assumption is conditional independence of features given the class, not that words are independent in real language. $$ s(c)=P(c)\prod_{w\in V}P(w\mid c)^{C(w,d)} $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ C(w,d) counts word w in document d. C(w,c) counts it in training documents belonging to class c. Nc is the total token count for class c. Add-one smoothing adds one to every vocabulary count and therefore adds |V| to the denominator. This is the same arithmetic taught for WSD, with document classes instead of senses. ### Worked classification · two-token message A spam classifier has vocabulary {win, prize, meeting}. Priors are P(spam)=P(ham)=0.5. Spam token counts are [3,2,0]; ham counts are [0,0,5]. With add-one smoothing, classify “win prize”. **Given** |V|=3; N(spam)=5; N(ham)=5; message has win once and prize once. **To find** Class scores and predicted label. ### Formula and why it applies $$ s(c)=P(c)P(\text{win}\mid c)P(\text{prize}\mid c) $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ The model is multinomial Naive Bayes; multiply the two token likelihoods and the class prior. ### Solution #### Step 1. Compute the smoothed denominators $$ 5+3=8 $$ #### Step 2. Compute spam likelihoods $$ P(\text{win}\mid S)=\frac{3+1}{8}=\frac12 $$ $$ P(\text{prize}\mid S)=\frac{2+1}{8}=\frac38 $$ #### Step 3. Compute ham likelihoods $$ P(\text{win}\mid H)=P(\text{prize}\mid H)=\frac{0+1}{8}=\frac18 $$ #### Step 4. Multiply the spam score $$ s(S)=\frac12\times\frac12\times\frac38=\frac{3}{32}=0.09375 $$ #### Step 5. Multiply the ham score $$ s(H)=\frac12\times\frac18\times\frac18=\frac{1}{128}=0.0078125 $$ #### Step 6. Compare 0.09375 > 0.0078125. These are unnormalised scores; the label is chosen by the larger score. Predicted label: spam. **Yaad rakho:** Class prior × har word ki class-likelihood; sabse bada score wins. --- ## TEE · Q6b · VSM, TF and IDF in retrieval Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f23/q6b Previous examination · complete worked answers **Question. Define Vector Space Model, Term Frequency and Inverse Document Frequency with reference to information retrieval.** [10 marks] ### Answer **A vocabulary** is the ordered collection of distinct terms allowed by a representation. Its order defines what every coordinate means. A **vector** is simply an ordered number-list. A **document–term matrix** puts one document in each row and one vocabulary term in each column. With vocabulary [cat,dog,fish], the count vector [2,0,1] means cat appears twice, dog zero times and fish once. With [dog,cat,fish], the same text becomes [0,2,1]. A vector without its vocabulary order is incomplete. | Document | cat | dog | fish | | --- | --- | --- | --- | | cat cat fish | 2 | 0 | 1 | | dog fish | 0 | 1 | 1 | **Nominal categories** have no inherent order, such as cities. **Ordinal categories** do have an order, such as low < medium < high. Assigning arbitrary word IDs does not create meaningful numerical distances between words. Vector ka har box ek fixed word ka hai. Pehle column order fix, phir numbers. **TF–IDF weights a term highly when it is frequent in a particular document but uncommon across the corpus.** The supplied Unit 2 slides use normalized term frequency and base-10 inverse document frequency: $$ \operatorname{TF}(t,d)=\frac{C(t,d)}{N_d} $$ $$ \operatorname{IDF}(t)=\log_{10}\frac{D}{\operatorname{df}(t)} $$ $$ \operatorname{TFIDF}(t,d)=\operatorname{TF}(t,d)\operatorname{IDF}(t). $$ $$ N_d $$ is the document's token count after the specified preprocessing; $$ D $$ is number of documents; $$ \operatorname{df}(t) $$ is the number of documents containing the term at least once. A word repeated ten times in one document contributes one to document frequency. **Why the log?** It compresses the ratio's scale. If a term occurs in all documents, $$ D/df=1 $$ , so IDF=0 under this unsmoothed convention. Other libraries use smoothed IDF, different TF or normalization. Use the question's formula; do not silently substitute a library default. **Question. A corpus contains 10 documents. The word model appears in 2 documents and occurs 3 times in a 12-token document. Compute TF, IDF and TF–IDF using the slide convention.** #### Given D=10, df(model)=2, C(model,d)=3, Nd=12; log base 10. #### To find The TF–IDF weight. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ , then multiply. #### Step 1. TF $$ 3/12=0.25 $$ . #### Step 2. Document ratio $$ 10/2=5 $$ . #### Step 3. IDF $$ \log_{10}5\approx0.698970 $$ . #### Step 4. Weight $$ 0.25\times0.698970\approx0.174743 $$ . **Answer.** TF=0.25, IDF≈0.698970, TF–IDF≈0.174743. **Question. A term occurs 4 times in a 20-token document and is present in all 8 corpus documents. Calculate its TF–IDF.** #### Given C=4, Nd=20, D=8, df=8. #### To find Its weight under unsmoothed base-10 IDF. #### Formula and why it applies $$ TF=C/N_d $$ , $$ IDF=\log_{10}(D/df) $$ . #### Step 1. TF $$ 4/20=0.2 $$ . #### Step 2. IDF $$ \log_{10}(8/8)=\log_{10}1=0 $$ . #### Step 3. Multiply $$ 0.2\times0=0 $$ . **Answer.** Weight 0: this term does not distinguish documents under this convention. **Question. Find TF–IDF vectors for D1=“cat cat dog”, D2=“dog fish”, D3=“cat fish fish”, using [cat,dog,fish], normalized TF and log10 IDF.** #### Given No further preprocessing. Each term occurs in exactly two of the three documents. #### To find All three vectors. #### Formula and why it applies First count df by documents, then compute each document's TF, then multiply by its term's IDF. #### Step 1. Common IDF $$ D=3 $$ , $$ df(cat)=df(dog)=df(fish)=2 $$ . Each IDF $$ =\log_{10}(3/2)\approx0.176091 $$ . #### Step 2. D1 term frequencies Length 3 → $$ [2/3,1/3,0] $$ . #### Step 3. D1 weights $$ [(2/3)0.176091,(1/3)0.176091,0]\approx[0.117394,0.058697,0] $$ . #### Step 4. D2 weights Length 2; TF $$ [0,1/2,1/2] $$ . Vector $$ \approx[0,0.088046,0.088046] $$ . #### Step 5. D3 weights Length 3; TF $$ [1/3,0,2/3] $$ . Vector $$ \approx[0.058697,0,0.117394] $$ . **Answer.** D1≈[0.117394,0.058697,0]; D2≈[0,0.088046,0.088046]; D3≈[0.058697,0,0.117394]. TF: iss document mein kitna. IDF: poore corpus mein kitna rare. Dono multiply. **Information retrieval (IR)** locates and ranks items relevant to an information need, usually expressed as a query. The collection may contain documents, product descriptions or scientific papers. Retrieval estimates relevance; it does not itself establish that every retrieved claim is true. Documents → cleaning → Build index → Query → same representation → Match & rank → Return relevant results ### The inverted index An inverted index maps a term to the documents containing it, often with positions or counts. For D1 “cats drink milk” and D2 “dogs drink water”, the posting list for drink is {D1,D2}, while milk maps only to D1. This avoids rescanning every document for every query. | Term | Posting list | | --- | --- | | cats | D1 | | drink | D1, D2 | | milk | D1 | | dogs | D2 | | water | D2 | ### Matching and ranking Boolean retrieval combines term membership using AND, OR and NOT. Vector-space retrieval represents documents and queries as weighted vectors, for example with TF-IDF, and compares them with cosine similarity. Learned retrieval can use dense embeddings for meaning-based matching. The representation used at query time must be compatible with the document index. ### Search engines and recommendation A search engine receives an explicit query and ranks matching indexed pages. A recommender may derive an implicit query from a user’s liked items or profile and retrieve similar candidates. Both require candidate retrieval and ranking, but the source of the information need differs. ### Evaluate the results For retrieval, TP is the number of retrieved relevant documents, FP retrieved irrelevant documents, and FN relevant documents not retrieved. Precision and recall use the same formulas as classification. Ranking measures also consider the positions of relevant items; a relevant result at rank 1 is more useful than one far below. ### Retrieval evaluation A search system retrieves 8 documents; 6 are relevant. There are 10 relevant documents in the collection. Find precision, recall and F1. **Given** Retrieved=8; relevant retrieved=6; all relevant=10. **To find** P, R and F1. ### Formula and why it applies $$ P=\frac{\text{relevant retrieved}}{\text{retrieved}},\quad R=\frac{\text{relevant retrieved}}{\text{all relevant}} $$ $$ F_1=\frac{2PR}{P+R} $$ The denominator is 8 for precision and 10 for recall because they ask different questions. ### Solution #### Step 1. Identify the confusion counts TP=6; FP=8−6=2; FN=10−6=4. #### Step 2. Precision $$ P=\frac68=0.75 $$ #### Step 3. Recall $$ R=\frac6{10}=0.6 $$ #### Step 4. F1 $$ F_1=\frac{2(0.75)(0.6)}{0.75+0.6}=\frac{0.9}{1.35}=\frac23\approx0.666667 $$ Precision=0.75, recall=0.60, F1≈0.666667. **Yaad rakho:** Precision: jo laaye, unmein kitne sahi? Recall: jo sahi the, unmein kitne laaye? --- ## TEE · Q1a · Normalization changes vocabulary Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q1a Previous examination · complete worked answers **Question. “The vocabulary of a corpus remains the same before and after normalization.” State true or false and justify with an example.** [5 marks] ### Answer **False in general.** Vocabulary is the set of distinct tokens under a chosen tokenizer. In “Cats cats running”, raw case-sensitive vocabulary={Cats,cats,running}, size3. Lowercase merges Cats/cats: {cats,running}, size2. Lemmatization may yield {cat,run}; stopword removal can remove entries. For an already normalized corpus the set may happen to remain unchanged, so the correct claim is that normalization can change it, not that it always must. --- ## TEE · Q1b · Good corpus characteristics Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q1b Previous examination · complete worked answers **Question. State the characteristics of a good corpus.** [5 marks] ### Answer A useful corpus is suited to its task and population: representative genres/languages, adequate size and diversity, balanced or intentionally documented sampling, reliable consistent annotations, clean encoding/tokenization, metadata and provenance, legal/ethical permission, documented train/test splits, and versioned reproducibility. “Large” alone does not imply representative. Example: a medical-tagging corpus needs clinical language and negation annotation; only movie reviews would be unsuitable. --- ## TEE · Q1c · Word2Vec versus Doc2Vec Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q1c Previous examination · complete worked answers **Question. Differentiate Word2Vec and Doc2Vec.** [5 marks] ### Answer | Aspect | Word2Vec | Doc2Vec | | --- | --- | --- | | Representation | One learned vector per word type | Learns document/paragraph vectors as well as, in variants, word vectors | | Training | CBOW predicts centre from context; Skip-gram predicts context from centre | PV-DM combines document/context to predict words; PV-DBOW predicts sampled words from document ID | | Unit | Word/token type | Sentence, paragraph or document | | New text | Retrieve known word vectors; combine via a separate pooling choice | Infer a new document vector while learned model weights stay fixed | | Use | Word similarity, features | Document similarity, retrieval or classification | Averaging Word2Vec word vectors is a possible document representation, but it is not the Doc2Vec training algorithm. This older paper names Doc2Vec explicitly; the current course names word and sentence embeddings but does not explicitly list Doc2Vec. --- ## TEE · Q1d · Ambiguity in NLP Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q1d Previous examination · complete worked answers **Question. Explain the types of ambiguity in NLP with examples.** [5 marks] ### Answer **Ambiguity** means an expression supports more than one interpretation. Good exam answers name the ambiguity and explain both readings. | Challenge | Example | Why a system can fail | | --- | --- | --- | | Lexical/sense ambiguity | bank: finance or river edge | One written word has several senses | | Syntactic ambiguity | I saw the man with a telescope. | The telescope may modify saw or the man | | Reference ambiguity | Riya told Neha that she won. | She may refer to either person without more context | | Sarcasm | Great, another delay. | Literal positive word conveys frustration | | Negation | The film was not good. | Dropping not reverses meaning | | Domain shift | “Positive” in a medical report | Meaning/use differs by domain | | Code mixing | Kal meeting reschedule karna. | Several languages and spelling conventions | | Noisy text | tmrw, repeated letters, typos | Vocabulary/token boundaries become uncertain | | Low-resource setting | Few labelled examples in a language | Models lack representative training evidence | | Bias and privacy | Skewed data or personal details | Errors and harms can persist in deployed systems | **Homophones** sound alike but may differ in spelling (pair/pear). **Homographs** share spelling but can differ in meaning or pronunciation (lead the team / lead metal). These are different dimensions from syntactic ambiguity. Long-distance dependencies, cultural knowledge, idioms, morphology and multilingual writing systems also complicate processing. For “kick the bucket”, literal word-by-word interpretation can miss the idiomatic meaning. For “The book that the students recommended is useful”, the verb is agrees with book across an intervening phrase. Exam mein challenge ka naam + ek sentence + dono meanings/actual failure likho. **Structural ambiguity** occurs when a sentence has more than one valid parse tree. In “I saw the man with a telescope”, the prepositional phrase “with a telescope” can attach to saw (the instrument I used) or to the man (the man possessing the telescope). The words and their order stay the same, but the grouping changes. | Reading | Attachment | Meaning | | --- | --- | --- | | I [saw the man] [with a telescope] | PP modifies VP | I used a telescope to see the man. | | I saw [the man with a telescope] | PP modifies NP | The man had a telescope. | A parser may need context, probabilities or semantic constraints to choose between trees. A plain CFG describes permitted structures but does not itself assign a probability to each tree. Do not invent rule probabilities for an ordinary CFG question. Lexical ambiguity also matters: book can be a noun or verb. In “Book the flight”, the imperative analysis begins S→VP, followed by VP→V NP. A parser that initially chooses book as a noun may reach a dead end and must try another permitted analysis. **Yaad rakho:** Same words, alag grouping → alag meaning. --- ## TEE · Q2a · Normalize the Tanisha–Simran passage Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q2a Previous examination · complete worked answers **Question. Perform sentence segmentation, word tokenization, stopword removal, lowercasing and stemming on the following paragraph; show vocabulary before and after: “Tanisha and Simran are best friends. They play together with other friends. Tanisha likes to play badminton but Simran prefers to play online games. Tanisha wants to be a badminton player. Simran wants to become an online gamer.”** [10 marks] ### Answer No stoplist/stemmer is specified. For a reproducible answer choose stoplist {and,are,they,with,other,to,but,be,a,an} after lowercase; use the Porter stemmer. Other declared lists produce different valid counts. | Stage | Result | | --- | --- | | Sentence segmentation | Tanisha and Simran are best friends. They play together with other friends. Tanisha likes to play badminton but Simran prefers to play online games. Tanisha wants to be a badminton player. Simran wants to become an online gamer. | | Word tokens (punctuation omitted) | Tanisha, and, Simran, are, best, friends, They, play, together, with, other, friends, Tanisha, likes, to, play, badminton, but, Simran, prefers, to, play, online, games, Tanisha, wants, to, be, a, badminton, player, Simran, wants, to, become, an, online, gamer | | Lowercase | tanisha, and, simran, are, best, friends, they, play, together, with, other, friends, tanisha, likes, to, play, badminton, but, simran, prefers, to, play, online, games, tanisha, wants, to, be, a, badminton, player, simran, wants, to, become, an, online, gamer | | Declared stoplist | a, an, and, are, be, but, other, they, to, with | | After exact stop removal | tanisha, simran, best, friends, play, together, friends, tanisha, likes, play, badminton, simran, prefers, play, online, games, tanisha, wants, badminton, player, simran, wants, become, online, gamer | | Retained word | Porter stem | | --- | --- | | tanisha | tanisha | | simran | simran | | best | best | | friends | friend | | play | play | | together | togeth | | likes | like | | badminton | badminton | | prefers | prefer | | online | onlin | | games | game | | wants | want | | player | player | | become | becom | | gamer | gamer | Full final token sequence: tanisha, simran, best, friend, play, togeth, friend, tanisha, like, play, badminton, simran, prefer, play, onlin, game, tanisha, want, badminton, player, simran, want, becom, onlin, gamer Raw case-sensitive vocabulary (25): Simran, Tanisha, They, a, an, and, are, badminton, be, become, best, but, friends, gamer, games, likes, online, other, play, player, prefers, to, together, wants, with. After lowercase only: 25 distinct words. Final stem vocabulary (15): badminton, becom, best, friend, game, gamer, like, onlin, play, player, prefer, simran, tanisha, togeth, want. **Source / convention note.** Stoplist and stemming algorithm are choices, not supplied numerical givens; both are stated. --- ## TEE · Q2b · Text vectorization alternatives Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q2b Previous examination · complete worked answers **Question. Define word embeddings and explain one-hot, Bag of Words, bag of n-grams and TF–IDF with advantages and disadvantages.** [10 marks] ### Answer **A sparse representation** has many zero entries and often one coordinate per vocabulary item. BoW and TF–IDF are common sparse document representations. **A dense embedding** has comparatively few learned real-valued coordinates; one word's properties are spread across many coordinates. | Representation | What coordinates mean | Strength | Limitation | | --- | --- | --- | --- | | One-hot | One identity position per vocabulary word | Unambiguous ID | Different words are orthogonal; no learned similarity | | Count-based context vector | Context-word co-occurrence counts | Transparent corpus evidence | Large, sparse; frequent words may dominate | | TF–IDF document vector | Weighted vocabulary terms | Useful retrieval weighting | Word order and contextual meaning are limited | | Dense word embedding | Learned latent features | Compact semantic/syntactic relationships | Coordinates are not generally human-labelled features | **Distributional** describes learning from patterns of linguistic context. **Distributed** describes representing an item across multiple coordinates rather than a single active identity bit. These are related but different ideas. A one-hot ID alone does not learn distributional similarity. ### Model design choices Choose contexts (document, sentence, fixed window), weighting (raw counts, normalization, TF–IDF-like weighting or association scores), dimensionality reduction or learned embeddings, and a similarity measure. Cosine compares direction and is less affected by length than a raw dot product; Euclidean distance also reflects magnitude. Sparse = bahut zero. Dense = chhoti learned number-list. Similar context se useful similarity seekhte hain. | Method | Construction and example | Strength | Limitation | | --- | --- | --- | --- | | One-hot | One1in vocabulary coordinate: cat=(1,0,0) | Simple unique identity | Sparse; all distinct words equally distant | | BoW | Counts per vocabulary coordinate | Simple interpretable frequency | Loses order and context | | Bag of n-grams | Counts contiguous word groups, e.g. not good | Retains local order/phrases | Vocabulary grows; sparse unseen phrases | | TF–IDF | Term frequency weighted by inverse document frequency | Downweights ubiquitous terms for retrieval | No contextual meaning; depends on corpus statistics | Dense learned embeddings differ from these sparse count encodings: similarity may reflect learned context. Explain the representation before computing any metric. --- ## TEE · Q3a · Semantics, pragmatics and two ambiguities Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q3a Previous examination · complete worked answers **Question. Differentiate semantics and pragmatics. Explain ambiguity in “Time flies like an arrow” and “He crushed the key to my heart”.** [10 marks] ### Answer **Semantics** concerns conventional meaning and composition; **pragmatics** interprets an utterance using speaker intention and situation. “It is cold here” can semantically report temperature and pragmatically request closing a window. | Sentence | Readings / cause | | --- | --- | | Time flies like an arrow | Ordinary reading: time(N) flies(V), with like an arrow as comparison. Alternative imperative: time(V) the flies(N) as one times an arrow; lexical-category and structural ambiguity. | | He crushed the key to my heart | Literal key/crushing versus figurative destruction of affection or emotional access. Lexical/semantic ambiguity and metaphor need pragmatic context; there is no supplied situation proving a single intended reading. | --- ## TEE · Q3b · Naive Bayes text classifier Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q3b Previous examination · complete worked answers **Question. Explain Naive Bayes text classification with a suitable example.** [10 marks] ### Answer The paper supplies no numerical training corpus for this part. The following clearly labelled illustrative corpus demonstrates the complete calculation. ### Given | Training text | Class | | --- | --- | | Chinese Beijing Chinese | yes | | Chinese Chinese Shanghai | yes | | Chinese Macao | yes | | Tokyo Japan Chinese | no | Test: Chinese Chinese Chinese Tokyo Japan ### To find Classify the test using multinomial Naive Bayes. ### Formula: which one and why Assume add-one smoothing unless the question supplies probabilities. Keep repeated words as repeated factors. Use training vocabulary only; ignore out-of-vocabulary words unless an unknown-token model is defined. $$ \mathrm{score}(c)=P(c)\prod_{w\in V}\left(\frac{c(w,c)+1}{N_c+|V|}\right)^{c(w,d)} $$ ### Step 1. Vocabulary and class priors V = {beijing, chinese, japan, macao, shanghai, tokyo}; |V|=6. | Class | Documents | Prior | | --- | --- | --- | | no | 1 | 1/4 | | yes | 3 | 3/4 | ### Step 2. Count every training word within each class | Class | Total tokens | beijing | chinese | japan | macao | shanghai | tokyo | | --- | --- | --- | --- | --- | --- | --- | --- | | no | 3 | 0 | 1 | 1 | 0 | 0 | 1 | | yes | 8 | 1 | 5 | 0 | 1 | 1 | 0 | ### Step 3. Count the test words | Word | Count | | --- | --- | | chinese | 3 | | tokyo | 1 | | japan | 1 | OOV words: none ### Step 4. Calculate likelihoods and multiply with the prior | Class | Prior | Individual factors | Exact score | Decimal score | | --- | --- | --- | --- | --- | | no | 1/4 | chinese: (1+1)/(3+1×6) = 2/9, used 3 time(s); tokyo: (1+1)/(3+1×6) = 2/9, used 1 time(s); japan: (1+1)/(3+1×6) = 2/9, used 1 time(s) | 8/59049 | 0.000135480702467 | | yes | 3/4 | chinese: (5+1)/(8+1×6) = 3/7, used 3 time(s); tokyo: (0+1)/(8+1×6) = 1/14, used 1 time(s); japan: (0+1)/(8+1×6) = 1/14, used 1 time(s) | 81/268912 | 0.000301213779973 | ### Step 5. Compare and optionally normalize Normalized scores: no = 0.310241388, yes = 0.689758612 Division by the same total does not change the winning class. ### Final answer **Predicted class: yes.** **Yaad rakho:** Count → prior → smoothed word chances → repeated factors → compare. **Source / convention note.** Chinese corpus is an authored/reused teaching illustration here; it is actual given data in F22 Q5b, not this prompt. --- ## TEE · Q4a · Jaccard, cosine and Euclidean without preprocessing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q4a Previous examination · complete worked answers **Question. Explain Jaccard, cosine and Euclidean measures. Calculate them without preprocessing for D₁=“Data is the new oil of the digital economy” and D₂=“Data is a new oil”.** [10 marks] ### Answer ### Given Use exactly the given case and words; do not remove stopwords. ### To find Three measures in explicitly stated representations. ### Formula: which one and why $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|},\quad d_E(u,v)=\sqrt{\sum_i(u_i-v_i)^2} $$ Jaccard uses sets of distinct tokens. Cosine and Euclidean use raw-count BoW vectors in one shared order; do not secretly turn repeated the into a single count. ### Step 1. Build the common vocabulary and counts | Document | Data | is | the | new | oil | of | digital | economy | a | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | D1 | 1 | 1 | 2 | 1 | 1 | 1 | 1 | 1 | 0 | | D2 | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 1 | ### Step 2. Jaccard sets Intersection={Data,is,new,oil}, size4; union size9. J=4/9=0.444444; Jaccard distance=1−J=5/9. ### Step 3. Cosine Dot product=4. D1 squared length=1+1+4+1+1+1+1+1=11; D2 squared length=5. Cosine=4/√55=0.539359890. ### Step 4. Euclidean Coordinate differences: [0, 0, 2, 0, 0, 1, 1, 1, -1]. Sum of squares=8. Distance=√8=2.828427125. Smaller distance means greater proximity; this is a distance, not a probability. ### Final answer **Jaccard=0.444444; cosine=0.539360; Euclidean distance=2.828427.** --- ## TEE · Q4b · CBOW and Skip-gram Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q4b Previous examination · complete worked answers **Question. Explain the CBOW and Skip-gram architectures.** [10 marks] ### Answer **Word2Vec learns dense word embeddings by predicting words from nearby context.** Its two common architectures are CBOW and skip-gram. A vocabulary ID or one-hot vector is the input representation; the learned embedding table converts that identity into a short trainable vector. | Architecture | Input | Prediction | Memory cue | | --- | --- | --- | --- | | CBOW | Surrounding context words | Centre/target word | Context → centre | | Skip-gram | Centre/target word | Surrounding words | Centre → context | CBOW: the + sat → Average context embeddings → Predict: cat Skip-gram: cat → Embedding of cat → Predict: the and sat For vocabulary size $$ V $$ and embedding dimension $$ d $$ , an input one-hot row is $$ 1\times V $$ ; the input embedding matrix is $$ V\times d $$ ; the hidden row is $$ 1\times d $$ ; the output matrix is $$ d\times V $$ ; output scores are $$ 1\times V $$ . Multiplying a one-hot row by the embedding matrix simply selects one row. $$ P(w_o\mid w_i)=\frac{\exp(u_{w_o}^{T}v_{w_i})}{\sum_{w\in V}\exp(u_w^{T}v_{w_i})}. $$ $$ v_{w_i} $$ is the input word embedding, $$ u_w $$ an output-context embedding, and the dot product is a compatibility score. Softmax turns all scores into probabilities that sum to one. Training increases the probability of observed context words. Negative sampling or hierarchical softmax can reduce computation in larger vocabularies; use the algorithm specified by the question. **Question. For the sentence “the cat sat on mat”, generate CBOW and skip-gram training data for centre word sat with a window of one token on each side.** #### Given Tokens indexed 1 to 5: the, cat, sat, on, mat. Centre = sat. #### To find The CBOW example and the skip-gram pairs. #### Formula and why it applies CBOW predicts the centre from its neighbours; skip-gram predicts each neighbour from the centre. #### Step 1. Identify neighbours Left neighbour = cat; right neighbour = on. “the” and “mat” are outside this window. #### Step 2. CBOW example Input [cat,on] → target sat. #### Step 3. Skip-gram examples (sat,cat) and (sat,on), with first item as input and second as predicted context. **Answer.** CBOW: [cat,on]→sat. Skip-gram: sat→cat and sat→on. **Question. The embeddings of context words cat and on are [1,2] and [3,0]. Compute the CBOW hidden vector using mean aggregation. Output vectors for [sat,ran] are [1,0] and [0,1]. Compute softmax probabilities.** #### Given Context embeddings [1,2], [3,0]; mean aggregation; two candidate output vectors. #### To find The hidden vector, logits and probabilities. #### Formula and why it applies Mean context vector $$ h=(v_1+v_2)/2 $$ . Logit $$ z_w=u_w^Th $$ . Softmax $$ P(w)=e^{z_w}/\sum_j e^{z_j} $$ . #### Step 1. Add coordinates $$ [1+3,\,2+0]=[4,2] $$ . #### Step 2. Divide by number of words $$ h=[4/2,2/2]=[2,1] $$ . #### Step 3. Compute logits $$ z_{sat}=1(2)+0(1)=2 $$ . $$ z_{ran}=0(2)+1(1)=1 $$ . #### Step 4. Exponentials $$ e^2\approx7.389056 $$ , $$ e^1\approx2.718282 $$ ; sum $$ \approx10.107338 $$ . #### Step 5. Divide $$ P(sat)\approx0.731059 $$ , $$ P(ran)\approx0.268941 $$ . **Answer.** h=[2,1]; sat is more probable (0.731059). **Question. A skip-gram centre embedding is v=[1,−1]. Output embeddings for [eat,sleep,drive] are [1,1], [0,1], [1,0]. Calculate the predicted context probabilities.** #### Given One centre vector and three output vectors; full softmax over these three words. #### To find Three logits and three probabilities. #### Formula and why it applies $$ z_j=u_j^Tv $$ , then $$ P(j)=e^{z_j}/\sum_k e^{z_k} $$ . #### Step 1. Eat dot product $$ 1(1)+1(-1)=0 $$ . #### Step 2. Sleep and drive dot products $$ z_{sleep}=0(1)+1(-1)=-1 $$ . $$ z_{drive}=1(1)+0(-1)=1 $$ . #### Step 3. Exponentiate and sum $$ e^0=1 $$ , $$ e^{-1}\approx0.367879 $$ , $$ e^1\approx2.718282 $$ . Sum $$ \approx4.086161 $$ . #### Step 4. Normalize $$ P(eat)\approx0.244728 $$ ; $$ P(sleep)\approx0.090031 $$ ; $$ P(drive)\approx0.665241 $$ . **Answer.** drive has the highest probability in this toy learned model. **Limitations:** a standard static Word2Vec vector merges a word's senses; it has limited representation of subword morphology and out-of-vocabulary words; its local prediction objective does not directly fit a full global co-occurrence table. Similarity is learned from data and is not a guarantee of factual or causal understanding. CBOW = neighbours se missing centre. Skip-gram = centre se neighbours. --- ## TEE · Q5a · Password regex Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q5a Previous examination · complete worked answers **Question. Write a regex for an 8–15 character password containing letters and digits, at least one uppercase letter and at least one special character from !%@\*, with no other characters.** [10 marks] ### Answer `^(?=.{8,15}$)(?=.*[A-Z])(?=.*[0-9])(?=.*[!%@*])[A-Za-z0-9!%@*]+$` | Part | Purpose | | --- | --- | | (?=.{8,15}$) | Length8–15 | | (?=.\*[A-Z]) | At least one uppercase; this also meets the letter requirement | | (?=.\*[0-9]) | At least one digit | | (?=.\*[!%@\*]) | At least one permitted special | | [A-Za-z0-9!%@\*]+ | Only permitted characters, including lowercase if desired | Use full-string matching. Abcdef1! passes; abcdef1! fails uppercase, Abcdefgh fails digit/special, Abc1! is too short, Abcdef1# contains forbidden #. A lowercase character is not separately required by the printed conditions. --- ## TEE · Q5b · Two NLP applications Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q5b Previous examination · complete worked answers **Question. Briefly explain any two applications of NLP.** [10 marks] ### Answer ### 1. Machine translation Translate meaning from source to target language using a model conditioned on the source sequence. Example: English “Where is the station?” to Hindi; word order and ambiguity require context. Train on parallel corpora and evaluate adequacy/fluency. ### 2. Information extraction Convert text into structured entities and relations. Example: “Acme acquired Beta in 2022” becomes acquirer=Acme, acquired=Beta, year=2022. Pipeline: tokenize → NER → relation extraction → validate output with source spans. --- ## TEE · Q6a · Mention-pair model Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q6a Previous examination · complete worked answers **Question. Explain the mention-pair model for reference resolution.** [10 marks] ### Answer A mention is a span referring to an entity. A mention-pair model classifies a pair (earlier candidate antecedent, later mention) as coreferent or not. Pairwise decisions are subsequently linked into entity clusters; a positive score alone does not ensure globally consistent clusters. Detect mentions → Generate earlier candidate pairs → Extract pair features → Score same-entity relation → Select/link antecedents → Form entity clusters 1. Text: “Rita bought a car. She likes it.” Detect Rita, a car, She, it. 2. For She, compare (Rita,She) and (a car,She). Features include person/number compatibility, animate type, syntactic role, sentence distance and lexical agreement. 3. Rita–She has compatible singular human features; car–She fails the ordinary inanimate reading. Select Rita. 4. For it, compare earlier mentions: car–it is singular/inanimate and contextually fits likes. Cluster {Rita,She} and {a car,it}. 5. Train a classifier on labelled mention pairs; at prediction, choose a sufficiently supported antecedent or leave a new entity. Avoid reflexive/binding violations and document leakage during evaluation. Advantages: interpretable pair features and supervised learning. Limitations: quadratic candidate growth, error propagation from mention detection, non-transitive pair predictions, and weak long-range/world-knowledge reasoning. A greedy nearest-candidate choice is not the same as a learned mention-pair model. --- ## TEE · Q6b · Classification evaluation including ROC/AUC Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q6b Previous examination · complete worked answers **Question. Explain accuracy, precision, recall, F1 score, ROC and AUC.** [10 marks] ### Answer Evaluation compares predicted results with known answers. A **true positive** is a correctly predicted positive; a **false positive** is a negative item predicted positive; a **false negative** is a missed positive; a **true negative** is a correctly rejected negative. $$ \text{Precision}=\frac{TP}{TP+FP} $$ $$ \text{Recall}=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2PR}{P+R}=\frac{2TP}{2TP+FP+FN} $$ $$ \text{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN} $$ Precision asks how many predicted positives were correct. Recall asks how many actual positives were found. F1 balances them by a harmonic mean. If a denominator is zero, state the convention rather than dividing by zero. ### Example 1 · evaluate a classifier A binary text classifier gives TP=6, FP=2, FN=4, TN=8. Compute precision, recall, F1 and accuracy. **Given** TP=6; FP=2; FN=4; TN=8. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{6}{6+2}=0.750000 $$ #### Step 2. Recall $$ R=\frac{6}{6+4}=0.600000 $$ #### Step 3. F1 $$ F_1=\frac{12}{12+2+4}=0.666667 $$ #### Step 4. Accuracy $$ A=\frac{6+8}{6+2+4+8}=0.700000 $$ Precision 0.7500; recall 0.6000; F1 0.6667; accuracy 0.7000. ### Example 2 · evaluate a classifier A binary text classifier gives TP=9, FP=3, FN=1, TN=7. Compute precision, recall, F1 and accuracy. **Given** TP=9; FP=3; FN=1; TN=7. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{9}{9+3}=0.750000 $$ #### Step 2. Recall $$ R=\frac{9}{9+1}=0.900000 $$ #### Step 3. F1 $$ F_1=\frac{18}{18+3+1}=0.818182 $$ #### Step 4. Accuracy $$ A=\frac{9+7}{9+3+1+7}=0.800000 $$ Precision 0.7500; recall 0.9000; F1 0.8182; accuracy 0.8000. ### Example 3 · evaluate a classifier A binary text classifier gives TP=4, FP=0, FN=4, TN=12. Compute precision, recall, F1 and accuracy. **Given** TP=4; FP=0; FN=4; TN=12. **To find** Four evaluation measures. ### Formula and why it applies $$ P=\frac{TP}{TP+FP},\quad R=\frac{TP}{TP+FN} $$ $$ F_1=\frac{2TP}{2TP+FP+FN},\quad A=\frac{TP+TN}{TP+FP+FN+TN} $$ Use predicted positives in precision, actual positives in recall, and all examples in accuracy. ### Solution #### Step 1. Precision $$ P=\frac{4}{4+0}=1.000000 $$ #### Step 2. Recall $$ R=\frac{4}{4+4}=0.500000 $$ #### Step 3. F1 $$ F_1=\frac{8}{8+0+4}=0.666667 $$ #### Step 4. Accuracy $$ A=\frac{4+12}{4+0+4+12}=0.800000 $$ Precision 1.0000; recall 0.5000; F1 0.6667; accuracy 0.8000. ### ROC and AUC $$ \mathrm{TPR}=\frac{TP}{TP+FN},\qquad\mathrm{FPR}=\frac{FP}{FP+TN} $$ A classifier produces scores. Sweep the decision threshold; at each threshold calculate TPR and FPR. Plot FPR horizontally and TPR vertically. AUC is the area under that curve, equivalently the probability that a randomly chosen positive has a higher score than a randomly chosen negative (ties conventionally contribute half). AUC1 is perfect ranking;0.5 is chance ranking. This evaluates ranking over thresholds, not the accuracy at one threshold. | Illustrative threshold | TPR | FPR | | --- | --- | --- | | Above all scores | 0 | 0 | | Middle threshold | 0.8 | 0.2 | | Below all scores | 1 | 1 | For straight segments through (0,0),(.2,.8),(1,1), trapezoid area=.2×(0+.8)/2+.8×(.8+1)/2=.08+.72=.80. These points illustrate integration; they are not supplied exam data. Precision–recall may be more informative for rare positives. --- ## TEE · Q7a · Organization and facility mentions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q7a Previous examination · complete worked answers **Question. Identify organization and facility mentions in this White House visitor passage using the supplied ACE categories: “After the 9/11 attacks, the system changed radically. Now, anyone who wants to tour the White House must apply through the office of his or her representative in Congress, which forwards the names to the White House for clearance. Once they get the green light, visitors show up at the appointed time on 15th Street between E and F Streets and join the line to enter through the southeast gate. Anyone who has flown on an airline in recent years will recognize the familiar territory of identity checks and electronic scans, although you do get to keep your shoes on. At the head of the line, rangers from the National Park Service check photo IDs against a list of names.”** [10 marks] ### Answer | Mention in order | Type | Reason | | --- | --- | --- | | White House (tour) | FAC | Physical building being toured | | office of his or her representative | ORG under administrative-office reading | Staff/organizational office receiving applications; no visit to the physical office is described | | Congress | ORG | Legislative organization | | White House (clearance) | ORG | Institution/staff granting clearance, a metonymic use | | 15th Street | FAC | Transport infrastructure | | E and F Streets | FAC (two coordinated streets) | Roads, not organizations | | southeast gate | FAC | Physical entrance component | | an airline | ORG generic mention | Organization operating flights; generic, not a proper named company | | National Park Service | ORG | Government agency | The prompt uses ACE entity mentions: nominal/generic mentions can be included, unlike a named-only tagger. 9/11 is an event/date mention, not ORG/FAC; visitors and rangers are people. If a narrower classroom guideline asks only named entities, omit the generic office/airline/gate spans and explicitly state that convention. Context decides whether White House means building or administration. **Source / convention note.** Boundary and generic-mention decisions depend on the provided ACE annotation convention; the answer distinguishes them from a named-only interpretation. --- ## TEE · Q7b · Tagging and two complete parse trees Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-r23/q7b Previous examination · complete worked answers **Question. Explain POS tagging. Tag and parse (1) John and Mary bought a refrigerator with three doors. (2) It was purchased from a very small store near their house.** [10 marks] ### Answer POS tagging assigns grammatical categories in context; constituency parsing groups those words hierarchically. Here with three doors modifies refrigerator; near their house modifies store. These attachments fit the natural reading; the paper gives no fixed CFG. `John/NNP and/CC Mary/NNP bought/VBD a/DT refrigerator/NN with/IN three/CD doors/NNS` SNPNNPJohnCCandNNPMaryVPVBDboughtNPDTaNNrefrigeratorPPINwithNPCDthreeNNSdoors [S [NP [NNP John] [CC and] [NNP Mary]] [VP [VBD bought] [NP [DT a] [NN refrigerator] [PP [IN with] [NP [CD three] [NNS doors]]]]]] `It/PRP was/VBD purchased/VBN from/IN a/DT very/RB small/JJ store/NN near/IN their/PRP$ house/NN` SNPPRPItVPVBDwasVPVBNpurchasedPPINfromNPDTaADJPRBveryJJsmallNNstorePPINnearNPPRP$theirNNhouse [S [NP [PRP It]] [VP [VBD was] [VP [VBN purchased] [PP [IN from] [NP [DT a] [ADJP [RB very] [JJ small]] [NN store] [PP [IN near] [NP [PRP$ their] [NN house]]]]]]]] The first subject is coordinated; bought is past tense. The second VP is passive auxiliary was plus past participle purchased. Very modifies small, and their is a possessive determiner. --- ## TEE · Q1a · Need and difficulties of NLP Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q1a Previous examination · complete worked answers **Question. Explain the need for NLP and the difficulties in natural language processing.** [5 marks] ### Answer **Natural language processing (NLP) is the field concerned with computational methods for analysing, interpreting and generating human language.** Natural languages such as English, Hindi and Marathi evolved through human use; their meanings are not specified by the rigid rules of a programming language. An NLP system may receive text, speech converted to text, or another linguistic representation. Its output depends on the task: a category, a tag for every word, a translated sentence, a ranked document list or a generated answer. It does not always “understand” in the human sense; its performance must be evaluated for the specific task. | Task | Input | Output | | --- | --- | --- | | Sentiment analysis | “The service was excellent.” | Positive sentiment | | Text classification | An email | Spam / not spam | | POS tagging | “Birds fly.” | Birds/NOUN fly/VERB | | Named entity recognition | “Asha joined Google in Mumbai.” | Asha: person; Google: organisation; Mumbai: location | | Machine translation | A Hindi sentence | A corresponding English sentence | | Information retrieval | A search query | Ranked relevant documents | | Question answering | Question plus knowledge/context | An answer | | Summarization | A long article | A shorter text preserving key information | ### How NLP relates to AI, ML and DL **AI** is the broader study of systems performing tasks associated with intelligent behaviour. **Machine learning** learns patterns from data. **Deep learning** uses multiple trainable neural layers. NLP is an application/research field that can use hand-written rules, statistical ML or deep learning. Therefore, not every NLP system is a neural network. Linguistics supplies knowledge about language structure. Statistics supplies uncertainty models. Computer science supplies algorithms, storage and efficient implementation. A practical NLP system combines these ideas. NLP = human language ko computational task mein badlo: label, meaning, search ya generated sentence. **Ambiguity** means an expression supports more than one interpretation. Good exam answers name the ambiguity and explain both readings. | Challenge | Example | Why a system can fail | | --- | --- | --- | | Lexical/sense ambiguity | bank: finance or river edge | One written word has several senses | | Syntactic ambiguity | I saw the man with a telescope. | The telescope may modify saw or the man | | Reference ambiguity | Riya told Neha that she won. | She may refer to either person without more context | | Sarcasm | Great, another delay. | Literal positive word conveys frustration | | Negation | The film was not good. | Dropping not reverses meaning | | Domain shift | “Positive” in a medical report | Meaning/use differs by domain | | Code mixing | Kal meeting reschedule karna. | Several languages and spelling conventions | | Noisy text | tmrw, repeated letters, typos | Vocabulary/token boundaries become uncertain | | Low-resource setting | Few labelled examples in a language | Models lack representative training evidence | | Bias and privacy | Skewed data or personal details | Errors and harms can persist in deployed systems | **Homophones** sound alike but may differ in spelling (pair/pear). **Homographs** share spelling but can differ in meaning or pronunciation (lead the team / lead metal). These are different dimensions from syntactic ambiguity. Long-distance dependencies, cultural knowledge, idioms, morphology and multilingual writing systems also complicate processing. For “kick the bucket”, literal word-by-word interpretation can miss the idiomatic meaning. For “The book that the students recommended is useful”, the verb is agrees with book across an intervening phrase. Exam mein challenge ka naam + ek sentence + dono meanings/actual failure likho. --- ## TEE · Q1b · Levels of language analysis Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q1b Previous examination · complete worked answers **Question. Explain the levels of language analysis with examples.** [5 marks] ### Answer **A phoneme** is a sound unit that can distinguish meanings in a language, such as the /p/ and /b/ contrast in “pat” and “bat”. A phoneme is a sound, not necessarily one alphabet letter. **A morpheme** is the smallest meaning-bearing grammatical unit: cats = cat + plural -s; unhappy = un- + happy. **A lexeme** is an abstract vocabulary item grouping its inflected forms: write, writes, wrote and writing belong to WRITE. A **phrase** is a group of words functioning together; a **clause** normally contains a predicate and its associated participants; a **sentence** expresses a complete grammatical unit. | Level | What it analyses | Concrete example | | --- | --- | --- | | Phonetics / phonology | Physical sounds / sound patterns in a language | Distinguish bat from pat | | Morphology / lexical analysis | Word structure and lexical items | un + help + ful → unhelpful | | Syntax | Grammatical arrangement and relationships | The dog chased the cat: subject, verb, object | | Semantics | Meaning of words and composed expressions | The dog is the chaser; the cat is chased | | Discourse | Connections across sentences | “Riya arrived. She smiled.” → She refers to Riya | | Pragmatics | Intended meaning in the situation | “It is cold here” may request closing the window | word forms → grammatical structure → sentence meaning → cross-sentence context → speaker intention ### Three worked interpretations **“The cats are sleeping.”** Morphology identifies cat + s as plural. Syntax connects “cats” to “are sleeping”. Semantics represents animals engaged in sleeping. **“Aman bought a laptop. It was expensive.”** Sentence-level syntax alone does not identify “it”. Discourse connects “it” to the laptop across the sentence boundary. **“Wonderful, another power cut!”** The word wonderful is positive literally, but situational context can indicate sarcasm and frustration. This is a pragmatic interpretation. The analysis levels are conceptual descriptions. Modern systems may learn several jointly rather than executing a compulsory separate module for every level. Morphology = word ke andar; syntax = sentence ka structure; semantics = meaning; discourse = sentences ka link; pragmatics = intention. --- ## TEE · Q1c · Medical chatbot Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q1c Previous examination · complete worked answers **Question. Briefly explain a medical chatbot and its NLP processing.** [5 marks] ### Answer A medical chatbot accepts a patient’s natural-language message and provides an appropriately bounded response, such as appointment information or approved health education. It must distinguish intents and preserve symptoms, negations and durations. Patient message → Tokenization + intent/entity detection → Retrieve approved information → Generate grounded response → Escalate to clinician when needed Example: “I have no fever but a cough for three days” yields cough present, fever absent and duration three days. A safe administrative bot can suggest an appointment; it must not turn uncertain text into a definitive diagnosis. Evaluate intent accuracy, entity/negation accuracy, response correctness, privacy and escalation behaviour. --- ## TEE · Q1d · Sentiment with logistic regression Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q1d Previous examination · complete worked answers **Question. Explain sentiment analysis using logistic regression with an example.** [5 marks] ### Answer Binary logistic regression maps text features x to a probability: $$ z=w^Tx+b,\qquad P(y=1\mid x)=\sigma(z)=\frac{1}{1+e^{-z}} $$ Use positive-word count and negative-word count as an illustrative feature vector. With x=(2,1), w=(1,−1), b=0: z=1×2+(−1)×1=1; P(positive)=1/(1+e⁻¹)=0.731059. A declared 0.5 threshold predicts positive. Fit weights from labelled reviews by minimizing cross-entropy; the hand-chosen weights here only illustrate the computation. Labelled reviews → Tokenize / vectorize → Fit logistic weights → Probability → Threshold / class --- ## TEE · Q2a · CBOW and Skip-gram architecture Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q2a Previous examination · complete worked answers **Question. Explain CBOW and Skip-gram with suitable architecture diagrams.** [10 marks] ### Answer **Word2Vec learns dense word embeddings by predicting words from nearby context.** Its two common architectures are CBOW and skip-gram. A vocabulary ID or one-hot vector is the input representation; the learned embedding table converts that identity into a short trainable vector. | Architecture | Input | Prediction | Memory cue | | --- | --- | --- | --- | | CBOW | Surrounding context words | Centre/target word | Context → centre | | Skip-gram | Centre/target word | Surrounding words | Centre → context | CBOW: the + sat → Average context embeddings → Predict: cat Skip-gram: cat → Embedding of cat → Predict: the and sat For vocabulary size $$ V $$ and embedding dimension $$ d $$ , an input one-hot row is $$ 1\times V $$ ; the input embedding matrix is $$ V\times d $$ ; the hidden row is $$ 1\times d $$ ; the output matrix is $$ d\times V $$ ; output scores are $$ 1\times V $$ . Multiplying a one-hot row by the embedding matrix simply selects one row. $$ P(w_o\mid w_i)=\frac{\exp(u_{w_o}^{T}v_{w_i})}{\sum_{w\in V}\exp(u_w^{T}v_{w_i})}. $$ $$ v_{w_i} $$ is the input word embedding, $$ u_w $$ an output-context embedding, and the dot product is a compatibility score. Softmax turns all scores into probabilities that sum to one. Training increases the probability of observed context words. Negative sampling or hierarchical softmax can reduce computation in larger vocabularies; use the algorithm specified by the question. **Question. For the sentence “the cat sat on mat”, generate CBOW and skip-gram training data for centre word sat with a window of one token on each side.** #### Given Tokens indexed 1 to 5: the, cat, sat, on, mat. Centre = sat. #### To find The CBOW example and the skip-gram pairs. #### Formula and why it applies CBOW predicts the centre from its neighbours; skip-gram predicts each neighbour from the centre. #### Step 1. Identify neighbours Left neighbour = cat; right neighbour = on. “the” and “mat” are outside this window. #### Step 2. CBOW example Input [cat,on] → target sat. #### Step 3. Skip-gram examples (sat,cat) and (sat,on), with first item as input and second as predicted context. **Answer.** CBOW: [cat,on]→sat. Skip-gram: sat→cat and sat→on. **Question. The embeddings of context words cat and on are [1,2] and [3,0]. Compute the CBOW hidden vector using mean aggregation. Output vectors for [sat,ran] are [1,0] and [0,1]. Compute softmax probabilities.** #### Given Context embeddings [1,2], [3,0]; mean aggregation; two candidate output vectors. #### To find The hidden vector, logits and probabilities. #### Formula and why it applies Mean context vector $$ h=(v_1+v_2)/2 $$ . Logit $$ z_w=u_w^Th $$ . Softmax $$ P(w)=e^{z_w}/\sum_j e^{z_j} $$ . #### Step 1. Add coordinates $$ [1+3,\,2+0]=[4,2] $$ . #### Step 2. Divide by number of words $$ h=[4/2,2/2]=[2,1] $$ . #### Step 3. Compute logits $$ z_{sat}=1(2)+0(1)=2 $$ . $$ z_{ran}=0(2)+1(1)=1 $$ . #### Step 4. Exponentials $$ e^2\approx7.389056 $$ , $$ e^1\approx2.718282 $$ ; sum $$ \approx10.107338 $$ . #### Step 5. Divide $$ P(sat)\approx0.731059 $$ , $$ P(ran)\approx0.268941 $$ . **Answer.** h=[2,1]; sat is more probable (0.731059). **Question. A skip-gram centre embedding is v=[1,−1]. Output embeddings for [eat,sleep,drive] are [1,1], [0,1], [1,0]. Calculate the predicted context probabilities.** #### Given One centre vector and three output vectors; full softmax over these three words. #### To find Three logits and three probabilities. #### Formula and why it applies $$ z_j=u_j^Tv $$ , then $$ P(j)=e^{z_j}/\sum_k e^{z_k} $$ . #### Step 1. Eat dot product $$ 1(1)+1(-1)=0 $$ . #### Step 2. Sleep and drive dot products $$ z_{sleep}=0(1)+1(-1)=-1 $$ . $$ z_{drive}=1(1)+0(-1)=1 $$ . #### Step 3. Exponentiate and sum $$ e^0=1 $$ , $$ e^{-1}\approx0.367879 $$ , $$ e^1\approx2.718282 $$ . Sum $$ \approx4.086161 $$ . #### Step 4. Normalize $$ P(eat)\approx0.244728 $$ ; $$ P(sleep)\approx0.090031 $$ ; $$ P(drive)\approx0.665241 $$ . **Answer.** drive has the highest probability in this toy learned model. **Limitations:** a standard static Word2Vec vector merges a word's senses; it has limited representation of subword morphology and out-of-vocabulary words; its local prediction objective does not directly fit a full global co-occurrence table. Similarity is learned from data and is not a guarantee of factual or causal understanding. CBOW = neighbours se missing centre. Skip-gram = centre se neighbours. --- ## TEE · Q2b · Seven regex constructions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q2b Previous examination · complete worked answers **Question. Define regular expressions and their uses. Write expressions for (i) all alphabetic strings, (ii) lowercase alphabetic strings ending in b, (iii) a/b strings where each a is immediately preceded and followed by b, (iv) consecutive repeated words, (v) a line starting with an integer and ending with a word, (vi) strings containing the whole words grotto and raven, (vii) capture the first English word, handling opening punctuation.** [10 marks] ### Answer A regular expression specifies a pattern for matching text. NLP uses it for token extraction, cleaning, validation and rule-based entities. The following uses a Python/PCRE-like engine with lookaheads and backreferences; these extended features go beyond a strictly finite-state regular language. | Task | Pattern | Why it works | | --- | --- | --- | | All alphabetic strings | `^[A-Za-z]+$` | Only letters; + requires at least one. Use \* if empty string is allowed. | | Lowercase strings ending in b | `^[a-z]*b$` | The final b is mandatory. | | a/b strings: every a surrounded by b | `^(?!a)(?!.*aa)(?!.*a$)[ab]*$` | Forbid starting a, adjacent aa, ending a. Then every a has a b on both sides. | | Two consecutive identical words | `\b([A-Za-z]+)\s+\1\b` | Capture word, whitespace, same captured word. Case-sensitive unless ignore-case is enabled. | | Start with integer; end with word | `^[0-9]+\b.*\b[A-Za-z]+$` | Same line; signed integers need optional [+-]. | | Whole grotto and whole raven anywhere | `^(?=.*\bgrotto\b)(?=.*\braven\b).*$` | Two positive lookaheads require both whole words. Dot excludes newlines unless DOTALL. | | Capture first word despite opening punctuation | `^[^A-Za-z]*([A-Za-z]+(?:[\x27’][A-Za-z]+)*)` | Group1 is first alphabetic word, with internal apostrophes allowed. | Tests: b and bab pass rule3; a, ab, ba and baab fail. “Humbert Humbert” matches rule4; “the big bug” does not. “grottos raven” fails rule6. Regex word rules are practical ASCII conventions here, not a full linguistic tokenizer. --- ## TEE · Q3a · Text preprocessing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q3a Previous examination · complete worked answers **Question. Explain text preprocessing with applications and examples.** [10 marks] ### Answer **Tokenization divides text into units called tokens for further processing.** A token may be a word, punctuation mark, character or subword, depending on the tokenizer. **Sentence segmentation** identifies sentence boundaries. Tokenization is not necessarily just splitting on spaces. | Method | Example | Trade-off | | --- | --- | --- | | Word | “cats sleep” → [cats,sleep] | Easy to interpret; unfamiliar words can be unseen tokens | | Character | cat → [c,a,t] | Handles spelling units; creates longer sequences | | Subword | unhappiness → pieces such as [un,happi,ness] in an illustrative vocabulary | Balances vocabulary size and rare-word representation | | Sentence | “Birds fly. Fish swim.” → two sentences | Abbreviations and punctuation complicate boundaries | Actual subword splits depend on the learned vocabulary; the example is illustrative, not a universal tokenizer output. Word segmentation is especially important where spaces do not reliably separate words or in compound-rich languages. **Question. Tokenize “Dr. Rao can't come today.” into words and punctuation under this rule: preserve Dr. and can't as whole tokens, and split only the final full stop.** #### Given The tokenization rule is explicitly supplied. #### To find The resulting token sequence. #### Formula and why it applies Apply the stated boundary rule; do not assume every period or apostrophe is a split point. #### Step 1. Handle abbreviation Dr. remains one token. #### Step 2. Handle contraction can't remains one token. #### Step 3. Write sequence [Dr., Rao, can't, come, today, .] **Answer.** 6 tokens under the specified rule. Another tokenizer may legitimately produce a different sequence. ### Three difficult cases **New York** may be two tokens but one named entity. **3.14** should not normally end a sentence at the decimal point. **don't** may remain whole or split into do + n't, depending on the tokenizer. Always state the convention when calculating counts. Token = chosen unit. Space split ko universal rule mat samajh. **Normalization reduces superficial variation so comparable forms can be processed consistently.** Common operations include lowercasing, Unicode normalization, appropriate spelling correction, contraction handling, and optional stemming or lemmatization. **Stopwords** are frequent words selected for removal for a particular task, such as the or a. Removing them may reduce feature size, but removing not, no or nor can destroy sentiment meaning. Punctuation can convey sentence boundaries, emotion or mathematical structure. Lowercasing can erase differences such as US versus us. **Question. Preprocess “The CATS are not sleeping!!!” by lowercasing, removing punctuation, removing only stopwords {the,are}, and lemmatizing plural cats→cat and sleeping→sleep.** #### Given Operations and stopword set are fixed. Do not remove not. #### To find The final token list. #### Formula and why it applies Apply the requested operations in order, keeping a visible intermediate result after each transformation. #### Step 1. Lowercase the cats are not sleeping!!! #### Step 2. Tokenize and remove punctuation [the,cats,are,not,sleeping] #### Step 3. Remove specified stopwords Remove the and are → [cats,not,sleeping]. #### Step 4. Lemmatize cats→cat; sleeping→sleep → [cat,not,sleep]. **Answer.** [cat,not,sleep]. The negative meaning is retained. ### Task decides preprocessing For a simple keyword-retrieval model, removing selected function words can help. For machine translation, question answering, grammar analysis or transformer input, aggressive removal may discard necessary syntax or violate the model's expected input format. Emoji can carry sentiment; convert or retain them when relevant instead of always deleting them. Clean karna = useful signal bachake noise hatao. Har word/punctuation delete karna cleaning nahi. **Stemming** applies string-based rules, often stripping suffixes, to reduce related forms to a stem. The result need not be a valid dictionary word. **Lemmatization** uses vocabulary and morphological analysis, often with POS information, to return a dictionary lemma. | Criterion | Stemming | Lemmatization | | --- | --- | --- | | Basis | Surface form/suffix rules | Lexicon and morphological/POS analysis | | Output | May be nonword, e.g. studi | Dictionary form, e.g. study | | Context/POS | Usually little or none | Often important | | Irregular forms | May miss went→go | Can map went→go with verb analysis | | Cost | Generally simpler | Usually more linguistic processing | Porter applies a sequence of suffix-removal rules subject to conditions; Snowball provides related language-specific stemming algorithms; Lancaster is often more aggressive; regex stemming uses explicit patterns; Lovins is another established stemming approach. Exact output depends on the algorithm and implementation. ### Three worked reductions 1. **studies, studying:** an English stemmer may yield studi/study depending on the rule sequence; verb lemmatization aims at study. State the exact algorithm if exact outputs are asked. 2. **better:** adjective-aware lemmatization can map it to good; naïve suffix removal cannot infer that irregular relation. 3. **saw:** as a past-tense verb it can become see; as a cutting-tool noun it remains saw. POS and context matter. **Over-stemming** merges forms that should remain distinct; **under-stemming** fails to merge related forms. Normalization can increase recall in search while losing useful distinctions, so it is a task-dependent choice. Stem = rules se cut; lemma = dictionary form, meaning/POS ka dhyan. --- ## TEE · Q3b · Penn Treebank tagging Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q3b Previous examination · complete worked answers **Question. Using Penn Treebank tags, tag every word and ignore punctuation: (1) It is a nice night. (2) This crap game is over a garage in Fifty-second Street. (3) Nobody ever takes the newspapers she sells. (4) I am sitting in Mindy’s restaurant putting on the gefilte fish, which is a dish I am very fond of. (5) What flights do you have from Milwaukee to Tampa?** [10 marks] ### Answer Read the role of each word in its sentence. Split possessive ’s into POS; punctuation has no requested tag. The original supplies a Penn-tag annex. ### Sentence 1 `It/PRP is/VBZ a/DT nice/JJ night/NN` ### Sentence 2 `This/DT crap/NN game/NN is/VBZ over/IN a/DT garage/NN in/IN Fifty-second/JJ Street/NNP` ### Sentence 3 `Nobody/NN ever/RB takes/VBZ the/DT newspapers/NNS she/PRP sells/VBZ` ### Sentence 4 `I/PRP am/VBP sitting/VBG in/IN Mindy/NNP ’s/POS restaurant/NN putting/VBG on/RP the/DT gefilte/JJ fish/NN which/WDT is/VBZ a/DT dish/NN I/PRP am/VBP very/RB fond/JJ of/IN` ### Sentence 5 `What/WDT flights/NNS do/VBP you/PRP have/VB from/IN Milwaukee/NNP to/IN Tampa/NNP` | Tag | Meaning | Why here | | --- | --- | --- | | PRP | Personal pronoun | It, I, she, you | | DT / WDT | Determiner / wh-determiner | a/the versus What before flights; which relative modifier | | VB/VBP/VBZ/VBG | Base/non-3sg-present/3sg-present/gerund-participle | have after do; am/do; is/takes; sitting/putting | | NN/NNS/NNP | Common singular/plural/proper noun | game/newspapers/Milwaukee | | IN/RP | Preposition or subordinator / particle | in/over/from; on in the idiomatic putting on reading | | JJ/RB/POS | Adjective/adverb/possessive ending | nice/very/’s | Annotation ambiguities: “crap” is a noun adjunct under NN; an adjectival analysis can use JJ. “Fifty-second” may be NNP within an official street name instead of JJ; “gefilte” can be a noun adjunct in some taggers. Explain the chosen contextual reading rather than claiming all taggers must agree. **Source / convention note.** Context-sensitive tags and possessive tokenization are explicitly stated; reasonable corpus guidelines can differ on noun adjuncts. --- ## TEE · Q4a · Mention-pair coreference Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q4a Previous examination · complete worked answers **Question. Explain the mention-pair model with a detailed example.** [10 marks] ### Answer A mention is a span referring to an entity. A mention-pair model classifies a pair (earlier candidate antecedent, later mention) as coreferent or not. Pairwise decisions are subsequently linked into entity clusters; a positive score alone does not ensure globally consistent clusters. Detect mentions → Generate earlier candidate pairs → Extract pair features → Score same-entity relation → Select/link antecedents → Form entity clusters 1. Text: “Rita bought a car. She likes it.” Detect Rita, a car, She, it. 2. For She, compare (Rita,She) and (a car,She). Features include person/number compatibility, animate type, syntactic role, sentence distance and lexical agreement. 3. Rita–She has compatible singular human features; car–She fails the ordinary inanimate reading. Select Rita. 4. For it, compare earlier mentions: car–it is singular/inanimate and contextually fits likes. Cluster {Rita,She} and {a car,it}. 5. Train a classifier on labelled mention pairs; at prediction, choose a sufficiently supported antecedent or leave a new entity. Avoid reflexive/binding violations and document leakage during evaluation. Advantages: interpretable pair features and supervised learning. Limitations: quadratic candidate growth, error propagation from mention detection, non-transitive pair predictions, and weak long-range/world-knowledge reasoning. A greedy nearest-candidate choice is not the same as a learned mention-pair model. --- ## TEE · Q4b · Text classification pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q4b Previous examination · complete worked answers **Question. Explain the text-classification pipeline with a diagram.** [10 marks] ### Answer **Text classification** assigns a document to one or more predefined categories. Spam detection usually uses two classes; news classification may use several topics. In single-label multiclass classification, choose one of several categories. In multilabel classification, several labels may apply to the same document. Define labels → Prepare labelled examples → Train representation + model → Evaluate on unseen text → Predict new labels ### An exam-ready pipeline | Stage | What you do | Why it matters | | --- | --- | --- | | Task definition | Decide what counts as spam or which news topics exist. | Unclear labels create inconsistent training data. | | Data collection and annotation | Collect representative examples and assign correct labels. | The model learns from these examples. | | Split data | Separate training, validation and test data. | Test examples must not influence training choices. | | Preprocess | Tokenize and normalise where suitable; retain meaningful negation. | Reduce irrelevant variation without deleting signal. | | Represent | Use BoW, TF-IDF or learned embeddings. | Models need numerical features. | | Train | Fit Naive Bayes, logistic regression or another appropriate model. | Learn how features predict categories. | | Evaluate | Measure relevant precision, recall and F1 on held-out examples. | Accuracy alone may hide poor minority-class performance. | | Deploy and monitor | Apply the same preprocessing and monitor changing language. | Incoming data can differ from old examples. | Naive Bayes is especially useful for hand calculations. With a bag-of-words representation, its class score is the prior multiplied by the conditional probability of each token; repeated tokens repeat the factor. The “naive” assumption is conditional independence of features given the class, not that words are independent in real language. $$ s(c)=P(c)\prod_{w\in V}P(w\mid c)^{C(w,d)} $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ C(w,d) counts word w in document d. C(w,c) counts it in training documents belonging to class c. Nc is the total token count for class c. Add-one smoothing adds one to every vocabulary count and therefore adds |V| to the denominator. This is the same arithmetic taught for WSD, with document classes instead of senses. ### Worked classification · two-token message A spam classifier has vocabulary {win, prize, meeting}. Priors are P(spam)=P(ham)=0.5. Spam token counts are [3,2,0]; ham counts are [0,0,5]. With add-one smoothing, classify “win prize”. **Given** |V|=3; N(spam)=5; N(ham)=5; message has win once and prize once. **To find** Class scores and predicted label. ### Formula and why it applies $$ s(c)=P(c)P(\text{win}\mid c)P(\text{prize}\mid c) $$ $$ P(w\mid c)=\frac{C(w,c)+1}{N_c+|V|} $$ The model is multinomial Naive Bayes; multiply the two token likelihoods and the class prior. ### Solution #### Step 1. Compute the smoothed denominators $$ 5+3=8 $$ #### Step 2. Compute spam likelihoods $$ P(\text{win}\mid S)=\frac{3+1}{8}=\frac12 $$ $$ P(\text{prize}\mid S)=\frac{2+1}{8}=\frac38 $$ #### Step 3. Compute ham likelihoods $$ P(\text{win}\mid H)=P(\text{prize}\mid H)=\frac{0+1}{8}=\frac18 $$ #### Step 4. Multiply the spam score $$ s(S)=\frac12\times\frac12\times\frac38=\frac{3}{32}=0.09375 $$ #### Step 5. Multiply the ham score $$ s(H)=\frac12\times\frac18\times\frac18=\frac{1}{128}=0.0078125 $$ #### Step 6. Compare 0.09375 > 0.0078125. These are unnormalised scores; the label is chosen by the larger score. Predicted label: spam. **Yaad rakho:** Class prior × har word ki class-likelihood; sabse bada score wins. --- ## TEE · Q5a · Corpora and structure Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q5a Previous examination · complete worked answers **Question. Define a corpus. Explain corpus types, characteristics and structure.** [10 marks] ### Answer **A corpus is a collection of language data assembled for study or computational use.** Its plural is corpora. A corpus can contain books, news, reviews or conversations. It need not be labelled. **Annotation** adds linguistic information such as POS tags, entities, parse structures or sense labels. | Structure | Meaning | Use | | --- | --- | --- | | Plain | Collection without category labels | Word counts, language modelling | | Categorized | Documents grouped by genre/topic | Classification | | Overlapping categories | One document has several categories | Multi-topic news | | Temporal | Documents indexed by time | Language change | | Annotated | Linguistic labels attached | Tagging, parsing, WSD | | Parallel | Aligned texts in different languages | Translation | A **tagset** is the allowed label inventory. A **treebank** is a corpus annotated with syntax trees. **Metadata** records source, date, genre, language, annotation method and limitations. A common benchmark supports comparison under shared data and evaluation conditions. **Raw example:** “Riya works in Pune.” **POS annotation:** Riya/PROPN works/VERB in/ADP Pune/PROPN. **NER annotation:** Riya/PERSON, Pune/LOCATION. Each annotation answers a different task. A good corpus is relevant to its domain, sufficiently representative, consistently annotated and appropriately split. More data alone do not repair duplicated examples, incorrect labels or a mismatch between training and real use. Corpus=text collection. Annotation=text ke saath labels. Metadata=collection ki information. | Resource | Contents | Typical use | | --- | --- | --- | | Gutenberg selection | Literary/book texts | Long-form language and vocabulary | | Web text | Informal domain-specific text | Study usage beyond edited books | | Chat corpus | Conversational posts | Dialogue, informal forms | | Brown | English texts across genres | Genre comparison | | Reuters | News with topic labels | Text classification; overlapping topics | | Inaugural | Speeches indexed by time | Vocabulary change across years | | Wordlist | Words/names/stopwords | Lexical filtering | | Pronouncing dictionary | Words and phonetic forms | Speech, stress, rhyme | **Gutenberg versus web/chat:** book language does not represent every spelling or conversational pattern in messages. **Brown versus Reuters:** Brown emphasizes genres; Reuters news can have overlapping topic labels. **Inaugural addresses:** dates provide a time dimension. Corpus sizes depend on version, subset and tokenization. Explain their purpose and structure rather than assuming a historical count is a permanent property. Gutenberg=books; web/chat=informal; Brown=genres; Reuters=news topics; Inaugural=time. Corpus readers expose several views. **raw** returns a character string, **words** returns token occurrences, and **sents** returns a list of sentences where each sentence contains tokens. | Conceptual method | Returned view | Its length counts | | --- | --- | --- | | fileids() | Document IDs | Files | | raw(fileid) | Character string | Characters | | words(fileid) | Token list | Token occurrences | | sents(fileid) | Sentence lists | Sentences | | categories() | Genre/topic labels | Categories | | tagged\_words(fileid) | Word–tag pairs | Annotated tokens | **Question. For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types.** #### Given Punctuation is included in token count but excluded from word types. #### To find For sentence lists [the,cat,sleeps,.] and [the,dog,runs,.], count all tokens, sentences and lowercase alphabetic word types. #### Formula and why it applies Count every token occurrence, then distinct alphabetic forms. Different counting rules give different totals. #### Step 1. Token count 4+4=8. #### Step 2. Sentence count Two outer lists → 2. #### Step 3. Types {the,cat,sleeps,dog,runs} → 5. **Answer.** 8 tokens, 2 sentences, 5 word types. For “Cats run. Dogs sleep.”, a reader may return six tokens including two periods, but only four alphabetic word tokens. State the convention before calculating TTR or probabilities. raw=characters, words=tokens, sents=sentences. Teen alag counts. --- ## TEE · Q5b · Multinomial Naive Bayes Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q5b Previous examination · complete worked answers **Question. Explain Naive Bayes for text classification. Using the four documents below, classify “Chinese Chinese Chinese Tokyo Japan”: Chinese Beijing Chinese=yes; Chinese Chinese Shanghai=yes; Chinese Macao=yes; Tokyo Japan Chinese=no.** [10 marks] ### Answer Multinomial NB assumes token draws are conditionally independent given the class; it combines the class prior and class-specific word probabilities. Repeated Chinese contributes its likelihood three times. ### Given | Training text | Class | | --- | --- | | Chinese Beijing Chinese | yes | | Chinese Chinese Shanghai | yes | | Chinese Macao | yes | | Tokyo Japan Chinese | no | Test: Chinese Chinese Chinese Tokyo Japan ### To find Classify the test using multinomial Naive Bayes. ### Formula: which one and why Assume add-one smoothing unless the question supplies probabilities. Keep repeated words as repeated factors. Use training vocabulary only; ignore out-of-vocabulary words unless an unknown-token model is defined. $$ \mathrm{score}(c)=P(c)\prod_{w\in V}\left(\frac{c(w,c)+1}{N_c+|V|}\right)^{c(w,d)} $$ ### Step 1. Vocabulary and class priors V = {beijing, chinese, japan, macao, shanghai, tokyo}; |V|=6. | Class | Documents | Prior | | --- | --- | --- | | no | 1 | 1/4 | | yes | 3 | 3/4 | ### Step 2. Count every training word within each class | Class | Total tokens | beijing | chinese | japan | macao | shanghai | tokyo | | --- | --- | --- | --- | --- | --- | --- | --- | | no | 3 | 0 | 1 | 1 | 0 | 0 | 1 | | yes | 8 | 1 | 5 | 0 | 1 | 1 | 0 | ### Step 3. Count the test words | Word | Count | | --- | --- | | chinese | 3 | | tokyo | 1 | | japan | 1 | OOV words: none ### Step 4. Calculate likelihoods and multiply with the prior | Class | Prior | Individual factors | Exact score | Decimal score | | --- | --- | --- | --- | --- | | no | 1/4 | chinese: (1+1)/(3+1×6) = 2/9, used 3 time(s); tokyo: (1+1)/(3+1×6) = 2/9, used 1 time(s); japan: (1+1)/(3+1×6) = 2/9, used 1 time(s) | 8/59049 | 0.000135480702467 | | yes | 3/4 | chinese: (5+1)/(8+1×6) = 3/7, used 3 time(s); tokyo: (0+1)/(8+1×6) = 1/14, used 1 time(s); japan: (0+1)/(8+1×6) = 1/14, used 1 time(s) | 81/268912 | 0.000301213779973 | ### Step 5. Compare and optionally normalize Normalized scores: no = 0.310241388, yes = 0.689758612 Division by the same total does not change the winning class. ### Final answer **Predicted class: yes.** **Yaad rakho:** Count → prior → smoothed word chances → repeated factors → compare. --- ## TEE · Q6a · GloVe Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q6a Previous examination · complete worked answers **Question. Explain GloVe word embeddings.** [10 marks] ### Answer **GloVe (Global Vectors) learns word embeddings by fitting weighted relationships in a global word–context co-occurrence matrix.** Word2Vec uses local prediction examples; GloVe uses aggregate counts directly in its objective. $$ X_i=\sum_k X_{ik},\qquad P_{ik}=\frac{X_{ik}}{X_i}. $$ $$ X_{ik} $$ is how often context word $$ k $$ appears around target $$ i $$ . $$ X_i $$ is the total of row $$ i $$ . $$ P_{ik} $$ is the resulting context probability. The ratio $$ P(k\mid i)/P(k\mid j) $$ helps identify how a probe context differs between two words. For targets ice and steam, a probe such as solid should favour ice; gas should favour steam; a probe shared similarly by both gives a ratio near one. These are data tendencies, not universal numerical identities. **Question. For context solid, ice has count 12 out of row total 20; steam has count 2 out of row total 40. Compute P(solid|ice), P(solid|steam) and their ratio.** #### Given X(ice,solid)=12, X(ice)=20; X(steam,solid)=2, X(steam)=40. #### To find Two conditional probabilities and the ice/steam ratio. #### Formula and why it applies $$ P(k\mid i)=X_{ik}/X_i $$ , and $$ R=P(k\mid ice)/P(k\mid steam) $$ . #### Step 1. Ice probability $$ 12/20=0.6 $$ . #### Step 2. Steam probability $$ 2/40=0.05 $$ . #### Step 3. Ratio $$ 0.6/0.05=12 $$ . **Answer.** The probe solid is 12 times as probable in ice's context under these counts. **Question. For probe water, ice and steam both have row total 50. Counts are 10 and 10. Calculate the ratio.** #### Given Both probabilities use a denominator of 50. #### To find The probability ratio. #### Formula and why it applies Use row-normalized counts before taking the ratio. #### Step 1. Ice probability $$ 10/50=0.2 $$ . #### Step 2. Steam probability $$ 10/50=0.2 $$ . #### Step 3. Divide $$ 0.2/0.2=1 $$ . **Answer.** Ratio 1: this probe does not distinguish the targets in this toy data. **Question. For probe gas, counts are 1/100 around ice and 20/100 around steam. Compute the ice-to-steam ratio.** #### Given Conditional counts have the same row denominator. #### To find The ratio and interpretation. #### Formula and why it applies $$ R=P(gas\mid ice)/P(gas\mid steam) $$ . #### Step 1. Ice probability $$ 1/100=0.01 $$ . #### Step 2. Steam probability $$ 20/100=0.2 $$ . #### Step 3. Ratio $$ 0.01/0.2=0.05 $$ . **Answer.** Ratio 0.05, so gas is much more strongly associated with steam. Ratio >1: pehla word favoured. <1: doosra. ≈1: probe dono mein similar. --- ## TEE · Q6b · BoW and TF–IDF movie corpus Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q6b Previous examination · complete worked answers **Question. Find Bag of Words and TF–IDF for: “This movie is very scary and long”; “This movie is not scary and is slow”; “This movie is spooky and good”.** [10 marks] ### Answer ### Given | Document | Original text | | --- | --- | | 1 | This movie is very scary and long | | 2 | This movie is not scary and is slow | | 3 | This movie is spooky and good | Stopwords: none supplied ### To find BoW, IDF and TF–IDF ### Formula: which one and why Use lowercase alphabetic tokens and exactly the stated stoplist. If no stoplist is supplied, retain function words. Adopt raw-count TF and natural-log IDF; the paper does not prescribe another TF/log convention. $$ \mathrm{TFIDF}(w,d)=c(w,d)\ln\frac{N}{\mathrm{df}(w)} $$ A term shared by all N documents has IDF ln(1)=0. Counts and TF–IDF weights are different representations. ### Step 1. Tokenize each document | Document | Tokens | | --- | --- | | 1 | this, movie, is, very, scary, and, long | | 2 | this, movie, is, not, scary, and, is, slow | | 3 | this, movie, is, spooky, and, good | ### Step 2. Fix the vocabulary order and, good, is, long, movie, not, scary, slow, spooky, this, very ### Step 3. Count occurrences (Bag of Words) | Document | and | good | is | long | movie | not | scary | slow | spooky | this | very | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | | 2 | 1 | 0 | 2 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | | 3 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | ### Step 4. Calculate document frequency and IDF | Term | df | N / df | ln(N/df) | | --- | --- | --- | --- | | and | 3 | 1 | 0.000000000 | | good | 1 | 3 | 1.098612289 | | is | 3 | 1 | 0.000000000 | | long | 1 | 3 | 1.098612289 | | movie | 3 | 1 | 0.000000000 | | not | 1 | 3 | 1.098612289 | | scary | 2 | 3/2 | 0.405465108 | | slow | 1 | 3 | 1.098612289 | | spooky | 1 | 3 | 1.098612289 | | this | 3 | 1 | 0.000000000 | | very | 1 | 3 | 1.098612289 | ### Step 5. Multiply each TF by its IDF | Document | and | good | is | long | movie | not | scary | slow | spooky | this | very | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | 1 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | 0.405465 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | | 2 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 0.405465 | 1.098612 | 0.000000 | 0.000000 | 0.000000 | | 3 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 1.098612 | 0.000000 | 0.000000 | ### Final answer The final table is the TF–IDF matrix in the declared vocabulary order. A different explicitly stated log base rescales all IDFs; length-normalized TF changes each document scale. --- ## TEE · Q7a · Information extraction, chunking and NER Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q7a Previous examination · complete worked answers **Question. Explain information-extraction architecture with a diagram. Explain chunking and named-entity recognition with examples.** [10 marks] ### Answer **Information extraction (IE) converts unstructured language into structured information such as named entities, relations and events.** A structured output has explicit fields or records that can be stored, searched or analysed. IE is broader than NER: finding two names does not yet determine how those entities are related. ### The three levels of extracted information | Level | What it identifies | Example | | --- | --- | --- | | Entity extraction / NER | A span and its entity type | Asha → person; Orbit Labs → organisation | | Relation extraction | A labelled connection between entities | works-for(Asha, Orbit Labs) | | Event extraction | An occurrence with roles and attributes | Joining: employee=Asha, employer=Orbit Labs, date=2025 | An **entity** is a participant such as a person or organisation. A **relation** links participants. An **event** describes something that happened, may happen or did not happen. An **event role** names a participant's function: buyer, seller, employee, employer, location or time. These labels must follow the chosen extraction schema. Raw text → Sentence/token processing → Entity spans + types → Reference resolution → Relations and event roles → Normalize and store records ### An extraction pipeline explained | Step | Operation | Reason | | --- | --- | --- | | Specify schema | Choose fields and allowed relations/events. | A system needs to know what records it is building. | | Preprocess | Segment sentences and tokenize without losing useful structure. | Locate expressions and boundaries. | | Recognise entities | Find spans and assign types. | Names can contain several words. | | Resolve references | Link pronouns/descriptions to the right entities. | Two mentions may denote one participant. | | Extract links/events | Use patterns, syntax or learned models. | Co-occurrence alone does not identify a relationship. | | Handle qualifiers | Preserve negation, time and uncertainty. | Not hired differs from hired. | | Normalize | Standardize dates, names or quantities where justified. | Equivalent forms should be comparable. | | Store/evaluate | Save records and compare them with annotated answers. | Wrong links matter even if entity names are correct. | ### Worked application 1 · turn a sentence into a record **Question. Extract entities and a joining event from “Asha joined Orbit Labs in Mumbai on 3 May 2025.” Use fields person, organisation, location, event and date.** #### Given One affirmative sentence; Orbit Labs is an organisation and Mumbai is a location. #### To find Entity spans/types and one structured joining record. #### Method and why it applies Find the complete entity spans, then assign their roles using the verb joined and its grammatical context. ### Solution #### Step 1. Mark entity spans | Span | Type | | --- | --- | | Asha | Person | | Orbit Labs | Organisation | | Mumbai | Location | | 3 May 2025 | Date | #### Step 2. Identify the event trigger Joined signals a joining event. The subject Asha is the joining person; Orbit Labs is the joined organisation. #### Step 3. Attach contextual fields In Mumbai supplies a location associated with the stated joining event. On 3 May 2025 supplies its date. Do not infer the company's permanent headquarters from this sentence alone. #### Step 4. Normalize the date The unambiguous date can be stored as 2025-05-03 while preserving its original text. #### Step 5. Build the record | Field | Value | | --- | --- | | event | joining | | person | Asha | | organisation | Orbit Labs | | location | Mumbai | | date | 2025-05-03 | #### Step 6. State the information limit The sentence supports a joining event on that date. By itself it does not prove that Asha still works there indefinitely. **Answer.** A joining record for Asha and Orbit Labs at the stated location/date, with complete entity spans preserved. ### Worked application 2 · direction of a relation matters **Question. From “Nova acquired Luma for ₹50 crore in 2025”, extract an acquisition event with buyer, acquired organisation, amount and year.** #### Given Nova and Luma are organisation names. The active verb is acquired; no seller is named. #### To find A correctly directed acquisition record. #### Method and why it applies Use grammatical roles to identify buyer and acquired organisation. Merely returning the unordered pair {Nova,Luma} loses the event direction. ### Solution #### Step 1. Recognise participant entities Nova is the subject organisation; Luma is the object organisation. #### Step 2. Locate the trigger Acquired identifies an acquisition event. #### Step 3. Assign roles | Role | Entity | | --- | --- | | Buyer/acquirer | Nova | | Acquired organisation | Luma | #### Step 4. Extract amount and time Amount text=₹50 crore; year=2025. Currency and magnitude belong together; do not drop crore or replace it by lakh. #### Step 5. Create the record | Field | Value | | --- | --- | | event | acquisition | | buyer | Nova | | acquired\_organisation | Luma | | amount | INR 50 crore | | year | 2025 | | seller | Not stated | #### Step 6. Reject an unsupported inference The sentence does not name the former owner/seller. Luma is the acquired organisation, not automatically the seller. **Answer.** Acquisition(buyer=Nova, acquired organisation=Luma, amount=INR 50 crore, year=2025). Seller remains unspecified. ### Worked application 3 · reference, negation and time **Question. Extract employment-related events from “Ravi joined Vega in 2021. He left the company in 2024. He did not join Orion.” Keep event type, person, organisation, date and polarity.** #### Given He refers to Ravi; the company refers to Vega. The last sentence explicitly negates joining Orion. #### To find A structured event list without claiming an unsupported current employer. #### Method and why it applies Resolve references, identify each event separately and preserve whether it is affirmed or negated. ### Solution #### Step 1. Resolve the repeated mentions | Mention | Referent | | --- | --- | | He in sentence 2 | Ravi | | the company | Vega | | He in sentence 3 | Ravi | #### Step 2. Extract the first event Joining; person=Ravi; organisation=Vega; date=2021; polarity=affirmed. #### Step 3. Extract the second event Leaving; person=Ravi; organisation=Vega; date=2024; polarity=affirmed. #### Step 4. Extract the third statement Joining; person=Ravi; organisation=Orion; date=not stated; polarity=negated. This must not create an affirmative employment relation with Orion. #### Step 5. Write the event table | Event | Person | Organisation | Date | Polarity | | --- | --- | --- | --- | --- | | joining | Ravi | Vega | 2021 | Affirmed | | leaving | Ravi | Vega | 2024 | Affirmed | | joining | Ravi | Orion | Not stated | Negated | #### Step 6. Interpret current status cautiously The text says that Ravi left Vega and did not join Orion. It supplies no other current employer. Do not fill that field by guessing. **Answer.** Three qualified event statements, including a negated Orion joining. No current employer can be established from this text. ### Approaches and their trade-offs | Approach | Mechanism | Advantage | Limitation | | --- | --- | --- | --- | | Dictionary/rule based | Match known names, trigger words and patterns such as person joined organisation. | Transparent and useful in a narrow domain. | New names, paraphrases and complex syntax can escape patterns. | | Supervised statistical | Learn entity/relation/event labels from annotated examples and features. | Combines several cues consistently. | Requires representative labels and careful feature design. | | Neural/contextual | Use contextual representations to classify spans and links or generate a structured record. | Can capture paraphrases and broader context. | May still invent or misassign fields; schema and factual checks remain necessary. | Applications include extracting company events from news, conditions/treatments from clinical text, skills from resumes and attributes from product descriptions. Extraction quality is measured against annotated records: an entity match can require the correct span and type; a relation match additionally requires the correct participants and direction. Precision measures how many extracted records are correct; recall measures how many gold records were found. For a 10-mark answer, begin with the definition and structured output, distinguish NER/relation/event extraction, draw and explain the pipeline, work through a record with roles and qualifiers, compare approaches, then state evaluation and limitations. “Extract keywords” alone is too narrow: keywords do not necessarily represent a factual relation. **Yaad rakho:** NER naam pakadta hai; relation kaun-kisse jodta hai; event kya hua batata hai. Negation aur date mat girao. **Chunking**, also called shallow parsing, groups adjacent tagged words into meaningful non-overlapping units, commonly noun phrases. It usually identifies local phrase boundaries without producing the complete hierarchical structure of a sentence. **Chinking** removes specified tokens from an existing candidate chunk, splitting or trimming it. The/DT small/JJ cat/NN sat/VBD on/IN the/DT mat/NN → NP: [The small cat] → Outside: sat on → NP: [the mat] A POS tagger labels individual words. A chunker groups words into phrases. A full parser shows how phrases relate recursively to form the entire sentence. These are related tasks with different outputs. | Operation | Input | Output | | --- | --- | --- | | POS tagging | The small cat | The/DT small/JJ cat/NN | | NP chunking | Tagged tokens | [NP The small cat] | | Chinking | A broad candidate chunk | Remove verbs/prepositions, keep noun-phrase segments | | Full parsing | Sentence + grammar | A nested tree with sentence, noun phrase and verb phrase nodes | **Rule-based chunking** uses tag patterns. Statistical or learned chunkers infer boundaries from labeled examples. A useful NP rule is an optional determiner, followed by zero or more adjectives, followed by a noun. In NLTK-style notation: `NP: {
?*}`. Here ? means zero or one occurrence; \* means zero or more; + means one or more when used. The braces mean “form a chunk around the matched sequence”. **Question. Apply the NP rule optional DT, zero or more JJ, then one NN to “The/DT quick/JJ brown/JJ fox/NN jumps/VBZ over/IN the/DT lazy/JJ dog/NN”.** #### Given Tagged input is supplied. Rule: DT? JJ\* NN. #### To find All noun-phrase chunks. #### Formula and why it applies `NP → DT? JJ* NN` The rule groups a noun with its preceding determiner and adjectives; a verb or preposition does not match this NP pattern. #### Solution ##### Step 1. Start at The The matches optional DT; quick and brown both match JJ\*; fox matches NN. ##### Step 2. Form the first NP **[NP The quick brown fox]** ##### Step 3. Skip nonmatching tokens jumps/VBZ and over/IN do not satisfy the NP pattern. ##### Step 4. Match the second NP the is DT, lazy is JJ, dog is NN: **[NP the lazy dog]**. **Final answer.** [NP The quick brown fox] jumps over [NP the lazy dog]. **Question. Using NP → DT? JJ\* NN, identify the chunks in “bright/JJ stars/NN shine/VBP and/CC a/DT moon/NN glows/VBZ”. Use the supplied tags exactly.** #### Given Input tags label stars as NN for this exercise. Rule requires optional DT, zero or more JJ, then NN. #### To find The matched NP spans. #### Formula and why it applies `NP → DT? JJ* NN` The optional determiner may be absent and the adjective repetition may be empty. #### Solution ##### Step 1. Match bright stars No determiner is present, which DT? permits. bright matches JJ; stars matches NN. ##### Step 2. Inspect shine and and VBP and CC fall outside the NP rule. ##### Step 3. Match a moon a matches DT; no adjective is present; moon matches NN. **Final answer.** Chunks are [NP bright stars] and [NP a moon]. With ordinary Penn tagging, stars is usually NNS; broaden the noun pattern to include NNS when that tag is used. **Question. Begin with one candidate chunk containing “The/DT cat/NN sat/VBD on/IN the/DT mat/NN”. Chink out verbs and prepositions. Show the remaining chunks.** #### Given Initial candidate: [The cat sat on the mat]. Remove tags VBD and IN. #### To find The chunk segments after exclusion. #### Formula and why it applies Chinking removes matched tokens from the existing chunk. Removing the middle verb/preposition region divides the original candidate into two remaining contiguous spans. #### Solution ##### Step 1. Locate the tokens to remove sat is VBD and on is IN. ##### Step 2. Remove those tokens from chunk membership The remaining token groups are “The cat” and “the mat”. sat and on still exist in the sentence; they are simply outside these chunks. ##### Step 3. Write the final chunked sentence **[The cat]** sat on **[the mat]**. **Final answer.** Two chunks remain: The cat; the mat. **Yaad rakho:** Chunking = andar group karo; chinking = unwanted tokens ko group se bahar karo. **Named Entity Recognition (NER)** identifies text spans referring to entities and assigns their types, such as PERSON, ORGANIZATION and LOCATION. Many practical schemas also include DATE, TIME, MONEY or domain-specific types. The exact allowed labels depend on the application's annotation scheme. NER has two parts: **boundary detection** finds where the entity starts and ends; **classification** assigns the appropriate type. A multiword name should normally be one entity span: “New York” is a location span, not two unrelated location decisions. Text: Mary joined Google in New York on Monday. → Mary → PERSON → Google → ORGANIZATION → New York → LOCATION → Monday → DATE Not every noun is a named entity. In “a developer works at Google”, developer is an ordinary job description while Google is a named organization. A specialized HR system may define a JOB\_TITLE category, but it must state that schema explicitly. NER identifies mentions; entity linking is the separate task of connecting a mention to a unique database identity. | Task | Question it answers | Example | | --- | --- | --- | | POS | What grammatical role does this token have? | Google/NNP | | Chunking | Which adjacent words form a phrase? | [NP a skilled developer] | | NER | Which span names an entity, and of what type? | [Google] ORGANIZATION | | Entity linking | Which real entity does this mention denote? | Connect Paris to the intended city or person record | **Applications.** NER structures search results and documents, extracts organizations/people from news, routes support tickets by product or organization, supports recommendation, and extracts specialized entities from healthcare or legal text. Context is essential: “Apple released a laptop” names an organization, while “she ate an apple” describes fruit. **Yaad rakho:** NER mein do kaam: naam kahan se kahan tak, aur kis category ka. --- ## TEE · Q7b · Distributional hypothesis and cosine Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/papers/legacy-f22/q7b Previous examination · complete worked answers **Question. Justify “You shall know a word by the company it keeps.” Calculate cosine similarity of w₁=[0.2,0.1,0.3,0.4] and w₂=[0.3,0,0.2,0.5].** [10 marks] ### Answer Words occurring in similar contexts often have related meanings; vectors encode their context distributions. This is a statistical tendency, not proof that every similar-context pair is synonymous—antonyms can share contexts too. ### Given u=[0.2, 0.1, 0.3, 0.4]; v=[0.3, 0, 0.2, 0.5] ### To find Cosine similarity in the given coordinate order. ### Formula: which one and why $$ \cos(u,v)=\frac{u\cdot v}{\|u\|\|v\|} $$ ### Step 1. Multiply matching coordinates 0.2×0.3 + 0.1×0 + 0.3×0.2 + 0.4×0.5 = 0.32 ### Step 2. Calculate both squared lengths u: 0.2² + 0.1² + 0.3² + 0.4² = 0.30000000000000004; v: 0.3² + 0² + 0.2² + 0.5² = 0.38 ### Step 3. Take square roots and divide 0.32 / (√0.30000000000000004 × √0.38) = 0.947758204 ### Final answer **Cosine=0.947758204.** This measures direction, not a probability. --- ## TEE · Write the whole paper under exam conditions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--instructions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/instructions TEE · actual previous paper · 100 marks · 180 minutes Use this actual previous paper as a three-hour, 100-mark practice paper. Q1 is compulsory (20 marks); choose any four full questions from Q2–Q7 (20 marks each). Each original question and its complete answer follow. Read the question, write your answer in your notebook, then compare the working below it. Budget roughly 36 minutes per attempted 20-mark question, including its subparts. Start each full question on a fresh page, as the supplied papers instruct. This is an old paper used for timed practice, not a forecast of the next TEE. --- ## TEE · Q1A · NLU and NLG Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q1a TEE · actual previous paper · 100 marks · 180 minutes Question. Differentiate between Natural Language Understanding (NLU) and Natural Language Generation (NLG) in NLP with suitable examples. [5 marks] ### Answer **Natural Language Understanding (NLU)** interprets language to identify its meaning, intent and relevant information. **Natural Language Generation (NLG)** produces language that expresses information in a useful form. | Basis | NLU | NLG | | --- | --- | --- | | Direction | Language → meaning or structured information | Meaning or structured information → language | | Typical input | A user utterance, document or sentence | Facts, a database record, a prompt or a representation | | Typical output | Intent, entities, sentiment or a semantic representation | A sentence, explanation, summary or dialogue response | | Main difficulty | Resolving ambiguity and context | Choosing accurate content and producing coherent, fluent wording | ### Example: a weather assistant User: “Will it rain in Mumbai tomorrow?” NLU recognises a weather-query intent, the location Mumbai and the date tomorrow. After a weather service supplies the forecast, NLG produces “Rain is likely in Mumbai tomorrow; carry an umbrella.” User language → NLU: identify request → Retrieve forecast → NLG: express answer The two functions often work together in a chatbot. Understanding the request and wording the reply are separate tasks; fluent generation alone does not prove correct understanding. **Yaad rakho:** NLU mein meaning nikalo; NLG mein meaning ko sentence banao. --- ## TEE · Q1B · Chunking and chinking Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q1b TEE · actual previous paper · 100 marks · 180 minutes Question. Explain the concepts of chunking and chinking in Natural Language Processing (NLP). Discuss how Part-of-Speech (POS) tagging supports these processes, and illustrate your explanation with a suitable example. [5 marks] ### Answer **Chunking** groups adjacent tagged words into shallow phrases, such as noun phrases, without building a complete sentence parse. **Chinking** removes a specified sequence of tags or words from a chunk, splitting a larger candidate chunk when required. ### POS tags provide the pattern A POS tag describes a grammatical category: DT = determiner, JJ = adjective, NN = singular noun, VBD = past-tense verb and IN = preposition. A noun-phrase rule can be written as `NP: {
?*+}`: an optional determiner, zero or more adjectives, followed by one or more nouns. ### Worked example | Word | The | small | cat | sat | on | the | mat | | --- | --- | --- | --- | --- | --- | --- | --- | | Tag | DT | JJ | NN | VBD | IN | DT | NN | Applying the NP rule gives **[The small cat] sat on [the mat]**. The first chunk contains DT–JJ–NN; the second contains DT–NN. “sat on” is not included because it does not match the noun-phrase rule. For chinking, begin with one broad chunk: **[The small cat sat on the mat]**. Remove the VBD and IN sequence “sat on”, for example with `}+{`. The result is **[The small cat] sat on [the mat]**. POS tagging therefore supplies grammatical labels; chunking uses labels to select phrases, while chinking uses labels to exclude unwanted material. Incorrect tags can lead to incorrect chunks. **Yaad rakho:** Chunking = phrase ko jodo; chinking = beech ka unwanted part nikalo. --- ## TEE · Q1C · Hyponymy, hypernymy and WSD Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q1c TEE · actual previous paper · 100 marks · 180 minutes Question. Define hyponymy and hypernymy with suitable linguistic examples. How can hypernymy be applied in Word Sense Disambiguation (WSD) to resolve the correct meaning of polysemous words? Provide an example? [5 marks] ### Answer **Hyponymy** is the “is-a-kind-of” relation from a specific concept to a broader concept. **Hypernymy** is the inverse relation from the broader concept to the specific one. “Rose” is a hyponym of “flower”; “flower” is a hypernym of “rose”. rose / tulip → flower → plant The arrow points towards a broader category. Hyponymy is not a part–whole relation: a wheel is part of a car, but it is not a kind of car. ### Using the hierarchy for WSD WSD selects the intended sense of a word from its context. Obtain candidate senses from a lexical resource such as WordNet, inspect their hypernym chains, and favour the chain most compatible with nearby words. 1. Sentence: “The mouse clicked the icon.” 2. Candidate 1: mouse → rodent → mammal → animal. Candidate 2: mouse → pointing device → computer peripheral. 3. The context words “clicked” and “icon” belong to the computer-use domain. They support the device hierarchy, rather than the animal hierarchy. 4. Select the computer-pointing-device sense. In “The mouse ate cheese,” the animal hierarchy is more compatible. Hypernym evidence supplies semantic generalisation; it is a useful signal rather than a guarantee, because short or unusual contexts may still be ambiguous. **Yaad rakho:** Hyponym specific hai; hypernym uska bada category hai. --- ## TEE · Q1D · Pragmatics and discourse Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q1d TEE · actual previous paper · 100 marks · 180 minutes Question. Compare pragmatic and discourse analysis with the help of appropriate examples. [5 marks] ### Answer **Pragmatic analysis** interprets an utterance using the situation, speaker intention, shared knowledge and conversational expectations. **Discourse analysis** studies how multiple utterances form a connected text or conversation. | Basis | Pragmatic analysis | Discourse analysis | | --- | --- | --- | | Focus | Intended meaning in context | Relationships and coherence across utterances | | Useful information | Speaker, listener, place, time and common knowledge | Previous sentences, entities, topics and rhetorical relations | | Typical tasks | Interpret requests, implicatures and deixis | Resolve references and analyse cohesion, coherence and discourse structure | | Example | “It is cold here” may be a request to close a window | “Riya bought a book. She read it.” links She to Riya and it to book | In “Can you pass the salt?”, literal syntax asks about ability; pragmatic interpretation usually identifies a polite request. In “The road flooded. Therefore, the school closed,” discourse analysis identifies a cause–result link between the sentences. The levels interact: resolving “she” across a conversation may require both earlier text and real-world context. They are complementary rather than mutually exclusive. **Yaad rakho:** Pragmatics: bolne ka intention. Discourse: sentences ka connection. --- ## TEE · Q2A · Morpheme, lexeme and NER Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q2a TEE · actual previous paper · 100 marks · 180 minutes Question. i. Differentiate between Morpheme and Lexeme with examples. ii. Discuss the significance of NER in three different NLP applications. [10 marks] ### Answer ### i. Morpheme and lexeme A **morpheme** is the smallest linguistic unit that contributes meaning or grammatical function. A **lexeme** is an abstract vocabulary item representing a family of inflected word forms. | Basis | Morpheme | Lexeme | | --- | --- | --- | | Unit | A meaningful component of a word | An abstract dictionary vocabulary item | | Examples | un-, happy, -ness; cat and plural -s | RUN covers run, runs, ran and running | | Independence | Free morphemes can stand alone; bound morphemes cannot | A lexeme is realised through one or more word forms | | NLP relevance | Morphological segmentation, stemming and analysis | Lemmatization and lexical meaning | “Cats” contains two morphemes: cat + plural -s. “cat” and “cats” are forms of the lexeme CAT. “Unhappiness” can be segmented into un + happy + ness, although its spelling changes at the boundary. A morpheme and a syllable are different: syllables concern pronunciation, while morphemes concern meaning or grammar. ### ii. Named Entity Recognition in applications **NER** locates spans that name entities and assigns types such as PERSON, ORGANIZATION, LOCATION, DATE or domain-specific categories. | Application | Entities and example | Why NER matters | | --- | --- | --- | | News information extraction | “Asha joined Infosys in Pune.” → Asha/PERSON, Infosys/ORG, Pune/LOCATION | Creates searchable people–organisation–location records and supports later relation extraction | | Healthcare document processing | Find disease, drug and dosage spans in a clinical note | Supports record indexing and medication extraction; specialist labels and validation are required | | Customer support and commerce | Extract product names, dates and order identifiers from “My Pixel arrived damaged on Monday.” | Routes complaints, links the correct product and fills structured support fields | NER answers “Which entity spans occur here, and what type are they?” It does not by itself establish every relation between those entities or resolve all names to unique real-world identities. **Yaad rakho:** Morpheme = word ka meaningful piece; lexeme = word family. NER = naam dhoondho aur type do. --- ## TEE · Q2B · Laplace-smoothed bigram probability Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q2b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the following training corpus: 1. artists paint murals in city 2. musicians play songs in city 3. painters paint murals on walls 4. signs are near city Using a bigram language model with Laplace smoothing, evaluate the probability of the test sentence: “artists paint murals near city”. Assume the and for each sentence. [10 marks] ### Answer ### Given - artists paint murals in city - musicians play songs in city - painters paint murals on walls - signs are near city Test: artists paint murals near city. Add one start token and one end token to every training and test sentence. ### To find The Laplace-smoothed bigram probability of the complete test sentence, including its end token. ### Formula: which one and why $$ P_{+1}(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)+1}{C(w_{i-1})+V} $$ C(previous,current) counts that adjacent pair in the training corpus. C(previous) counts its occurrences as a history. V is the vocabulary size. Add-one smoothing is necessary because an unseen pair must not force the whole sentence probability to zero. $$ P(w_1,\ldots,w_n,\langle/s\rangle\mid\langle s\rangle)=\prod_{i=1}^{n+1}P(w_i\mid w_{i-1}) $$ ### Solution ### Step 1. Fix the token and vocabulary convention Lowercase words and remove the final punctuation. The supplied course key counts both boundary tokens in V. Following that convention: are, artists, city, in, murals, musicians, near, on, paint, painters, play, signs, songs, walls, , gives **V = 16**. The start marker supplies history; it is not an extra probability factor before the first word. ### Step 2. List every required transition The complete test chain is → artists → paint → murals → near → city → . ### Step 3. Count first, then add one | Required bigram | History count | Pair count | Smoothed probability | | --- | --- | --- | --- | | → artists | 4 | 1 | (1+1)/(4+16) = 1/10 | | artists → paint | 1 | 1 | (1+1)/(1+16) = 2/17 | | paint → murals | 2 | 2 | (2+1)/(2+16) = 1/6 | | murals → near | 2 | 0 | (0+1)/(2+16) = 1/18 | | near → city | 1 | 1 | (1+1)/(1+16) = 2/17 | | city → | 3 | 3 | (3+1)/(3+16) = 4/19 | Count only adjacent words, never words that merely appear in the same sentence. The unseen pairs have count zero, so their numerator becomes one. ### Step 4. Multiply all six probabilities $$ P=\frac{2}{20}\times\frac{2}{17}\times\frac{3}{18}\times\frac{1}{18}\times\frac{2}{17}\times\frac{4}{19} $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. → artists | $$ 1\times\frac{1}{10}=\frac{1}{10} $$ | | 2. artists → paint | $$ \frac{1}{10}\times\frac{2}{17}=\frac{1}{85} $$ | | 3. paint → murals | $$ \frac{1}{85}\times\frac{1}{6}=\frac{1}{510} $$ | | 4. murals → near | $$ \frac{1}{510}\times\frac{1}{18}=\frac{1}{9180} $$ | | 5. near → city | $$ \frac{1}{9180}\times\frac{2}{17}=\frac{1}{78030} $$ | | 6. city → | $$ \frac{1}{78030}\times\frac{4}{19}=\frac{2}{741285} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{2}{741285}\approx 2.69801763\times10^{-6} $$ ### Step 5. Interpret the result This is a joint sentence probability, so a small value is normal. Do not add transition probabilities; each word must occur in sequence, so the chain multiplies them. ### Final answer **P ≈ 2.69801763 × 10-6** with the stated course vocabulary convention. **Yaad rakho:** Pair count mein 1 jodo, previous-word count mein V; phir saare fractions multiply karo. **Convention note:** Some implementations exclude from the predicted vocabulary. Their V is one smaller and their numerical probability differs. In an exam, state V explicitly and follow the convention specified in the question or class solution. --- ## TEE · Q3A · Ambiguity and bottom-up parsing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q3a TEE · actual previous paper · 100 marks · 180 minutes Question. i. Explain the various types of ambiguities encountered in Natural Language Processing (NLP). Illustrate each type with suitable examples. ii. Compare top-down and bottom-up parsing and apply bottom-up syntax parsing approach to the sentence “A cat chased the mouse” and draw a parse tree using the grammar rules: S → NP VP; NP → Pronoun | Proper-Noun | Det Nominal; Nominal → Noun | Nominal Noun | Nominal PP; VP → Verb | Verb NP | VP PP; PP → Prep NP. Lexicon: Det → {A, the}; Noun → {cat, mouse}; Verb → {chased}. [10 marks] ### Answer ### i. Types of ambiguity Ambiguity exists when the same language expression permits more than one interpretation. The correct interpretation usually requires grammatical, semantic or contextual evidence. | Type | Meaning | Example and readings | | --- | --- | --- | | Lexical | One word has multiple meanings | “She visited the bank”: financial institution or riverbank | | Syntactic | The sentence permits different structural attachments | “I saw the man with a telescope”: I used the telescope, or the man had it | | Semantic / scope | A structured sentence permits different interpretations | “Every student read a book”: one shared book, or possibly a different book per student | | Referential | A referring expression has more than one candidate referent | “Asha told Riya that she won”: she could refer to Asha or Riya | | Pragmatic | Intention depends on the situation | “Can you open the door?” can ask about ability or request an action | ### ii. Parsing approaches | Aspect | Top-down | Bottom-up | | --- | --- | --- | | Start | The start symbol S | The input words | | Operation | Expand a nonterminal using grammar productions | Combine words and constituents using productions in reverse | | Success | Leaves match the complete input | All input is reduced to S | | Possible difficulty | Naive recursive descent can loop on left recursion or need backtracking | A naive shift-reduce parser may face shift/reduce conflicts | ### Given The five tokens are A / cat / chased / the / mouse. The supplied grammar permits determiner + nominal noun phrases and a verb + noun phrase verb phrase. ### To find A bottom-up sequence of reductions and a parse tree whose root is S and whose leaves preserve the input order. ### Rule: which one and why A production A → B C permits B C to be reduced to A. Bottom-up parsing begins with the words; lexical rules assign their categories, and phrase rules combine those categories. ### Step 1. Assign lexical categories A → Det; cat → Noun; chased → Verb; the → Det; mouse → Noun. ### Step 2. Build the first noun phrase cat is Noun. Reduce Noun to Nominal using Nominal → Noun. Then reduce Det Nominal, representing “A cat”, to NP. ### Step 3. Build the object noun phrase mouse is Noun. Reduce it to Nominal. Reduce “the” + “mouse”, now Det Nominal, to NP. ### Step 4. Build the verb phrase “chased” is Verb and “the mouse” is NP. Reduce Verb NP to VP. ### Step 5. Build the sentence The remaining constituents are NP (“A cat”) and VP (“chased the mouse”). Apply S → NP VP in reverse to obtain S. Every input token is used exactly once. SNPVPDetNominalVerbNPNounDetNominalNounAcatchasedthemouse Bottom-up parse tree for A cat chased the mouse. **Final answer:** [S [NP [Det A] [Nominal [Noun cat]]] [VP [Verb chased] [NP [Det the] [Nominal [Noun mouse]]]]]. **Yaad rakho:** Bottom-up: words se phrases, phrases se S. Top-down: S se words. --- ## TEE · Q3B · Co-occurrence matrix and cosine Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q3b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the given paragraph: The teacher wrote on the board. The students listened to the teacher. The board was full of notes. The student asked a question. The teacher answered from the book. Target words: teacher, student, board, book. Context words: wrote, listened, notes, question. i. Construct a co-occurrence matrix for the target and context words. Take the window size as one whole sentence. ii. Evaluate the similarity between the vectors of the target words “teacher-student” and “teacher-board” using the cosine similarity. iii. Plot the target word vectors by considering only two dimensions: “wrote” and “notes”. [10 marks] ### Answer ### Given Five sentences, four target rows and four context columns. Treat students and student as the same target lemma, and count each target–context pair once when both occur in a sentence. ### To find The 4 × 4 sentence-window co-occurrence matrix, two cosine similarities and the requested 2-D plot. ### Formula: which one and why $$ M_{ij}=\sum_{s=1}^{5}\mathbf{1}[\text{target}_i\text{ and context}_j\text{ occur in sentence }s] $$ The indicator is 1 when both words appear in that sentence and 0 otherwise. Each matrix row is the target word’s vector: its list of four context counts. $$ \cos(\mathbf a,\mathbf b)=\frac{\mathbf a\cdot\mathbf b}{\|\mathbf a\|\|\mathbf b\|} $$ The dot product multiplies corresponding coordinates and adds them. The length is the square root of the sum of squared coordinates. Cosine compares directions while allowing different vector lengths. ### Solution ### Step 1. Record the contribution of each sentence | Sentence | Targets present | Relevant context | Counts added | | --- | --- | --- | --- | | 1 | teacher, board | wrote | teacher–wrote +1; board–wrote +1 | | 2 | student, teacher | listened | student–listened +1; teacher–listened +1 | | 3 | board | notes | board–notes +1 | | 4 | student | question | student–question +1 | | 5 | teacher, book | none | No requested context count | ### Step 2. Construct every requested column | Target | wrote | listened | notes | question | | --- | --- | --- | --- | --- | | teacher | 1 | 1 | 0 | 0 | | student | 0 | 1 | 0 | 1 | | board | 1 | 0 | 1 | 0 | | book | 0 | 0 | 0 | 0 | ### Step 3. Calculate teacher–student $$ \mathbf t\cdot\mathbf s=(1)(0)+(1)(1)+(0)(0)+(0)(1)=1 $$ $$ \|\mathbf t\|=\sqrt{1^2+1^2+0^2+0^2}=\sqrt2 $$ $$ \|\mathbf s\|=\sqrt{0^2+1^2+0^2+1^2}=\sqrt2 $$ $$ \cos(\mathbf t,\mathbf s)=\frac1{\sqrt2\sqrt2}=\frac12=0.5 $$ ### Step 4. Calculate teacher–board $$ \mathbf t\cdot\mathbf b=(1)(1)+(1)(0)+(0)(1)+(0)(0)=1 $$ $$ \|\mathbf b\|=\sqrt{1^2+0^2+1^2+0^2}=\sqrt2 $$ $$ \cos(\mathbf t,\mathbf b)=\frac1{\sqrt2\sqrt2}=0.5 $$ ### Step 5. Select only the two requested plot coordinates | Word | x = wrote | y = notes | | --- | --- | --- | | teacher | 1 | 0 | | student | 0 | 0 | | board | 1 | 1 | | book | 0 | 0 | 001122teacherstudent, bookboardwrotenotes The plot uses only the requested two coordinates; cosine uses all context columns. ### Final answer Both requested cosine similarities are **0.5**. Student and book overlap at the origin of the 2-D projection, but their complete four-dimensional vectors differ. The book row is zero, so a cosine involving book would be undefined without an explicit convention. **Yaad rakho:** Har sentence mein target aur context saath? Ek count. Cosine ke liye poori row lo, plot ke liye sirf do columns. --- ## TEE · Q4A · Three preprocessing techniques Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q4a TEE · actual previous paper · 100 marks · 180 minutes Question. Explain any three basic text preprocessing techniques used in Natural Language Processing (NLP). Discuss the purpose of each technique and illustrate them with suitable examples. [10 marks] ### Answer Text preprocessing transforms raw text into a representation suitable for a chosen NLP task. The best steps depend on the task; removing information indiscriminately can reduce performance. ### 1. Tokenization Tokenization divides text into units called tokens, such as words, subwords or sentences. These units are required for counting words and building many model inputs. “The cats are sleeping.” can become [The, cats, are, sleeping, .] under a punctuation-preserving word tokenizer. A different tokenizer may split contractions or punctuation differently, so the convention must be consistent during training and prediction. ### 2. Lowercasing Lowercasing maps uppercase letters to lowercase so that variants such as “Book”, “BOOK” and “book” can share one feature. This reduces vocabulary duplication in a case-insensitive search system. However, “US” and “us” are different, and capitals help NER recognise names. Therefore lowercasing should be chosen according to the application. ### 3. Lemmatization Lemmatization maps an inflected word to its dictionary base form, often using its POS and context. Examples: cats → cat; running/VBG → run; better/JJR → good. It allows a search for “run” to match relevant inflections. A stemmer instead uses simpler truncation rules and can produce a non-word stem; lemmatization is linguistically informed and can need more processing. ### Combined example | Stage | Representation | | --- | --- | | Raw | The cats are running. | | Tokenized | The | cats | are | running | . | | Lowercased | the | cats | are | running | . | | Lemmatized with correct POS | the | cat | be | run | . | These steps do not automatically remove stop words or punctuation. Those are separate choices. For sentiment analysis, preserve negation such as “not”; for syntax parsing, function words and word order are often essential. **Yaad rakho:** Preprocess task dekh ke karo: har word ko blindly delete mat karo. --- ## TEE · Q4B · HMM emission, transition and sequence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q4b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the following POS tagged corpus. Apply Hidden Markov Model (HMM) approach to: 1. Cat/Noun chase/Verb mice/Noun 2. The/Det dog/Noun runs/Verb 3. A/Det cat/Noun eats/Verb cheese/Noun i. Construct the emission probability matrix. ii. Construct the transmission probability matrix. iii. Calculate the probability of sequence of events: The/Det cat/Noun eats/Verb mice/Noun. Note: Include start and end characters for all the sentences. [10 marks] ### Answer ### Given | Sentence | Tagged sequence | | --- | --- | | 1 | cat/Noun chase/Verb mice/Noun | | 2 | the/Det dog/Noun runs/Verb | | 3 | a/Det cat/Noun eats/Verb cheese/Noun | Test words: the cat eats mice. Given tag sequence: Det → Noun → Verb → Noun. Include Start and End for all sentences. Lowercase the corpus so sentence-initial Cat and cat are one word. ### To find Emission probabilities, transition probabilities and the joint probability of the specified tag-and-word sequence. ### Formula: which one and why $$ P(w\mid t)=\frac{C(t,w)}{C(t)} $$ An emission probability asks: among tokens carrying this tag t, what fraction are the word w? $$ P(t_j\mid t_i)=\frac{C(t_i,t_j)}{\sum_k C(t_i,t_k)} $$ A transition probability asks: among transitions leaving tag tᵢ, what fraction go to tⱼ? Include End in the outgoing counts. The paper says “transmission”; the standard HMM term is transition. $$ P(\mathbf w,\mathbf t)=P(t_1\mid\mathrm{Start})\prod_{i=1}^{n}P(w_i\mid t_i)\prod_{i=2}^{n}P(t_i\mid t_{i-1})P(\mathrm{End}\mid t_n) $$ ### Solution ### Step 1. Count tokens for each tag Det has 2 tokens; Noun has 5 tokens; Verb has 3 tokens. The repeated noun cat occurs twice with Noun; every other displayed noun occurs once. The three verb types each occur once. ### Step 2. Build the emission matrix | Tag / word | a | cat | chase | cheese | dog | eats | mice | runs | the | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Det | 1/2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1/2 | | Noun | 0 | 2/5 | 0 | 1/5 | 1/5 | 0 | 1/5 | 0 | 0 | | Verb | 0 | 0 | 1/3 | 0 | 0 | 1/3 | 0 | 1/3 | 0 | Each row sums to one. A zero means no occurrence under that tag in this unsmoothed training corpus; no add-one smoothing was requested. ### Step 3. Write the three tag chains Start → Noun → Verb → Noun → End Start → Det → Noun → Verb → End Start → Det → Noun → Verb → Noun → End ### Step 4. Count outgoing transitions | From | To Det | To Noun | To Verb | To End | Outgoing total | | --- | --- | --- | --- | --- | --- | | Start | 2 | 1 | 0 | 0 | 3 | | Det | 0 | 2 | 0 | 0 | 2 | | Noun | 0 | 0 | 3 | 2 | 5 | | Verb | 0 | 2 | 0 | 1 | 3 | ### Step 5. Divide each row by its outgoing total | From / To | Det | Noun | Verb | End | | --- | --- | --- | --- | --- | | Start | 2/3 | 1/3 | 0 | 0 | | Det | 0 | 1 | 0 | 0 | | Noun | 0 | 0 | 3/5 | 2/5 | | Verb | 0 | 2/3 | 0 | 1/3 | End is terminal, so it has no estimated outgoing row. The four nonterminal rows each sum to one. ### Step 6. Choose the factors for the given sequence | Event in order | Probability | | --- | --- | | Start → Det | 2/3 | | Det emits the | 1/2 | | Det → Noun | 1 | | Noun emits cat | 2/5 | | Noun → Verb | 3/5 | | Verb emits eats | 1/3 | | Verb → Noun | 2/3 | | Noun emits mice | 1/5 | | Noun → End | 2/5 | ### Step 7. Multiply, including the end transition $$ P=\frac23\times\frac12\times1\times\frac25\times\frac35\times\frac13\times\frac23\times\frac15\times\frac25 $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. Start → Det | $$ 1\times\frac{2}{3}=\frac{2}{3} $$ | | 2. Det emits the | $$ \frac{2}{3}\times\frac{1}{2}=\frac{1}{3} $$ | | 3. Det → Noun | $$ \frac{1}{3}\times1=\frac{1}{3} $$ | | 4. Noun emits cat | $$ \frac{1}{3}\times\frac{2}{5}=\frac{2}{15} $$ | | 5. Noun → Verb | $$ \frac{2}{15}\times\frac{3}{5}=\frac{2}{25} $$ | | 6. Verb emits eats | $$ \frac{2}{25}\times\frac{1}{3}=\frac{2}{75} $$ | | 7. Verb → Noun | $$ \frac{2}{75}\times\frac{2}{3}=\frac{4}{225} $$ | | 8. Noun emits mice | $$ \frac{4}{225}\times\frac{1}{5}=\frac{4}{1125} $$ | | 9. Noun → End | $$ \frac{4}{1125}\times\frac{2}{5}=\frac{8}{5625} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac8{5625}\approx0.0014222222 $$ ### Final answer **Joint sequence probability = 8/5625 ≈ 0.0014222222.** This is the probability of the specified words and tags together. It is not the probability of the word sequence summed over every possible tag path. **Yaad rakho:** Tag se tag = transition; tag se word = emission. Dono alternate multiply, end ko mat bhoolo. **Boundary reminder:** The initial factor is P(Det | Start) = 2/3, based on three sentence starts, and the final factor is P(End | Noun) = 2/5. The overall fraction of Det tokens, 2/10, is not the start probability. Using it and omitting End, as in the handwritten key’s last line, computes a different quantity. --- ## TEE · Q5A · TF–IDF matrix Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q5a TEE · actual previous paper · 100 marks · 180 minutes Question. For the following two sentences: Sentence A: “The student read a science book in the library” Sentence B: “The teacher read a history book in the classroom” 1. Construct the TF-IDF matrix for these sentences after removing the stop words {The, a, in}. 2. Identify and comment on the important terms in each sentence based on their TF-IDF scores. [10 marks] ### Answer ### Given Two documents, N = 2. Remove the listed stop words case-insensitively, so both “The” and “the” are removed. Use raw count TF and natural-log IDF, the convention used in the course key. ### To find The term-by-document TF–IDF matrix and the terms that distinguish each sentence. ### Formula: which one and why $$ \mathrm{TF}(t,d)=C(t,d),\qquad \mathrm{IDF}(t)=\ln\frac{N}{\mathrm{df}(t)} $$ $$ \mathrm{TFIDF}(t,d)=\mathrm{TF}(t,d)\,\mathrm{IDF}(t) $$ C(t,d) is the number of occurrences of term t in document d. df(t) is the number of documents containing t, not its total token count. N is the number of documents. IDF downweights terms shared by many documents. ### Solution ### Step 1. Clean and list the tokens A: student, read, science, book, library. B: teacher, read, history, book, classroom. Each retained token appears once in its sentence. ### Step 2. Fix one common vocabulary order student, read, science, book, library, teacher, history, classroom. Each row must refer to the same term in both documents. ### Step 3. Count document frequency and calculate IDF | Term type | df | IDF calculation | | --- | --- | --- | | read, book | 2 | ln(2/2) = ln(1) = 0 | | Every other term | 1 | ln(2/1) = ln(2) ≈ 0.693147 | A term absent from a document has TF = 0. A present term has TF = 1 here. ### Step 4. Multiply TF by IDF | Term | TF in A | TF in B | IDF | TF–IDF A | TF–IDF B | | --- | --- | --- | --- | --- | --- | | student | 1 | 0 | 0.693147 | 0.693147 | 0 | | read | 1 | 1 | 0 | 0 | 0 | | science | 1 | 0 | 0.693147 | 0.693147 | 0 | | book | 1 | 1 | 0 | 0 | 0 | | library | 1 | 0 | 0.693147 | 0.693147 | 0 | | teacher | 0 | 1 | 0.693147 | 0 | 0.693147 | | history | 0 | 1 | 0.693147 | 0 | 0.693147 | | classroom | 0 | 1 | 0.693147 | 0 | 0.693147 | ### Step 5. Interpret the weights A is distinguished by student, science and library. B is distinguished by teacher, history and classroom. “read” and “book” have zero IDF in this tiny unsmoothed two-document collection because they appear in every document; that does not mean those words are meaningless in English. ### Final answer All six document-specific terms receive **0.693147** in their own document and 0 in the other. Shared read and book receive 0 in both. **Yaad rakho:** TF: is document mein kitni baar? IDF: kitne documents mein common? Rare-across-documents term ko zyada weight. If normalized TF is explicitly requested, divide each present-token count by five; each nonzero weight becomes 0.693147/5 = 0.1386294. Do not silently switch conventions. --- ## TEE · Q5B · Naïve Bayes word-sense classification Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q5b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the sentence: “The bat hit the ball hard on the cricket field.” Two possible senses of “bat”: Sense 1: Bat (Sports equipment – cricket/baseball); Sense 2: Bat (Nocturnal flying mammal). | Sentence ID | Context words | Sense | | --- | --- | --- | | 1 | ball, cricket, score, stadium | Sports | | 2 | night, cave, wings, fly | Animal | | 3 | player, run, match, bat | Sports | | 4 | nocturnal, vampire, insect | Animal | | 5 | team, field, bowl, bat | Sports | | 6 | dark, flying, cave, night | Animal | a) Using the Naïve Bayes algorithm, classify the sense of “bat” in the given sentence. Apply Laplace Smoothing wherever applicable. b) Discuss the significance of Laplace Smoothing in Naïve Bayes Classification. [10 marks] ### Answer ### Given There are three Sports examples and three Animal examples. Treat training context lists as bag-of-words token counts. Exclude the ambiguous target “bat” from the test context. Use the fixed training vocabulary; ignore out-of-vocabulary test words hit and hard. The usable test features are ball, cricket and field. ### To find The posterior-score comparison for Sports and Animal, and the role of smoothing. ### Formula: which one and why $$ \mathrm{score}(s)=P(s)\prod_{w\in\mathrm{context}}P(w\mid s) $$ $$ P(w\mid s)=\frac{C(w,s)+1}{N_s+V} $$ P(s) is the class prior, C(w,s) counts occurrences of context word w in class s, Nₛ is the total number of context tokens in that class and V is the shared vocabulary size. Conditional independence lets us multiply word probabilities. The common denominator P(context) is unnecessary when only comparing classes. ### Solution ### Step 1. Calculate the priors $$ P(S)=\frac36=0.5,\qquad P(A)=\frac36=0.5 $$ ### Step 2. Count class tokens and vocabulary Sports has 4 + 4 + 4 = 12 tokens. Animal has 4 + 3 + 4 = 11 tokens. The shared training vocabulary has 20 distinct words: ball, cricket, score, stadium, player, run, match, bat, team, field, bowl, night, cave, wings, fly, nocturnal, vampire, insect, dark, flying. “bat”, “night” and “cave” repeat but count only once each towards V. ### Step 3. Look up each usable context word | Feature | Sports count | Animal count | P(word | Sports) | P(word | Animal) | | --- | --- | --- | --- | --- | | ball | 1 | 0 | (1+1)/(12+20) = 2/32 | (0+1)/(11+20) = 1/31 | | cricket | 1 | 0 | 2/32 | 1/31 | | field | 1 | 0 | 2/32 | 1/31 | ### Step 4. Multiply the Sports score $$ \mathrm{score}(S)=0.5\times\frac2{32}\times\frac2{32}\times\frac2{32}=0.0001220703125 $$ ### Step 5. Multiply the Animal score $$ \mathrm{score}(A)=0.5\times\frac1{31}\times\frac1{31}\times\frac1{31}\approx0.00001678359236 $$ ### Step 6. Compare the scores 0.0001220703125 is greater than 0.00001678359236. Therefore select Sports. Optional normalization gives P(Sports | retained features) ≈ 0.87913; normalization is not needed for the decision. ### Final answer The intended sense is **bat as sports equipment**. Ball, cricket and field favour that sense. **Yaad rakho:** Pehle class prior, phir har context-word likelihood multiply; bigger score wali sense choose karo. ### Why Laplace smoothing matters Without smoothing, all three test features have zero count in Animal. One zero factor makes the entire class score zero, even if other evidence could support that class. Add-one smoothing assigns nonzero probability to every word in the defined vocabulary and redistributes probability mass. It does not mean every unseen word outside the vocabulary can be inserted without changing the vocabulary model. ### If following the supplied paper key exactly The key also multiplies probabilities for hit and hard using the same denominators 32 and 31. That convention produces Sports = 1.19209290 × 10⁻⁷ and Animal ≈ 1.74647163 × 10⁻⁸, again selecting Sports. Its treatment of out-of-vocabulary words is not a normalized fixed-vocabulary model. State your feature convention clearly; the complete calculation above uses a consistent fixed training vocabulary. --- ## TEE · Q6A · Coreference, pronouns and Word2Vec Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q6a TEE · actual previous paper · 100 marks · 180 minutes Question. i. Differentiate between co-reference resolution and pronoun resolution with suitable examples. ii. Explain the working principle of the Continuous Bag of Words (CBOW) model and Skipgram model in Word2Vec. Illustrate how it predicts the target word from its surrounding context. [10 marks] ### Answer ### i. Coreference and pronoun resolution **Coreference resolution** groups expressions that refer to the same entity, including names, descriptions and pronouns. **Pronoun resolution** identifies the referent of a pronoun; it is narrower in scope. | Expression / task | Example | | --- | --- | | Coreference cluster | “Dr Rao arrived. The surgeon opened her bag.” → {Dr Rao, the surgeon, her} | | Pronoun resolution | For the same text, resolve “her” to Dr Rao. | | Difference | Resolving the description “the surgeon” is coreference work even when no pronoun is involved. | ### ii. Word2Vec Word2Vec learns dense word embeddings by predicting words from local context. A dense embedding is a short learned list of real numbers for a word; words used in similar contexts can obtain related vectors. CBOW and Skip-gram use opposite prediction directions. | Feature | CBOW | Skip-gram | | --- | --- | --- | | Input | Surrounding context words | One centre word | | Prediction target | The centre word | Surrounding context words | | Training example | [the, cat, on, the] → sat | sat → the; sat → cat; sat → on; sat → the | | Representation | Combine context embeddings, often by averaging | Use the centre-word embedding for each context prediction | Example sentence: “The cat sat on the mat.” With a radius-two window around sat, the context is [The, cat, on, the]. In CBOW, obtain the four context embeddings, average or sum them, calculate scores for vocabulary words, and train the model to assign high probability to sat. In Skip-gram, use sat as input and create four centre–context training pairs. CBOW: nearby words → combine embeddings → predict centre word Skip-gram: centre word → its embedding → predict nearby words Training adjusts embedding parameters to improve prediction; the final word vectors are the learned representation. Thus the question’s phrase “predicts the target word from surrounding context” describes CBOW specifically; Skip-gram reverses that direction. **Yaad rakho:** CBOW: neighbours se centre. Skip-gram: centre se neighbours. --- ## TEE · Q6B · Hobbs: Ravi and Suresh Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q6b TEE · actual previous paper · 100 marks · 180 minutes Question. i. Explain Hobb’s Algorithm. Discuss the steps with help of example and state its limitation. ii. Sentence: Ravi met Suresh at the station after he arrived from Delhi. Using Hobbs’ algorithm, determine the most likely antecedent of the pronoun “he”. Show the steps of traversal according to the algorithm. [10 marks] ### Answer ### i. Method **Hobbs’ algorithm** is a syntax-based pronoun-resolution procedure that searches constituency parse trees in a specified order. It proposes noun phrases as antecedents and filters incompatible candidates, for example by number and gender agreement. **Read the tree labels first:** NP means noun phrase; S means sentence/clause; VP means verb phrase. X is the current NP or S search node, and p is the upward path used to reach it. Breadth-first means shallower nodes before deeper nodes; left-to-right fixes the order at the same depth. An N-bar (N′) node is the noun-centred part of an NP in a parse using that label. ### Complete ordered procedure 1. **Locate the pronoun.** Start at the NP immediately dominating the pronoun: the smallest NP containing it. 2. **Make the first climb.** Move upward to the first NP or S above the starting NP. Call it X and record the path p. 3. **Search initially to the left.** Traverse branches below X to the left of p, breadth-first and left-to-right. Propose an NP only if another NP or S lies between that candidate and X. Check agreement and other applicable constraints for each proposed candidate; accept the first eligible compatible one. 4. **Check the highest sentence.** If X is the highest S and no candidate has been accepted, search previous sentence trees, most recent first. In each, traverse breadth-first and left-to-right and propose NPs. Accept the first eligible compatible candidate; if the available context contains none, report no resolved antecedent. 5. **Otherwise climb again.** Move from X to the next NP or S above it. Make that node the new X and record the new upward path p. 6. **Test a higher NP.** If the new X is an NP, propose X only when p did not pass through its nominal N-bar child. Apply the same compatibility checks. Use the labels in the supplied parse; do not invent a missing N-bar branch. 7. **Search left again.** Traverse the new X’s branches to the left of p, breadth-first and left-to-right, proposing NPs. The extra intervening-NP/S requirement in Step 3 belongs to the initial search. 8. **Search permitted right branches.** If X is an S, traverse branches to the right of p, breadth-first and left-to-right, but do not descend below any NP or S encountered. Propose the encountered eligible NPs and test their compatibility. 9. **Repeat or finish.** Return to Step 4 until a compatible candidate is accepted or the available sentence trees are exhausted. Record the candidate order and why each candidate was accepted or rejected. **Why the initial restriction matters:** in “John saw him”, the subject NP John is directly under the clause S, with no additional NP or S between them. Step 3 therefore does not propose John as the ordinary local antecedent of him. A reflexive such as himself is a different grammatical case. The worked question below shows the specific parse and traversal; selecting an antecedent is not simply choosing the nearest name. ### ii. Apply the method **Given:** the pronoun he occurs as the subject of the subordinate clause “after he arrived from Delhi”. Assume that this clause is attached to the main verb phrase. The relevant main-clause NPs are Ravi, Suresh and the station. **To find:** the first eligible singular-male antecedent in Hobbs’ syntactic search order. ``` S (main) ├── NP: Ravi ← first higher-left candidate └── VP ├── VP: met [NP Suresh] at [NP the station] └── SBAR: after └── S (embedded) ← first X ├── NP: he ← start └── VP: arrived from Delhi ``` ### Step 1. Start at the pronoun NP Locate NP “he” inside the after-clause. Move to that clause’s S node. No earlier noun phrase in this small clause provides a candidate. ### Step 2. Move to the higher sentence The embedded S is not the highest S, so Step 5 climbs through SBAR and VP to the main S. Step 7 now searches to the left of that new path. This is the later left search, so the extra initial-search restriction from Step 3 is not imposed again. ### Step 3. Examine candidates in structural order The main subject NP Ravi is encountered before the deeper object NP Suresh in the relevant breadth-first search. Ravi is singular and compatible with “he”. “the station” would fail the ordinary person constraint. ### Step 4. Choose the first compatible candidate Select Ravi under the assumed parse and agreement settings. Suresh remains a possible interpretation in ordinary language, but the syntactic preference chooses Ravi first. ### Limitations - The result depends on the accuracy and attachment choices of the parse tree. - Syntactic ordering alone does not model all semantic plausibility, discourse focus or world knowledge. - Agreement can eliminate candidates but cannot distinguish two compatible male singular people. - Long-distance references and sentences with multiple possible parses can remain ambiguous. **Final answer:** Hobbs’ syntactic preference is **Ravi** for this analysis. This is an algorithmic selection, not proof of the speaker’s intended referent. **Yaad rakho:** Nearest naam blindly nahi: tree ki search order aur agreement follow karo. --- ## TEE · Q7A · Text classification pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q7a TEE · actual previous paper · 100 marks · 180 minutes Question. Explain the working of text classification in Natural Language Processing (NLP) and discuss the key steps involved in building a text classification system in context of real-world applications such as spam detection or news categorization. [10 marks] ### Answer **Text classification** assigns one or more predefined labels to a document or message based on its content. In spam detection the labels may be spam and not-spam; in news categorization they may be sports, business and politics. Labelled text → Prepare and split data → Represent text → Train classifier → Evaluate → Classify new text 1. **Define the labels and task.** Decide whether each text gets one label or multiple labels. Establish what counts as spam so annotators use a consistent rule. 2. **Collect and label representative examples.** Gather messages from the intended application and assign reliable labels. Include enough examples of each class and protect private information. 3. **Split training, validation and test data.** Use training data to learn, validation data to choose settings and an untouched test set for final evaluation. Keep duplicate or closely related texts from leaking between splits. 4. **Preprocess consistently.** Tokenize and normalize only as appropriate. In spam detection, URLs, currency symbols and capitalisation may contain useful information. 5. **Create features.** A classical baseline uses BoW or TF–IDF. A neural system may use embeddings or a pretrained contextual model. Fit the vocabulary and IDF on training data only. 6. **Train a classifier.** For example, multinomial Naïve Bayes estimates class priors and word likelihoods; logistic regression learns a decision boundary. Training uses the labelled examples. 7. **Evaluate.** Inspect a confusion matrix, precision, recall and F1 as well as accuracy. False spam flags may hide legitimate mail, so the costs of errors matter. 8. **Deploy and monitor.** Apply the same preprocessing and feature mapping to new texts. Monitor changing vocabulary and error patterns, and retrain when suitable new labelled data becomes available. ### Example prediction A message such as “Claim your free prize now” is transformed into its feature representation. The classifier compares the evidence for the two labels and outputs spam if its decision score crosses the chosen threshold. The label comes from learned patterns; the mere presence of one word should not be treated as a universally correct rule. **Yaad rakho:** Labelled examples → features → model → evaluation → new text ka label. --- ## TEE · Q7B · Corpora and Jaccard similarity Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-final-2025--final-2025-q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-final-2025/final-2025-q7b TEE · actual previous paper · 100 marks · 180 minutes Question. i. Differentiate between a flat text corpus and an annotated corpus, and discuss how each can be applied in various NLP tasks with suitable examples. ii. Explain a real-world scenario where the Jaccard Similarity measure is more appropriate than Euclidean or Manhattan distance, and justify why Jaccard is the best choice in that case. [10 marks] ### Answer ### i. Flat and annotated corpora A **corpus** is a collection of language data. A flat text corpus stores text without the linguistic labels needed by a particular task. An annotated corpus adds labels or structure such as POS tags, entity spans, parse trees or sentence sentiment. | Basis | Flat text corpus | Annotated corpus | | --- | --- | --- | | Example | A collection of plain news articles | The same articles with PERSON/ORG spans marked | | Information | The original text and possibly simple metadata | Text plus explicit task-related linguistic labels | | Applications | Word frequency, collocations and unsupervised language-model training | Supervised POS tagging, NER or parser training and evaluation | | Preparation | Relatively inexpensive to gather, but still needs cleaning | Requires annotation guidelines, labour and quality checks | “Flat” describes the lack of the annotations being discussed, not an absence of all file organization. Both corpus types may still have document boundaries and metadata. ### ii. Jaccard in a set-based application Consider recommending articles using their topic-tag sets. Article A has {AI, NLP, ethics}; article B has {AI, NLP, healthcare}. We care about overlap among present tags, not how many irrelevant tags are absent from both. ### Given A = {AI, NLP, ethics}; B = {AI, NLP, healthcare}. ### To find The fraction of distinct present tags shared by the articles. ### Formula and why $$ J(A,B)=\frac{|A\cap B|}{|A\cup B|} $$ The intersection counts common items; the union counts all distinct items present in either set. ### Step 1. Find common tags A ∩ B = {AI, NLP}, so the intersection size is 2. ### Step 2. Find all distinct present tags A ∪ B = {AI, NLP, ethics, healthcare}, so the union size is 4. ### Step 3. Divide $$ J(A,B)=\frac24=0.5 $$ Jaccard directly measures proportional overlap and ignores joint absences. Euclidean or Manhattan distance on binary tag vectors counts mismatches but does not by itself normalize by the union size. If frequency or magnitude were meaningful, those distance measures could be more appropriate. Here, membership-only tags and sparse sets make Jaccard a suitable choice. **Yaad rakho:** Jaccard: common present items / total distinct present items. --- ## TEE · Write the whole paper under exam conditions Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--instructions/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/instructions TEE · actual previous paper · 100 marks · 180 minutes Use this actual previous paper as a three-hour, 100-mark practice paper. Q1 is compulsory (20 marks); choose any four full questions from Q2–Q7 (20 marks each). Each original question and its complete answer follow. Read the question, write your answer in your notebook, then compare the working below it. Budget roughly 36 minutes per attempted 20-mark question, including its subparts. Start each full question on a fresh page, as the supplied papers instruct. This is an old paper used for timed practice, not a forecast of the next TEE. --- ## TEE · Q1A · Generic NLP pipeline Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q1a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q1a TEE · actual previous paper · 100 marks · 180 minutes Question. Draw block diagram of generic NLP pipeline and explain the task associated with each block. [5 marks] ### Answer An NLP pipeline is an ordered workflow that transforms language data into a task-specific result. The exact components depend on the application; a generic supervised pipeline is shown below. Collect text → Clean and preprocess → Represent text → Train / apply model → Evaluate and deliver output | Block | Task | Example for sentiment analysis | | --- | --- | --- | | Data acquisition | Collect representative text and any required labels | Reviews labelled positive or negative | | Cleaning and preprocessing | Handle noise, tokenize and normalize appropriately | Remove HTML while preserving “not” | | Text representation | Convert tokens into usable model features | TF–IDF values or contextual embeddings | | Modeling | Learn patterns from training data and predict new labels | Train a classifier on labelled review features | | Evaluation | Measure performance on unseen held-out data and inspect errors | Precision, recall, F1 and confusion matrix | | Deployment and feedback | Deliver predictions and monitor changing data | Show review sentiment and monitor new product vocabulary | Training/validation/test separation must precede fitting the learned preprocessing and feature statistics. New inputs must use the same feature mapping as training. Not every task uses every linguistic level explicitly: a modern end-to-end model may combine several operations internally. **Yaad rakho:** Text lo → prepare karo → numbers banao → model lagao → result check aur use karo. --- ## TEE · Q1B · Content and function words Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q1b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q1b TEE · actual previous paper · 100 marks · 180 minutes Question. i. Define function words and content words. ii. Which category (function words and content words) belong to open class and closed class? Justify your answer. iii. For the sentence “The small cat is sleeping under the tree”, identify the function words and content words. [5 marks] ### Answer **Content words** carry much of a sentence’s lexical meaning, usually including nouns, main verbs, adjectives and many adverbs. **Function words** primarily express grammatical relationships, including determiners, prepositions, conjunctions, pronouns and auxiliary verbs. | Category | Typical class | Reason | | --- | --- | --- | | Content words | Open class | New nouns, verbs and adjectives enter the language readily, such as new technology names | | Function words | Closed class | The grammatical inventory is relatively stable; new determiners or pronouns are much less frequently added | | Word | Role in this sentence | Category | | --- | --- | --- | | The | Determiner | Function | | small | Adjective | Content | | cat | Noun | Content | | is | Auxiliary supporting progressive tense | Function | | sleeping | Main verb | Content | | under | Preposition | Function | | the | Determiner | Function | | tree | Noun | Content | **Final answer:** content words = small, cat, sleeping, tree; function words = The, is, under, the. “is” is auxiliary here; a word’s role must be judged in context. “Closed” means relatively resistant to new members, not that change is logically impossible. **Yaad rakho:** Content words scene batate hain; function words grammar ka connection banate hain. --- ## TEE · Q1C · Word2Vec and GloVe Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q1c/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q1c TEE · actual previous paper · 100 marks · 180 minutes Question. Differentiate between word2vec and GloVe in terms of their learning strategies. Which type of corpus would be more suitable for each method, and why? [5 marks] ### Answer Both Word2Vec and GloVe learn dense word vectors from distributional information: words occurring in related contexts tend to receive related representations. They differ mainly in the training objective and how corpus co-occurrence evidence is used. | Basis | Word2Vec | GloVe | | --- | --- | --- | | Learning strategy | Predictive: CBOW predicts a centre word; Skip-gram predicts nearby words | Count-based objective fitted to global word–context co-occurrence statistics | | Training evidence | Local context windows converted into prediction examples | A corpus-wide co-occurrence matrix and relationships among counts | | Parameters learned | Word/context embeddings that make context predictions accurate | Word/context embeddings and biases that fit weighted log co-occurrence counts | | Corpus considerations | Large token sequences with useful local context, including domain-specific text | A sufficiently large corpus from which reliable global co-occurrence statistics can be collected | For a domain such as medicine, either method can benefit from a representative medical corpus. Word2Vec is convenient when generating local training windows directly from a stream of tokens. GloVe is natural when a global sparse co-occurrence matrix can be constructed and stored. Corpus size, domain match, preprocessing and available compute matter; there is no rule that Word2Vec requires informal text while GloVe requires formal text. Both standard forms produce one static vector per vocabulary word, so multiple senses are blended. They do not automatically provide a different vector for the same word in each sentence. **Yaad rakho:** Word2Vec predicts local neighbours; GloVe fits global co-occurrence counts. --- ## TEE · Q1D · Anaphora, cataphora and coreference Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q1d/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q1d TEE · actual previous paper · 100 marks · 180 minutes Question. Differentiate between cataphora and anaphora in terms of direction of reference. How coreference resolution is different from anaphora reference? [5 marks] ### Answer **Anaphora** points backward to an expression or information introduced earlier. **Cataphora** points forward to an expression introduced later. | Relation | Example | Direction | | --- | --- | --- | | Anaphora | “Riya arrived. She smiled.” She refers to the earlier Riya. | Backward | | Cataphora | “Before she spoke, Riya checked the microphone.” She refers to the later Riya. | Forward | **Coreference** means two or more expressions refer to the same entity. Coreference resolution groups such expressions, regardless of whether they are names, descriptions or pronouns and regardless of which is mentioned first. For “Dr Sen arrived. The surgeon smiled. She waved,” {Dr Sen, the surgeon, She} is a coreference cluster. The latter two mentions are also backward-looking references. Coreference is an identity relation; anaphora describes a dependency on prior discourse and is not always identity. For example, “I bought a house. The roof leaks” contains a bridging reference: the roof depends on the house for interpretation, but roof and house are not the same entity. **Yaad rakho:** Ana = pehle ki taraf; cata = aage ki taraf; coreference = same entity. --- ## TEE · Q2A · NLP tasks and syntax versus semantics Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q2a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q2a TEE · actual previous paper · 100 marks · 180 minutes Question. i. Discuss five different tasks associated with Natural Language Processing. ii. Compare and contrast syntactic analysis and semantic analysis in NLP. Support your explanation with examples. [10 marks] ### Answer ### i. Five NLP tasks | Task | What it does | Example | | --- | --- | --- | | Machine translation | Expresses source-language content in a target language | English “Good morning” → Hindi greeting | | Sentiment analysis | Identifies the opinion or attitude expressed | “The service was excellent” → positive sentiment | | Named Entity Recognition | Finds and types entity spans | “Asha joined Google” → PERSON and ORGANIZATION | | Question answering | Returns an answer supported by a question’s relevant information | From a passage, identify who discovered a described result | | Text classification | Assigns predefined category labels | Classify an email as spam or legitimate | ### ii. Syntactic and semantic analysis | Basis | Syntactic analysis | Semantic analysis | | --- | --- | --- | | Main question | How are words grammatically organised? | What do the words and sentence mean? | | Typical output | POS tags, constituency or dependency structure | Word senses, semantic relations or a meaning representation | | Knowledge used | Grammar rules and structural patterns | Lexical meaning, context and semantic constraints | | Example | Identify “The boy” as subject NP and “ate an apple” as VP | Interpret the boy as eater and apple as the thing eaten | “Colourless green ideas sleep furiously” can have a plausible English grammatical structure while remaining semantically unusual. Syntax alone therefore does not guarantee ordinary meaning. Conversely, “boy eat apple yesterday” conveys a possible event despite grammatical errors. The analyses support each other. In “I saw the man with a telescope,” syntactic PP attachment determines which meaning is available, and semantic knowledge can help choose a plausible attachment. A full NLP system often needs both. **Yaad rakho:** Syntax = sentence ka structure; semantics = us structure ka meaning. --- ## TEE · Q2B · Laplace smoothing on a second corpus Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q2b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q2b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the following training corpus: 1. cats chase mice in the garden 2. dogs bark loudly at night 3. birds sing in the morning 4. children play games in the park Using a bigram language model with Laplace smoothing, evaluate the probability of the test sentence: “cats bark in the park.” Assume and as start and end of the sentence. [10 marks] ### Answer ### Given - cats chase mice in the garden - dogs bark loudly at night - birds sing in the morning - children play games in the park Test: cats bark in the park. Add one start token and one end token to every training and test sentence. ### To find The Laplace-smoothed bigram probability of the complete test sentence, including its end token. ### Formula: which one and why $$ P_{+1}(w_i\mid w_{i-1})=\frac{C(w_{i-1},w_i)+1}{C(w_{i-1})+V} $$ C(previous,current) counts that adjacent pair in the training corpus. C(previous) counts its occurrences as a history. V is the vocabulary size. Add-one smoothing is necessary because an unseen pair must not force the whole sentence probability to zero. $$ P(w_1,\ldots,w_n,\langle/s\rangle\mid\langle s\rangle)=\prod_{i=1}^{n+1}P(w_i\mid w_{i-1}) $$ ### Solution ### Step 1. Fix the token and vocabulary convention Lowercase words and remove the final punctuation. The supplied course key counts both boundary tokens in V. Following that convention: at, bark, birds, cats, chase, children, dogs, games, garden, in, loudly, mice, morning, night, park, play, sing, the, , gives **V = 20**. The start marker supplies history; it is not an extra probability factor before the first word. ### Step 2. List every required transition The complete test chain is → cats → bark → in → the → park → . ### Step 3. Count first, then add one | Required bigram | History count | Pair count | Smoothed probability | | --- | --- | --- | --- | | → cats | 4 | 1 | (1+1)/(4+20) = 1/12 | | cats → bark | 1 | 0 | (0+1)/(1+20) = 1/21 | | bark → in | 1 | 0 | (0+1)/(1+20) = 1/21 | | in → the | 3 | 3 | (3+1)/(3+20) = 4/23 | | the → park | 3 | 1 | (1+1)/(3+20) = 2/23 | | park → | 1 | 1 | (1+1)/(1+20) = 2/21 | Count only adjacent words, never words that merely appear in the same sentence. The unseen pairs have count zero, so their numerator becomes one. ### Step 4. Multiply all six probabilities $$ P=\frac{2}{24}\times\frac{1}{21}\times\frac{1}{21}\times\frac{4}{23}\times\frac{2}{23}\times\frac{2}{21} $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. → cats | $$ 1\times\frac{1}{12}=\frac{1}{12} $$ | | 2. cats → bark | $$ \frac{1}{12}\times\frac{1}{21}=\frac{1}{252} $$ | | 3. bark → in | $$ \frac{1}{252}\times\frac{1}{21}=\frac{1}{5292} $$ | | 4. in → the | $$ \frac{1}{5292}\times\frac{4}{23}=\frac{1}{30429} $$ | | 5. the → park | $$ \frac{1}{30429}\times\frac{2}{23}=\frac{2}{699867} $$ | | 6. park → | $$ \frac{2}{699867}\times\frac{2}{21}=\frac{4}{14697207} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{4}{14697207}\approx 2.72160554\times10^{-7} $$ ### Step 5. Interpret the result This is a joint sentence probability, so a small value is normal. Do not add transition probabilities; each word must occur in sequence, so the chain multiplies them. ### Final answer **P ≈ 2.72160554 × 10-7** with the stated course vocabulary convention. **Yaad rakho:** Pair count mein 1 jodo, previous-word count mein V; phir saare fractions multiply karo. **Convention note:** Some implementations exclude from the predicted vocabulary. Their V is one smaller and their numerical probability differs. In an exam, state V explicitly and follow the convention specified in the question or class solution. --- ## TEE · Q3A · Top-down parsing Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q3a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q3a TEE · actual previous paper · 100 marks · 180 minutes Question. In the context of natural language processing, explain the concept of top-down and bottom-up parsing approaches. Apply top-down syntax parsing approach to the sentence “The tall boy saw a girl in the park.” and draw a parse tree using the CFG grammar rules: S → NP VP; NP → Det Adj N | Det N; VP → V NP | VP PP; PP → P NP. Lexicon: Det → {the | a}; Adj → {tall}; N → {boy | girl | park}; V → {saw}; P → {in}. [10 marks] ### Answer | Aspect | Top-down | Bottom-up | | --- | --- | --- | | Start | The start symbol S | The input words | | Operation | Expand a nonterminal using grammar productions | Combine words and constituents using productions in reverse | | Success | Leaves match the complete input | All input is reduced to S | | Possible difficulty | Naive recursive descent can loop on left recursion or need backtracking | A naive shift-reduce parser may face shift/reduce conflicts | ### Given The sentence contains the tokens the / tall / boy / saw / a / girl / in / the / park. Lowercase the initial “The” to match the lexicon. The full stop is punctuation outside the listed grammar. ### To find A derivation beginning at S and a parse tree matching the whole sentence. ### Rule: which one and why A context-free production replaces one nonterminal with its right-hand sequence. A top-down derivation begins at S and expands the desired structure. The supplied grammar attaches the PP through VP → VP PP; it does not provide NP → NP PP. ### Step 1. Expand the sentence S ⇒ NP VP. Choose NP → Det Adj N for “the tall boy”. ### Step 2. Match the subject words Det ⇒ the, Adj ⇒ tall, N ⇒ boy. The remaining material must be generated by VP. ### Step 3. Allow a verb phrase followed by a PP Expand the outer VP using VP → VP PP. Then expand the inner VP using VP → V NP. This finite derivation selects the recursive rule once. ### Step 4. Match the verb and object V ⇒ saw. Choose NP → Det N, then Det ⇒ a and N ⇒ girl. ### Step 5. Expand the prepositional phrase PP ⇒ P NP. P ⇒ in. NP ⇒ Det N, with Det ⇒ the and N ⇒ park. ### Step 6. Check the complete leaf order The leaves read “the tall boy saw a girl in the park”, exactly the normalized input. The PP “in the park” attaches to the VP under this supplied grammar. SNPVPDetAdjNVPPPVNPPNPDetNDetNThetallboysawagirlinthepark Top-down parse tree showing the PP attached to VP. ### Complete derivation S ⇒ NP VP ⇒ Det Adj N VP ⇒ the tall boy VP ⇒ the tall boy VP PP ⇒ the tall boy V NP PP ⇒ the tall boy saw Det N PP ⇒ the tall boy saw a girl P NP ⇒ the tall boy saw a girl in Det N ⇒ the tall boy saw a girl in the park. **Implementation caution:** VP → VP PP is left-recursive. A naive recursive-descent implementation may loop; it must transform or otherwise handle the grammar. A valid finite top-down derivation, as shown above, still exists. **Yaad rakho:** Rule jo diya hai wahi use karo: yahan “in the park” VP ka part hai. --- ## TEE · Q3B · Lesk word-sense disambiguation Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q3b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q3b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the sentence: “The bank can ensure your deposits are safe.” The word “bank” has the following two possible senses (WordNet-style glosses): Bank₁: Sloping land especially beside a body of water. Bank₂: A financial institution that accepts deposits and channels the money into lending activities. i. Apply the LESK algorithm to determine the correct sense of the word “bank” in the given sentence. Show the steps clearly. ii. Briefly discuss limitations of the LESK algorithm in handling ambiguous words in real-world contexts. [10 marks] ### Answer ### Given Context sentence and the two supplied dictionary glosses. Use a simplified Lesk method: lowercase, remove function words, exclude the target bank and compare distinct content-word overlap. Do not add synonyms that the question does not supply. ### To find Which candidate gloss has the greatest overlap with the local context. ### Formula: which one and why $$ \mathrm{score}(s)=|C\cap G_s|,\qquad \hat s=\arg\max_s\mathrm{score}(s) $$ C is the set of context content words. Gₛ is the content-word set of the gloss for sense s. The intersection contains words appearing in both; its size is the score. This is an overlap count, not a probability. ### Solution ### Step 1. Prepare the context Remove The, can, your, are and the target bank. A reasonable content set is C = {ensure, deposits, safe}. If “can” is retained under another fixed stop-list it matches neither gloss and does not change the result. ### Step 2. Prepare Bank₁’s gloss G₁ = {sloping, land, especially, beside, body, water}. There is no shared word with {ensure, deposits, safe}. ### Step 3. Calculate Bank₁’s score $$ C\cap G_1=\varnothing,\qquad \mathrm{score}(1)=0 $$ ### Step 4. Prepare Bank₂’s gloss and count overlap G₂ = {financial, institution, accepts, deposits, channels, money, lending, activities}. The shared word is deposits. ### Step 5. Calculate Bank₂’s score and select $$ C\cap G_2=\{\mathrm{deposits}\},\qquad \mathrm{score}(2)=1 $$ 1 > 0, so choose the financial-institution sense. ### Final answer **Bank₂: financial institution.** The explicit overlap “deposits” supports this choice. **Yaad rakho:** Lesk: context aur har meaning ki definition mein common words count karo; highest overlap choose karo. ### ii. Limitations - Short contexts or short glosses may produce no overlaps or ties. - Exact word matching misses related words and synonyms, such as deposit versus savings, unless an extension is defined. - Irrelevant frequent words can create misleading overlaps if preprocessing is poor. - Dictionary glosses may omit a domain-specific sense or fail to express the context’s wording. - Simplified Lesk uses limited local information and may miss discourse or real-world knowledge. Lemmatization, richer glosses and semantic similarity can improve variants, but the exact variant and tie-breaking rule should be stated. --- ## TEE · Q4A · Bag of Words and three cosine similarities Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q4a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q4a TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the following three documents: Document A: “The dog chased the cat.” Document B: “The cat chased the mouse.” Document C: “The dog ran after the cat.” Using the Bag of Words (BoW) model, represent each document as a vector after removing the stop words {The, by, was}. i. Construct the term-document matrix. ii. Compute the cosine similarity between each pair of documents (A–B, A–C, and B–C). iii. Interpret and comment on the results obtained in terms of document similarity. [10 marks] ### Answer ### Given Three documents. Lowercase and remove punctuation. Remove the specified stop words case-insensitively; “the” is removed everywhere. Keep “after” because it is not in the supplied stop list. Use raw word counts. ### To find A common-vocabulary count matrix, three cosine values and their interpretation. ### Formula: which one and why $$ \mathrm{BoW}_j(d)=C(v_j,d) $$ vⱼ is vocabulary term j. The coordinate stores its count in document d; order in the sentence is not retained. $$ \cos(\mathbf a,\mathbf b)=\frac{\sum_j a_jb_j}{\sqrt{\sum_j a_j^2}\sqrt{\sum_j b_j^2}} $$ ### Solution ### Step 1. Clean each document A: dog chased cat. B: cat chased mouse. C: dog ran after cat. ### Step 2. Fix vocabulary and matrix Vocabulary: dog, chased, cat, mouse, ran, after. | Term | A | B | C | | --- | --- | --- | --- | | dog | 1 | 0 | 1 | | chased | 1 | 1 | 0 | | cat | 1 | 1 | 1 | | mouse | 0 | 1 | 0 | | ran | 0 | 0 | 1 | | after | 0 | 0 | 1 | A = [1,1,1,0,0,0]; B = [0,1,1,1,0,0]; C = [1,0,1,0,1,1]. ### Step 3. Calculate vector lengths $$ \|A\|=\sqrt{1^2+1^2+1^2}=\sqrt3 $$ $$ \|B\|=\sqrt{1^2+1^2+1^2}=\sqrt3 $$ $$ \|C\|=\sqrt{1^2+1^2+1^2+1^2}=2 $$ ### Step 4. Calculate A–B $$ A\cdot B=(1)(0)+(1)(1)+(1)(1)+(0)(1)+(0)(0)+(0)(0)=2 $$ $$ \cos(A,B)=\frac2{\sqrt3\sqrt3}=\frac23\approx0.666667 $$ ### Step 5. Calculate A–C $$ A\cdot C=(1)(1)+(1)(0)+(1)(1)+(0)(0)+(0)(1)+(0)(1)=2 $$ $$ \cos(A,C)=\frac2{\sqrt3\cdot2}=\frac1{\sqrt3}\approx0.577350 $$ ### Step 6. Calculate B–C $$ B\cdot C=(0)(1)+(1)(0)+(1)(1)+(1)(0)+(0)(1)+(0)(1)=1 $$ $$ \cos(B,C)=\frac1{\sqrt3\cdot2}\approx0.288675 $$ ### Final answer **A–B = 0.666667; A–C = 0.577350; B–C = 0.288675.** A and B have the highest lexical overlap under this representation. B and C share only cat. BoW similarity reflects token overlap, not full semantic equivalence or word order. **Yaad rakho:** Same vocabulary order rakho. Corresponding counts multiply-add; phir dono lengths se divide. --- ## TEE · Q4B · A second HMM sequence Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q4b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q4b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the following POS tagged corpus. Apply Hidden Markov Model (HMM) approach to: 1. Dog/Noun eats/Verb bread/Noun 2. The/Det cat/Noun sleeps/Verb 3. A/Det dog/Noun eats/Verb fish/Noun i. Construct the emission probability matrix. ii. Construct the transmission probability matrix. iii. Calculate the probability of the given sequence of events: The/Det dog/Noun eats/Verb bread/Noun. Note: Include start and end characters for all the sentences. [10 marks] ### Answer The repeated verb eats has a different emission count from the final paper; recalculate it rather than copying the previous numerical result. ### Given | Sentence | Tagged sequence | | --- | --- | | 1 | dog/Noun eats/Verb bread/Noun | | 2 | the/Det cat/Noun sleeps/Verb | | 3 | a/Det dog/Noun eats/Verb fish/Noun | Lowercase the corpus. Test sequence: the/Det dog/Noun eats/Verb bread/Noun. Add Start and End markers. ### To find Emission and transition matrices, then the probability of the supplied tagged sequence. ### Formula: which one and why $$ P(w\mid t)=\frac{C(t,w)}{C(t)} $$ $$ P(t_j\mid t_i)=\frac{C(t_i,t_j)}{\sum_kC(t_i,t_k)} $$ Emission divides the word’s tag-specific count by the total tokens with that tag. Transition divides the tag-pair count by all transitions leaving the previous tag, including End. $$ P(\mathbf w,\mathbf t)=P(t_1\mid\mathrm{Start})\prod_iP(w_i\mid t_i)\prod_{i=2}^{n}P(t_i\mid t_{i-1})P(\mathrm{End}\mid t_n) $$ ### Solution ### Step 1. Count tag totals and word frequencies Det: the=1, a=1, total2. Noun: dog=2, bread=1, cat=1, fish=1, total5. Verb: eats=2, sleeps=1, total3. ### Step 2. Construct emission matrix | Tag / word | a | bread | cat | dog | eats | fish | sleeps | the | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Det | 1/2 | 0 | 0 | 0 | 0 | 0 | 0 | 1/2 | | Noun | 0 | 1/5 | 1/5 | 2/5 | 0 | 1/5 | 0 | 0 | | Verb | 0 | 0 | 0 | 0 | 2/3 | 0 | 1/3 | 0 | ### Step 3. Count transitions from the three chains Start → Noun → Verb → Noun → End Start → Det → Noun → Verb → End Start → Det → Noun → Verb → Noun → End | From | Det count | Noun count | Verb count | End count | Total | | --- | --- | --- | --- | --- | --- | | Start | 2 | 1 | 0 | 0 | 3 | | Det | 0 | 2 | 0 | 0 | 2 | | Noun | 0 | 0 | 3 | 2 | 5 | | Verb | 0 | 2 | 0 | 1 | 3 | ### Step 4. Divide outgoing counts | From / to | Det | Noun | Verb | End | | --- | --- | --- | --- | --- | | Start | 2/3 | 1/3 | 0 | 0 | | Det | 0 | 1 | 0 | 0 | | Noun | 0 | 0 | 3/5 | 2/5 | | Verb | 0 | 2/3 | 0 | 1/3 | All nonterminal rows sum to one. No smoothing is requested. ### Step 5. Write every sequence factor | Event | Probability | | --- | --- | | Start → Det | 2/3 | | Det emits the | 1/2 | | Det → Noun | 1 | | Noun emits dog | 2/5 | | Noun → Verb | 3/5 | | Verb emits eats | 2/3 | | Verb → Noun | 2/3 | | Noun emits bread | 1/5 | | Noun → End | 2/5 | ### Step 6. Multiply all factors $$ P=\frac23\times\frac12\times1\times\frac25\times\frac35\times\frac23\times\frac23\times\frac15\times\frac25 $$ Start with subtotal 1. At each row, multiply the previous subtotal by exactly one next factor. Simplify the fraction after multiplication, but do not round it. | Factor being included | Previous subtotal × next factor = new subtotal | | --- | --- | | 1. Start → Det | $$ 1\times\frac{2}{3}=\frac{2}{3} $$ | | 2. Det emits the | $$ \frac{2}{3}\times\frac{1}{2}=\frac{1}{3} $$ | | 3. Det → Noun | $$ \frac{1}{3}\times1=\frac{1}{3} $$ | | 4. Noun emits dog | $$ \frac{1}{3}\times\frac{2}{5}=\frac{2}{15} $$ | | 5. Noun → Verb | $$ \frac{2}{15}\times\frac{3}{5}=\frac{2}{25} $$ | | 6. Verb emits eats | $$ \frac{2}{25}\times\frac{2}{3}=\frac{4}{75} $$ | | 7. Verb → Noun | $$ \frac{4}{75}\times\frac{2}{3}=\frac{8}{225} $$ | | 8. Noun emits bread | $$ \frac{8}{225}\times\frac{1}{5}=\frac{8}{1125} $$ | | 9. Noun → End | $$ \frac{8}{1125}\times\frac{2}{5}=\frac{16}{5625} $$ | The last subtotal is the complete product. Only now convert the fraction to a decimal. $$ P=\frac{16}{5625}\approx0.0028444444 $$ ### Final answer **P = 16/5625 ≈ 0.0028444444** for these words and specified tags jointly. The two occurrences of eats give P(eats | Verb) = 2/3. **Yaad rakho:** Same tag structure ho sakta hai, lekin word emission counts dobara count karna. **Boundary reminder:** Include P(Det | Start) = 2/3 and P(End | Noun) = 2/5. The handwritten key’s final line instead inserts the overall Det fraction 2/10 and omits End; the complete boundary-aware probability requested here is 0.0028444444. --- ## TEE · Q5A · Why TF–IDF is useful Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q5a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q5a TEE · actual previous paper · 100 marks · 180 minutes Question. Justify with an appropriate example the importance of TF-IDF technique in the document representation and information retrieval task. [10 marks] ### Answer **TF–IDF** represents each document using weights that combine within-document term frequency with how unusual a term is across the collection. It supports information retrieval by giving larger weight to terms that are frequent in a document but not present in every document. ### Given Use three short documents: D₁ = “solar energy solar”; D₂ = “wind energy”; D₃ = “energy policy”. Use raw-count TF and natural-log unsmoothed IDF. N = 3. ### To find A TF–IDF representation illustrating why distinctive terms help retrieve relevant documents. ### Formula: which one and why $$ \mathrm{TFIDF}(t,d)=C(t,d)\ln\frac{N}{\mathrm{df}(t)} $$ C counts occurrences within a document; df counts documents containing the term. The logarithm reduces the scale of very large rarity ratios. ### Solution ### Step 1. Count term occurrences | Term | D₁ | D₂ | D₃ | df | | --- | --- | --- | --- | --- | | solar | 2 | 0 | 0 | 1 | | energy | 1 | 1 | 1 | 3 | | wind | 0 | 1 | 0 | 1 | | policy | 0 | 0 | 1 | 1 | ### Step 2. Calculate IDF $$ \mathrm{IDF}(solar)=\mathrm{IDF}(wind)=\mathrm{IDF}(policy)=\ln3\approx1.098612 $$ $$ \mathrm{IDF}(energy)=\ln(3/3)=0 $$ ### Step 3. Multiply each count by its IDF | Term | D₁ weight | D₂ weight | D₃ weight | | --- | --- | --- | --- | | solar | 2 × 1.098612 = 2.197224 | 0 | 0 | | energy | 0 | 0 | 0 | | wind | 0 | 1.098612 | 0 | | policy | 0 | 0 | 1.098612 | ### Step 4. Connect the representation to retrieval A query containing “solar” matches D₁’s high solar coordinate. A query containing “wind” matches D₂. “energy” alone cannot distinguish these documents under this unsmoothed collection because it occurs in all three. A retrieval system can compare query and document vectors, commonly by cosine similarity. ### Final answer TF–IDF preserves distinctive lexical evidence and reduces dominance by collection-wide frequent terms. D₁ is most relevant to a solar query under these features. **Yaad rakho:** Document mein frequent + collection mein rare = high TF–IDF. ### Strengths and limits The representation is sparse, interpretable and useful as a fast baseline. However, basic TF–IDF ignores word order and does not recognise synonyms or multiple word senses automatically. Its weights depend on the collection and on the chosen TF/IDF convention. Fit IDF on training or indexed documents, not on a hidden test set. --- ## TEE · Q5B · Sentence-window vectors and cosine Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q5b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q5b TEE · actual previous paper · 100 marks · 180 minutes Question. Consider the paragraph: The farmer planted seeds in the soil. The rain watered the soil. The sun helped the plants grow. The farmer harvested the plants. The soil became rich and fertile. Target words: farmer, soil, sun, plant. Context words: plant, water, grow, harvest. i. Construct a co-occurrence matrix for the target and context words. Take the window size as one whole sentence. ii. Evaluate the similarity between the vectors of the target words “farmer-soil”, “farmer-sun”, and “farmer-plant” using the cosine similarity. iii. Plot the target word vectors by considering only two dimensions: “planted” and “harvested”. [10 marks] ### Answer ### Given Five sentence windows. For alignment with the supplied matrix, target plant means the noun plants, occurring in sentences 3 and 4. Context plant covers planted and plants; watered → water; harvested → harvest. Count a target–context pair once per sentence in which both are present, including the plant–plant sentence-presence diagonal. ### To find The target–context matrix, three cosine similarities and a two-coordinate plot. ### Formula: which one and why $$ M_{ij}=\sum_s\mathbf1[\text{target}_i\text{ and context}_j\text{ present in }s] $$ $$ \cos(\mathbf a,\mathbf b)=\frac{\sum_j a_jb_j}{\sqrt{\sum_j a_j^2}\sqrt{\sum_j b_j^2}} $$ This is a sentence-presence matrix, not a position-based sliding window that excludes a token from its own context. Stating the convention is necessary because those alternatives give a different diagonal. ### Solution ### Step 1. Normalize and identify each sentence’s contribution | Sentence | Targets present | Context lemmas present | | --- | --- | --- | | 1 | farmer, soil | plant | | 2 | soil | water | | 3 | sun, plant | plant, grow | | 4 | farmer, plant | plant, harvest | | 5 | soil | none | ### Step 2. Fill all matrix rows | Target | plant | water | grow | harvest | | --- | --- | --- | --- | --- | | farmer | 2 | 0 | 0 | 1 | | soil | 1 | 1 | 0 | 0 | | sun | 1 | 0 | 1 | 0 | | plant | 2 | 0 | 1 | 1 | Farmer meets context plant in sentences 1 and 4. Soil meets plant in sentence 1 and water in sentence 2. Sun meets plant and grow in sentence 3. The noun plant occurs in sentences 3 and 4, producing its last row. ### Step 3. Compute farmer–soil $$ \mathbf f\cdot\mathbf s=(2)(1)+(0)(1)+(0)(0)+(1)(0)=2 $$ $$ \|\mathbf f\|=\sqrt{2^2+0^2+0^2+1^2}=\sqrt5,\quad\|\mathbf s\|=\sqrt2 $$ $$ \cos(\mathbf f,\mathbf s)=\frac2{\sqrt{10}}\approx0.632456 $$ ### Step 4. Compute farmer–sun $$ \mathbf f\cdot\mathbf u=(2)(1)+(0)(0)+(0)(1)+(1)(0)=2 $$ $$ \|\mathbf u\|=\sqrt2,\quad\cos(\mathbf f,\mathbf u)=\frac2{\sqrt{10}}\approx0.632456 $$ ### Step 5. Compute farmer–plant $$ \mathbf f\cdot\mathbf p=(2)(2)+(0)(0)+(0)(1)+(1)(1)=5 $$ $$ \|\mathbf p\|=\sqrt{2^2+0^2+1^2+1^2}=\sqrt6 $$ $$ \cos(\mathbf f,\mathbf p)=\frac5{\sqrt{30}}\approx0.912871 $$ ### Step 6. Plot the requested two coordinates Interpret the axis names planted and harvested as their normalized context features plant and harvest. Farmer=(2,1), soil=(1,0), sun=(1,0), plant=(2,1). Farmer and plant overlap; soil and sun overlap. 001122farmer, plantsoil, sunplant (planted)harvest (harvested) The plot uses only the requested two coordinates; cosine uses all context columns. ### Final answer Using the stated noun-target convention: **farmer–soil = 0.632456, farmer–sun = 0.632456, farmer–plant = 0.912871.** **Yaad rakho:** Plant noun aur plant verb ko same maana ya separate? Pehle convention likho, phir count karo. ### Two details that prevent an incorrect answer **Arithmetic:** 5/√30 = 0.912871; the source key’s printed approximation 0.9444 does not equal its own fraction. Keep the exact fraction until the last step. **Normalization:** If the question instead requires all noun and verb forms of plant to be merged as targets, sentence 1 also contains the plant target. Its row becomes [3,0,1,1], and farmer–plant becomes 7/√55 ≈ 0.943880. The paper leaves this choice unstated; the main worked solution explicitly uses the noun-target interpretation that reproduces its supplied matrix. --- ## TEE · Q6A · Pronoun approaches and lexical relations Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q6a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q6a TEE · actual previous paper · 100 marks · 180 minutes Question. i. Briefly describe any two approaches to pronoun resolution and mention one advantage of each. ii. Describe the difference between Homonymy and Synonymy with suitable examples. Discuss how these two relations differently affect natural language processing applications? [10 marks] ### Answer ### i. Two approaches to pronoun resolution | Approach | How it works | Advantage | | --- | --- | --- | | Rule-based / syntactic | Uses parse structure, agreement, recency and linguistic constraints. Hobbs’ algorithm searches parse trees for eligible antecedents. | Interpretable: a proposed antecedent can be traced to explicit rules. | | Statistical or neural | Learns candidate–pronoun compatibility or whole-document coreference patterns from labelled examples, using context and linguistic features or contextual embeddings. | Can learn interactions and contextual cues that are difficult to enumerate as hand-written rules. | For “Asha dropped the glass because it was slippery,” a system should evaluate candidate referents and context rather than merely choose the most recent noun. Rules can constrain number and syntax; learned models can use compatibility patterns, though neither approach is guaranteed to resolve every ambiguity. ### ii. Homonymy and synonymy | Relation | Definition | Example | | --- | --- | --- | | Homonymy | Same spelling or sound, unrelated meanings | bank as financial institution versus bank as river edge | | Synonymy | Different words with the same or closely related meanings in a context | car and automobile; big and large | Homonymy creates an ambiguity problem: a search for bank may retrieve finance or rivers. WSD uses context to choose the intended sense. Synonymy creates a vocabulary-matching problem: a document saying automobile may be relevant to a query saying car. Query expansion or semantic representations can improve recall by connecting related terms. The remedies differ. Do not merge unrelated homonym senses simply because their spelling matches. Do not insist on exact surface-form matching when different words express the same concept. Synonyms are also context-sensitive: “big mistake” and “large mistake” are not always equally natural. **Yaad rakho:** Homonym: same word-form, different meaning. Synonym: different words, similar meaning. --- ## TEE · Q6B · Hobbs: Ramesh and Shyam Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q6b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q6b TEE · actual previous paper · 100 marks · 180 minutes Question. i. Explain Hobbs’ Algorithm for pronoun resolution. State its main steps and limitations. ii. Apply Hobbs’ Algorithm to the sentence: “When Ramesh gave Shyam a book, he smiled.” Identify the antecedent of “he” by showing traversal steps. [10 marks] ### Answer **Hobbs’ algorithm** is a syntax-based pronoun-resolution procedure that searches constituency parse trees in a specified order. It proposes noun phrases as antecedents and filters incompatible candidates, for example by number and gender agreement. **Read the tree labels first:** NP means noun phrase; S means sentence/clause; VP means verb phrase. X is the current NP or S search node, and p is the upward path used to reach it. Breadth-first means shallower nodes before deeper nodes; left-to-right fixes the order at the same depth. An N-bar (N′) node is the noun-centred part of an NP in a parse using that label. ### Complete ordered procedure 1. **Locate the pronoun.** Start at the NP immediately dominating the pronoun: the smallest NP containing it. 2. **Make the first climb.** Move upward to the first NP or S above the starting NP. Call it X and record the path p. 3. **Search initially to the left.** Traverse branches below X to the left of p, breadth-first and left-to-right. Propose an NP only if another NP or S lies between that candidate and X. Check agreement and other applicable constraints for each proposed candidate; accept the first eligible compatible one. 4. **Check the highest sentence.** If X is the highest S and no candidate has been accepted, search previous sentence trees, most recent first. In each, traverse breadth-first and left-to-right and propose NPs. Accept the first eligible compatible candidate; if the available context contains none, report no resolved antecedent. 5. **Otherwise climb again.** Move from X to the next NP or S above it. Make that node the new X and record the new upward path p. 6. **Test a higher NP.** If the new X is an NP, propose X only when p did not pass through its nominal N-bar child. Apply the same compatibility checks. Use the labels in the supplied parse; do not invent a missing N-bar branch. 7. **Search left again.** Traverse the new X’s branches to the left of p, breadth-first and left-to-right, proposing NPs. The extra intervening-NP/S requirement in Step 3 belongs to the initial search. 8. **Search permitted right branches.** If X is an S, traverse branches to the right of p, breadth-first and left-to-right, but do not descend below any NP or S encountered. Propose the encountered eligible NPs and test their compatibility. 9. **Repeat or finish.** Return to Step 4 until a compatible candidate is accepted or the available sentence trees are exhausted. Record the candidate order and why each candidate was accepted or rejected. **Why the initial restriction matters:** in “John saw him”, the subject NP John is directly under the clause S, with no additional NP or S between them. Step 3 therefore does not propose John as the ordinary local antecedent of him. A reflexive such as himself is a different grammatical case. The worked question below shows the specific parse and traversal; selecting an antecedent is not simply choosing the nearest name. ### Given The pronoun he is the subject of the main clause. The preceding subordinate clause is “When Ramesh gave Shyam a book”. Assume the temporal clause attaches to the whole main sentence. ### To find The first suitable antecedent in the left-to-right breadth-first search of the preceding clause. ``` S (main) ← first X ├── SBAR: When │ └── S (embedded) ← intervening S │ ├── NP: Ramesh ← first candidate │ └── VP │ ├── V: gave │ ├── NP: Shyam │ └── NP: a book ├── NP: he ← start └── VP: smiled ``` ### Step 1. Locate the pronoun NP Start at NP “he” and move upward to its S ancestor. Record the path from he. ### Step 2. Search branches before the pronoun The initial when-clause lies to the left of the path. Apply Step 3: its embedded S lies between each of its noun phrases and the main S, so these NPs satisfy the initial intervening-NP/S requirement. Traverse left-to-right and breadth-first. ### Step 3. Propose noun phrases The subject NP Ramesh is encountered before the deeper object NP Shyam and object NP a book. Ramesh agrees with the assumed singular-male pronoun. ### Step 4. Accept the first compatible antecedent Select Ramesh. Shyam is also a semantically possible referent in ordinary discourse, but the algorithm’s stated syntactic preference selects Ramesh first. ### Limitations - Requires a suitable and accurate constituency parse. - Agreement information may be unknown or insufficient when several people match. - Ignores much of the event semantics, discourse focus and world knowledge. - An ambiguous parse or different attachment can change candidate order. **Final answer:** **Ramesh**, under the shown parse and ordinary agreement assumptions. A syntactic preference is not a guarantee about the author’s intended meaning. **Yaad rakho:** Left branches ko breadth-first dekho; pehla eligible matching NP choose karo. --- ## TEE · Q7A · Information retrieval Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q7a/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q7a TEE · actual previous paper · 100 marks · 180 minutes Question. i. Explain the concept of Information Retrieval, highlighting its role in Recommender Systems and Search Engines. ii. Discuss the various steps involved in the Information Retrieval process. [10 marks] ### Answer ### i. Concept and roles **Information Retrieval (IR)** finds and ranks documents or other information objects relevant to a user’s information need. A query is the user’s expression of that need. IR typically returns a ranked set of items; information extraction instead identifies structured facts or fields inside those items. | Application | Role of IR | Example | | --- | --- | --- | | Search engine | Matches a submitted query against a large indexed collection and ranks candidate pages | A search for “symptoms of dehydration” returns relevant health pages | | Recommender system | Retrieves candidate items using a user profile, recent activity or a content representation; a later model may personalize ranking | Retrieve articles similar to topics the user reads | Retrieval is a component of many recommenders; recommendation also involves preferences, feedback, novelty and business constraints. A recommender need not wait for an explicit textual query. ### ii. Retrieval process Collect → Preprocess → Index → Process query → Retrieve candidates → Rank → Evaluate 1. **Collect and define documents.** Crawl or ingest documents, record metadata and remove unwanted duplicates. 2. **Preprocess.** Tokenize and normalize using rules suitable for both documents and queries. Preserve task-relevant information. 3. **Build representations and an index.** An inverted index maps each term to the documents containing it; a dense retrieval system stores vector representations. 4. **Process the query.** Apply compatible tokenization, spelling handling or expansion, then obtain query features. 5. **Retrieve candidates.** Use the index to identify potentially relevant documents efficiently. 6. **Score and rank.** A lexical method may use TF–IDF/BM25-style evidence; a semantic method compares representations. Rank scores estimate relevance rather than proving correctness. 7. **Present results.** Return titles, snippets and links, possibly with filtering or diversification. 8. **Evaluate and update.** Use relevance judgments and measures such as precision, recall or ranking metrics. Refresh the index as documents change. Example: for “solar energy storage”, an inverted index retrieves documents containing those terms; a ranking function rewards useful matches. A semantic system may also retrieve relevant material using related terms such as batteries. **Yaad rakho:** IR mein relevant document dhoondho aur rank karo; extraction mein document ke andar facts nikalo. --- ## TEE · Q7B · Gutenberg corpus and normalization Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/mocks/timed-reexam-2026--reexam-2026-q7b/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/mocks/timed-reexam-2026/reexam-2026-q7b TEE · actual previous paper · 100 marks · 180 minutes Question. i. Discuss how the Gutenberg Corpus can be utilized for various text analysis tasks in Natural Language Processing? ii. Explain the importance of text normalization in Natural Language Processing (NLP). Illustrate your answer with a suitable example of an application where text normalization plays a crucial role. [10 marks] ### Answer ### i. Using the Gutenberg corpus The Gutenberg corpus provides literary texts, commonly used through tools such as NLTK’s Gutenberg collection. It supplies naturally occurring sentences and longer documents for exploring language patterns. The available subset and licensing should be checked before reuse in a new distribution. | Task | Use of the corpus | | --- | --- | | Frequency and vocabulary study | Count tokens and word types, inspect common words and compare lexical diversity across texts | | Concordances and collocations | View a word in context and discover words that occur near each other | | N-gram language modeling | Estimate unigram or bigram counts and compare sequence probabilities | | Stylistic or authorship analysis | Compare sentence lengths, vocabulary and function-word patterns across authors | | Text preprocessing experiments | Study the effect of tokenization, case normalization and lemmatization on counts | A sound workflow selects files, inspects encoding and boundaries, tokenizes consistently, computes the relevant statistics and interprets them against corpus composition. Literary text is not automatically representative of chat, medical records or present-day search queries; domain mismatch matters. ### ii. Text normalization **Normalization** converts selected surface variants into a consistent form so that equivalent expressions do not unnecessarily produce different features. Possible steps include case folding, whitespace cleanup, Unicode normalization, contraction expansion and lemmatization. Choose these according to the task. ### Example: product search 1. Raw query: “ RUNNING shoes ”. Product description: “Comfortable run shoes”. 2. Trim extra whitespace and apply consistent case: “running shoes”. 3. If the application chooses lemmatization, running → run. The query and the indexed description now share the intended run and shoes features. 4. Apply the same normalization to stored documents and future queries, otherwise matching remains inconsistent. Normalization reduces avoidable vocabulary variation and can improve recall. It must preserve distinctions important to the task: lowercasing US to us may lose meaning, and removing “not” can reverse sentiment. Aggressive stemming or synonym replacement can create incorrect matches. **Yaad rakho:** Same meaning ke surface variants ko consistent banao; useful meaning delete mat karo. --- ## TEE · All seven units: final revision Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/revision/tee-revision--all-units/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/revision/tee-revision/all-units TEE · revision, not a replacement for full lessons TEE is 100 marks in 3 hours, scaled to 50 in the course total. Q1 is compulsory; choose any four of Q2–Q7. Each full question carries 20 marks. The two supplied 2025–26 papers follow this rule. Do not use the short M2 route as complete TEE preparation. | Unit | Must be able to explain | Must be able to work | | --- | --- | --- | | 1 | NLP levels, building blocks, NLU/NLG, approaches, pipeline and challenges | Classify ambiguity and identify linguistic units in examples. | | 2 | Tokenization, case, stopwords, stemming/lemmatization, encoding, BoW/TF-IDF | Construct vectors; compute TF-IDF, cosine/Jaccard/Euclidean/Manhattan/Minkowski. | | 3 | Corpora, annotation, dictionaries, regex, N-gram models and smoothing | List n-grams; count probabilities; smooth; compute sentence probability/perplexity. | | 4 | POS approaches, HMM, chunking/chinking, CFG parsing, NER | Build emission/transition matrices; score paths; draw a tree; trace parser operations. | | 5 | Lexical relations, WordNet, WSD, distributional/vector semantics, Word2Vec, GloVe | Lesk and NB; co-occurrence vectors; similarity; CBOW/Skip-gram examples. | | 6 | Pragmatics/discourse, reference, coreference, pronouns, Hobbs and other approaches | Identify chains; show a tree traversal and state parse assumptions. | | 7 | Classification, sentiment, spelling, translation, QA, IE, IR | Design an application pipeline; calculate relevant model/evaluation examples. | ### TEE revision sequence First revise the formula sheet and one full example per numerical family. Next read each unit’s definitions and comparisons. Then work one complete previous paper using its original choice rule. Finally revisit only the steps you got wrong, and read the other paper’s new question forms. Keep examples and diagrams beside definitions when revising; a term you cannot illustrate is not yet ready to explain. For a 10-mark numerical, show the count table or matrix and the intermediate probabilities. For a 10-mark theory answer, organise definition, mechanism/steps, diagram/example, advantages and limitations around the actual question. Do not add unrelated paragraphs just to fill space. --- ## TEE · Numerical answer framework Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/revision/tee-revision--numerical-method/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/revision/tee-revision/numerical-method TEE · revision, not a replacement for full lessons | Question clue | Use this method | First thing to write | | --- | --- | --- | | “next word”, “sentence probability” | N-gram model | List the needed word transitions. | | “unseen pair”, “Laplace” | Add-one smoothing | State V and count each history. | | “perplexity” | Reciprocal Nth root | State P(test sequence) and N. | | “transition/emission matrix” | HMM count estimation | Separate tag-to-tag counts from tag-to-word counts. | | “given tag sequence” | HMM joint path | Initial × emission × transition × emission … | | “best tag sequence” | Viterbi or enumerate tiny paths | Initialise one score per tag; retain predecessor maxima. | | “gloss overlap” | Simplified Lesk | Preprocess context and glosses consistently. | | “classify word sense with counts” | Naive Bayes WSD | Priors from examples, likelihoods from tokens. | | “parse tree” | Given CFG + chosen parser | Show grammar expansions or stack reductions. | ### Errors that change the answer 1. Dividing a conditional count by the whole corpus instead of its history count. 2. Adding 1 to the numerator without adding |V| to the denominator. 3. Forgetting the end transition. 4. Counting repeated words as new vocabulary types. 5. Using P(tag|word) in place of P(word|tag). 6. Calling an unnormalised score a posterior. 7. Silently mixing case, lemmatisation or boundary conventions. ### Short theory answer format For 2 marks: give the exact definition and one specific example or distinction. For 3 marks: add the mechanism or two ordered steps; include a small diagram/table where it carries information. The actual allocation is examiner-dependent, so these are writing guides, not an official marking scheme. --- ## TEE · Latest and older papers: exact question map Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/about/pyq-analysis--all-papers/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/about/pyq-analysis/all-papers ### How to use this question map Start with the newest final and re-exam, then use older papers to practise variations. The table links each original lettered part to its complete worked answer. **Repeated questions are history, not a prediction.** A syllabus topic stays required even when no recovered paper asks it. **Marks:** 100 marks are attempted: compulsory Q1 (20), then any four other 20-mark questions. Most recovered papers offer140 marks; NLP December2023 offers120 because the complete original ends at Q6. A mixed question retains one mark allocation; its units are not counted as separate questions. Blank individual allocations remain unknown. ### 2026-02-09 · re-exam · R26 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1 · NLP pipeline | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1a/) | | **Q1b [5]** | Unit 1, 4 · Content and function words | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1b/) | | **Q1c [5]** | Unit 5 · Word2Vec and GloVe | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1c/) | | **Q1d [5]** | Unit 6 · Cataphora, anaphora and coreference | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q1d/) | | **Q2a [10]** | Unit 1 · Five NLP tasks; syntax and semantics | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q2a/) | | **Q2b [10]** | Unit 3 · Add-one bigram sentence probability | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q2b/) | | **Q3a [10]** | Unit 4 · Top-down CFG parsing | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q3a/) | | **Q3b [10]** | Unit 5 · Lesk with supplied glosses | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q3b/) | | **Q4a [10]** | Unit 2 · Bag of words and three cosine similarities | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q4a/) | | **Q4b [10]** | Unit 4 · HMM estimation and given tag-path joint probability | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q4b/) | | **Q5a [10]** | Unit 2 · Explain and illustrate TF-IDF | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q5a/) | | **Q5b [10]** | Unit 5 · Co-occurrence, cosine and normalized plot | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q5b/) · Target/context normalization and sentence-presence convention stated; alternative target lemmatization has a separately calculated result. | | **Q6a [10]** | Unit 5, 6 · Pronoun-resolution methods; homonymy and synonymy | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q6a/) | | **Q6b [10]** | Unit 6 · Hobbs antecedent traversal | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q6b/) · Hobbs requires a parse and compatibility assumptions. The assumed parse and traversal order are disclosed. | | **Q7a [10]** | Unit 7 · Information-retrieval stages | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q7a/) | | **Q7b [10]** | Unit 2, 3 · Gutenberg and text normalization | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/reexam-2026--reexam-2026-q7b/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1: 2 parts, 15 known marks; Unit 1, 4: 1 parts, 5 known marks; Unit 5: 3 parts, 25 known marks; Unit 6: 2 parts, 15 known marks; Unit 3: 1 parts, 10 known marks; Unit 4: 2 parts, 20 known marks; Unit 2: 2 parts, 20 known marks; Unit 5, 6: 1 parts, 10 known marks; Unit 7: 1 parts, 10 known marks; Unit 2, 3: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2025-12-02 · final · F25 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1 · NLU versus NLG | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1a/) | | **Q1b [5]** | Unit 4 · Chunking, chinking and POS | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1b/) | | **Q1c [5]** | Unit 5 · Hyponymy/hypernymy and WSD | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1c/) | | **Q1d [5]** | Unit 6 · Pragmatic and discourse analysis | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q1d/) | | **Q2a [10]** | Unit 1, 4 · Morpheme/lexeme and NER applications | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q2a/) | | **Q2b [10]** | Unit 3 · Add-one bigram sentence probability | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q2b/) | | **Q3a [10]** | Unit 1, 4 · Ambiguities and bottom-up CFG parse | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q3a/) | | **Q3b [10]** | Unit 5 · Co-occurrence, cosine and vector plot | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q3b/) | | **Q4a [10]** | Unit 2 · Three preprocessing techniques | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q4a/) | | **Q4b [10]** | Unit 4 · HMM estimation and given tag-path joint probability | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q4b/) | | **Q5a [10]** | Unit 2 · TF-IDF matrix | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q5a/) | | **Q5b [10]** | Unit 5 · Naive Bayes word-sense disambiguation | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q5b/) · Fixed vocabulary/OOV treatment is stated; source-key OOV smoothing differs. Both conventions and their consequence are explained. | | **Q6a [10]** | Unit 5, 6 · Coreference/pronouns; CBOW and Skip-gram | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q6a/) | | **Q6b [10]** | Unit 6 · Hobbs antecedent traversal | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q6b/) · Hobbs requires a parse and compatibility assumptions. The assumed parse and traversal order are disclosed. | | **Q7a [10]** | Unit 7 · Document-classification pipeline | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q7a/) | | **Q7b [10]** | Unit 2, 3 · Corpus structures and Jaccard example | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/final-2025--final-2025-q7b/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1: 1 parts, 5 known marks; Unit 4: 2 parts, 15 known marks; Unit 5: 3 parts, 25 known marks; Unit 6: 2 parts, 15 known marks; Unit 1, 4: 2 parts, 20 known marks; Unit 3: 1 parts, 10 known marks; Unit 2: 2 parts, 20 known marks; Unit 5, 6: 1 parts, 10 known marks; Unit 7: 1 parts, 10 known marks; Unit 2, 3: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2025-02-21 · re-exam · R25 140 marks offered; 100 attempted. Q1 total20 is printed, but the four individual part allocations are blank. Do not infer5marks as if printed. Sixth PDF page is blank. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [not printed individually]** | Unit 2 · BoW and one-hot encoding | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1a/) | | **Q1b [not printed individually]** | Unit 5 · Lesk WSD | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1b/) | | **Q1c [not printed individually]** | Unit 4 · Syntax parsing approaches | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1c/) | | **Q1d [not printed individually]** | Unit 1, 7 · NLU versus NLG and applications | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q1d/) | | **Q2a [10]** | Unit 2 · TF–IDF cosine after a fixed stoplist | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q2a/) | | **Q2b [10]** | Unit 7 · Weather Naive Bayes | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q2b/) | | **Q3a [5]** | Unit 6 · Coreference and anaphora | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q3a/) | | **Q3b [5]** | Unit 6 · Cohesion, coherence and discourse markers | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q3b/) | | **Q3c [10]** | Unit 7 · Question answering system | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q3c/) | | **Q4a [5]** | Unit 1 · NLP pipeline | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q4a/) | | **Q4b [5]** | Unit 1 · Morphology and syntax | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q4b/) | | **Q4c [10]** | Unit 3 · Most probable bigram continuations | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q4c/) | | **Q5a [6]** | Unit 2 · Preprocess the NLP passage | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q5a/) | | **Q5b [4]** | Unit 2 · Stemming and lemmatization tools | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q5b/) | | **Q5c [10]** | Unit 6 · Hobbs on John and Mary | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q5c/) | | **Q6a [10]** | Unit 3 · Corpora and acquisition | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q6a/) | | **Q6b [10]** | Unit 4 · HMM components and estimates | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q6b/) | | **Q7a [5]** | Unit 4 · Word classes, chunking and chinking | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q7a/) | | **Q7b [5]** | Unit 4 · Named entity recognition | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q7b/) | | **Q7c [10]** | Unit 5 · Co-occurrence with inconsistent requested comparisons | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r25--q7c/) · Source defect: follow-up names belong to another corpus. Exact supported matrix is complete; unsupported parts are explicitly unresolved from the original data. | **Original offered marks by mapped topic, grouped without double counting:** Unit 2: 4 parts, 20 known marks + 1 parts with no individual mark printed; Unit 5: 2 parts, 10 known marks + 1 parts with no individual mark printed; Unit 4: 4 parts, 20 known marks + 1 parts with no individual mark printed; Unit 1, 7: 1 parts, 0 known marks + 1 parts with no individual mark printed; Unit 7: 2 parts, 20 known marks; Unit 6: 3 parts, 20 known marks; Unit 1: 2 parts, 10 known marks; Unit 3: 2 parts, 20 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2024-12-03 · final · F24 140 marks offered; 100 attempted. Four Q1 part allocations are blank; their group totals20. Other printed lettered allocations determine each 20-mark main question. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [not printed individually]** | Unit 2, 7 · TF–IDF in retrieval | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1a/) | | **Q1b [not printed individually]** | Unit 4 · NER and approaches | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1b/) | | **Q1c [not printed individually]** | Unit 5 · Skip-gram and CBOW | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1c/) | | **Q1d [not printed individually]** | Unit 6 · Pragmatics and discourse | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q1d/) | | **Q2a [6]** | Unit 2, 4 · Six preprocessing operations | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q2a/) | | **Q2b [4]** | Unit 2, 5 · Lexical similarity and cosine | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q2b/) | | **Q2c [10]** | Unit 7 · Animal categorical Naive Bayes | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q2c/) | | **Q3a [10]** | Unit 6 · Hobbs and ambiguous professor–student reference | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q3a/) · Pronoun referent is ambiguous without extra gender/discourse information; structural algorithm preference is separated from semantic interpretation. | | **Q3b [10]** | Unit 7 · Classification and sentiment in industry | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q3b/) | | **Q4a [10]** | Unit 4 · Top-down syntax tree | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q4a/) | | **Q4b [10]** | Unit 3 · Data acquisition and four augmentation methods | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q4b/) | | **Q5a [10]** | Unit 2 · Feature engineering, BoW and Euclidean | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q5a/) | | **Q5b [5]** | Unit 5 · WSD approaches and uses | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q5b/) | | **Q5c [5]** | Unit 6 · Discourse analysis of a conversation | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q5c/) | | **Q6a [10]** | Unit 3 · Henry bigram continuations | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q6a/) · Case is retained as printed: Do and do are distinct. The corpus is not silently grammatically corrected. | | **Q6b [5]** | Unit 1 · Five linguistic levels | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q6b/) | | **Q6c [5]** | Unit 1 · NLU and NLG working together | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q6c/) | | **Q7a [10]** | Unit 4 · POS, chunks and boundary-aware HMM | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q7a/) · Printed tags are used as training data; START and END are explicitly required and included. | | **Q7b [10]** | Unit 5 · Co-occurrence, dot products and plot | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f24--q7b/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 2, 7: 1 parts, 0 known marks + 1 parts with no individual mark printed; Unit 4: 3 parts, 20 known marks + 1 parts with no individual mark printed; Unit 5: 3 parts, 15 known marks + 1 parts with no individual mark printed; Unit 6: 3 parts, 15 known marks + 1 parts with no individual mark printed; Unit 2, 4: 1 parts, 6 known marks; Unit 2, 5: 1 parts, 4 known marks; Unit 7: 2 parts, 20 known marks; Unit 3: 2 parts, 20 known marks; Unit 2: 1 parts, 10 known marks; Unit 1: 2 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2024-02-12 · re-exam · R24 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 6 · Anaphora versus cataphora | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1a/) | | **Q1b [5]** | Unit 3 · Five corpora | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1b/) | | **Q1c [5]** | Unit 1 · Linguistic foundations | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1c/) | | **Q1d [5]** | Unit 3 · Five regex operators | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q1d/) | | **Q2a [10]** | Unit 3 · Wordlist puzzle and lexical resource | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q2a/) · The dictionary is unspecified; an exact number of valid words would be fabricated. | | **Q2b [10]** | Unit 1, 7 · NLP applications and limitations | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q2b/) | | **Q3a [10]** | Unit 4 · Top-down restaurant parse | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q3a/) · Original CFG is incomplete for the given bare-plural object. Exact failure and explicit repaired derivation are both shown. | | **Q3b [10]** | Unit 2 · Preprocessing pipeline | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q3b/) | | **Q4a [10]** | Unit 5 · Embeddings and Skip-gram | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q4a/) | | **Q4b [10]** | Unit 2, 4 · POS, NER, chunking and TTR | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q4b/) | | **Q5a [10]** | Unit 5 · Lexical and vector semantics | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q5a/) | | **Q5b [10]** | Unit 7 · Classification label structures | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q5b/) | | **Q6a [10]** | Unit 2, 7 · TF–IDF in document representation | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q6a/) · The prompt requests an example but supplies no corpus. Numbers in the explanation are authored illustrations, not original numerical givens. | | **Q6b [10]** | Unit 4 · HMM tables and path | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q6b/) | | **Q7a [10]** | Unit 7 · Healthcare and finance applications | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q7a/) | | **Q7b [10]** | Unit 7 · Naive Bayes sentiment | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r24--q7b/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 6: 1 parts, 5 known marks; Unit 3: 3 parts, 20 known marks; Unit 1: 1 parts, 5 known marks; Unit 1, 7: 1 parts, 10 known marks; Unit 4: 2 parts, 20 known marks; Unit 2: 1 parts, 10 known marks; Unit 5: 2 parts, 20 known marks; Unit 2, 4: 1 parts, 10 known marks; Unit 7: 3 parts, 30 known marks; Unit 2, 7: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2023-12-01 · final · F23 120 marks offered; 100 attempted. Original has six main questions on three pages: Q1 compulsory and any four of Q2–Q6. Offered total 120, attempted 100. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 5, 6 · WSD and pronoun resolution | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1a/) | | **Q1b [5]** | Unit 5 · WordNet and its hierarchy | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1b/) | | **Q1c [5]** | Unit 2, 7 · Sentiment feature extraction | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1c/) | | **Q1d [5]** | Unit 7 · Binary, multiclass and multilabel | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q1d/) | | **Q2a [10]** | Unit 3 · Text corpus structure and applications | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q2a/) | | **Q2b [10]** | Unit 5 · Lesk versus Walker and bank disambiguation | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q2b/) · The original gives a sentence but no thesaurus/category mapping, so a unique numerical score cannot be recovered. | | **Q3a [10]** | Unit 6 · Hobbs algorithm and limitations | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q3a/) | | **Q3b [10]** | Unit 2 · Stemming and lemmatization | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q3b/) | | **Q4a [10]** | Unit 5 · Lexical and vector semantics | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q4a/) | | **Q4b [10]** | Unit 2 · Three-document TF–IDF | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q4b/) | | **Q5a [10]** | Unit 5 · Word embeddings and CBOW | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q5a/) | | **Q5b [10]** | Unit 4 · HMM tables and given path | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q5b/) | | **Q6a [10]** | Unit 7 · Text classification pipeline | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q6a/) | | **Q6b [10]** | Unit 2, 7 · VSM, TF and IDF in retrieval | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f23--q6b/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 5, 6: 1 parts, 5 known marks; Unit 5: 4 parts, 35 known marks; Unit 2, 7: 2 parts, 15 known marks; Unit 7: 2 parts, 15 known marks; Unit 3: 1 parts, 10 known marks; Unit 6: 1 parts, 10 known marks; Unit 2: 2 parts, 20 known marks; Unit 4: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### 2023-01-30 · re-exam · R23 140 marks offered; 100 attempted. Q1 header is unclear/conflicting [14/20], but all four parts print 5 marks each; sum=20. Main paper is 100 attempted marks. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 2 · Normalization changes vocabulary | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1a/) | | **Q1b [5]** | Unit 3 · Good corpus characteristics | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1b/) | | **Q1c [5]** | Unit 5 · Word2Vec versus Doc2Vec | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1c/) · legacy named-method extension; not separately named in current syllabus. Optional historical extension; not a separately prescribed current-syllabus method. | | **Q1d [5]** | Unit 1, 4 · Ambiguity in NLP | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q1d/) | | **Q2a [10]** | Unit 2 · Normalize the Tanisha–Simran passage | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q2a/) · Stoplist and stemming algorithm are choices, not supplied numerical givens; both are stated. | | **Q2b [10]** | Unit 2, 5 · Text vectorization alternatives | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q2b/) | | **Q3a [10]** | Unit 1, 5, 6 · Semantics, pragmatics and two ambiguities | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q3a/) | | **Q3b [10]** | Unit 7 · Naive Bayes text classifier | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q3b/) · Chinese corpus is an authored/reused teaching illustration here; it is actual given data in F22 Q5b, not this prompt. | | **Q4a [10]** | Unit 2 · Jaccard, cosine and Euclidean without preprocessing | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q4a/) | | **Q4b [10]** | Unit 5 · CBOW and Skip-gram | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q4b/) | | **Q5a [10]** | Unit 3 · Password regex | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q5a/) | | **Q5b [10]** | Unit 7 · Two NLP applications | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q5b/) | | **Q6a [10]** | Unit 6 · Mention-pair model | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q6a/) | | **Q6b [10]** | Unit 7 · Classification evaluation including ROC/AUC | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q6b/) | | **Q7a [10]** | Unit 4 · Organization and facility mentions | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q7a/) · Boundary and generic-mention decisions depend on the provided ACE annotation convention; the answer distinguishes them from a named-only interpretation. | | **Q7b [10]** | Unit 4 · Tagging and two complete parse trees | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-r23--q7b/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 2: 3 parts, 25 known marks; Unit 3: 2 parts, 15 known marks; Unit 5: 2 parts, 15 known marks; Unit 1, 4: 1 parts, 5 known marks; Unit 2, 5: 1 parts, 10 known marks; Unit 1, 5, 6: 1 parts, 10 known marks; Unit 7: 3 parts, 30 known marks; Unit 6: 1 parts, 10 known marks; Unit 4: 2 parts, 20 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. **Current-syllabus qualification:** Q1c: historical named-method extension, excluded from current-only priority totals. The original whole-part marks remain visible so the paper is faithfully represented. ### 2022-11-22 · final · F22 140 marks offered; 100 attempted. | Question [marks] | Current unit and topic | Question form | Answer / scope | | --- | --- | --- | --- | | **Q1a [5]** | Unit 1 · Need and difficulties of NLP | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1a/) | | **Q1b [5]** | Unit 1 · Levels of language analysis | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1b/) | | **Q1c [5]** | Unit 7 · Medical chatbot | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1c/) | | **Q1d [5]** | Unit 7 · Sentiment with logistic regression | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q1d/) | | **Q2a [10]** | Unit 5 · CBOW and Skip-gram architecture | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q2a/) | | **Q2b [10]** | Unit 3 · Seven regex constructions | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q2b/) | | **Q3a [10]** | Unit 2 · Text preprocessing | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q3a/) | | **Q3b [10]** | Unit 4 · Penn Treebank tagging | case | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q3b/) · Context-sensitive tags and possessive tokenization are explicitly stated; reasonable corpus guidelines can differ on noun adjuncts. | | **Q4a [10]** | Unit 6 · Mention-pair coreference | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q4a/) | | **Q4b [10]** | Unit 7 · Text classification pipeline | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q4b/) | | **Q5a [10]** | Unit 3 · Corpora and structure | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q5a/) | | **Q5b [10]** | Unit 7 · Multinomial Naive Bayes | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q5b/) | | **Q6a [10]** | Unit 5 · GloVe | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q6a/) | | **Q6b [10]** | Unit 2 · BoW and TF–IDF movie corpus | numerical | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q6b/) | | **Q7a [10]** | Unit 4, 7 · Information extraction, chunking and NER | theory | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q7a/) | | **Q7b [10]** | Unit 5 · Distributional hypothesis and cosine | mixed | [Read question + answer](https://deep-learning-study-room.vercel.app/read/nlp/tee/papers/legacy-f22--q7b/) | **Original offered marks by mapped topic, grouped without double counting:** Unit 1: 2 parts, 10 known marks; Unit 7: 4 parts, 30 known marks; Unit 5: 3 parts, 30 known marks; Unit 3: 2 parts, 20 known marks; Unit 2: 2 parts, 20 known marks; Unit 4: 1 parts, 10 known marks; Unit 6: 1 parts, 10 known marks; Unit 4, 7: 1 parts, 10 known marks. Joint-unit buckets remain joint; no invented split is assigned. These original-paper totals include any legacy tasks explicitly marked above; they are not a current-syllabus marks blueprint. ### Priority without skipping the syllabus The latest papers show the form and depth of questions, while the official current syllabus sets the learning scope. Authored drills, explanatory numbers and optional extensions are not counted as past questions. Source assumptions and ambiguous data are stated in their answers. --- ## TEE · What this NLP guide covers Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/about/scope--sources/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/about/scope/sources NLP · M2 and TEE · edition 3 October 2026 **M2 portion confirmed by you:** Unit 3 from PDF slide 62 to 160; Unit 4 complete (128 slides); Unit 5 slides 1–57. Slide 62 of Unit 3 starts N-grams. Slide 57 of Unit 5 completes the Naive Bayes WSD example. Vector-space models start on slide 58 and belong to TEE here. **M2 pattern confirmed by you:** both 2-mark questions compulsory; any two of the three 3-mark questions. Total attempted 10. Course policy gives one hour. Three newly authored, fully solved M2 practice papers use that pattern. **TEE:** all seven units of the official AY 2026–27 syllabus. The new ZIP supplies faculty slide packs for Units 1–5 plus a regular-expressions handout. Units 6–7 are syllabus-based explanations supplemented with the standard references below; their faculty slide packs were not supplied. The official syllabus includes GloVe and spelling correction, and both are retained even though the policy’s lecture table abbreviates them. **Previous papers found:** final exam dated 2 December 2025 and re-exam dated 9 February 2026, with answer keys, inside Semester V.zip. Programme copies were deduplicated. Every supplied question subpart is mapped to the current syllabus; no question was excluded as out of syllabus. Only wholly M2-eligible subparts appear in the M2 paper tab; mixed questions remain in the complete TEE papers. The current syllabus lists course code 702AI0C037. The supplied policy and old papers use 702AI0C009; the policy also retains an older programme/semester header. Topic coverage follows the current official syllabus and your exact M2 cutoff, rather than treating the old header as a different course. ### Conventions that must stay visible Course bigram answer keys count both start and end symbols in vocabulary size. Worked PYQs state and follow that convention; other exercises state their own vocabulary explicitly. A start symbol is conditioning context, so alternative textbook conventions may exclude it from prediction vocabulary and produce a different smoothed number. HMM path scores and Naive Bayes class scores are labelled as unnormalised when applicable. Any normalisation, case-folding or lemmatisation assumption is stated before counting. ### Reading references - Your uploaded course policy, official Semester V syllabus, five NLP slide packs and regular-expressions handout. - Your uploaded final and re-exam questions and answer keys (AY 2025–26). - [Jurafsky & Martin · Speech and Language Processing](https://web.stanford.edu/~jurafsky/slp3/) — terminology and standard methods. - [NLTK Book](https://www.nltk.org/book/) — corpus, tagging, parsing and information-extraction concepts. - [Tetreault · A Corpus-Based Evaluation of Centering and Pronoun Resolution](https://aclanthology.org/J01-4003.pdf) — precise Hobbs traversal. Examples are teaching problems unless explicitly identified by their paper and question number. Practice-paper marks indicate intended answer length, not an official forecast. Reading alone cannot guarantee a score; writing the solutions independently is what makes the procedures usable under exam time. ### TEE depth review · 10 October 2026 The core TEE route retains the faculty topics CYK/CKY, grammar conversion, interpolation and Stupid Backoff. Add-k and HMM forward/posterior calculations are available separately under Optional extensions; they are not part of the core reading or formula sequence. The two TEE timed-practice entries reproduce the supplied actual papers and are not additional unseen mock papers. Method references: [Jurafsky and Martin: Hidden Markov Models](https://web.stanford.edu/~jurafsky/slp3/A.pdf); [Jurafsky and Martin: Constituency Parsing](https://web.stanford.edu/~jurafsky/slp3/E.pdf). New numerical inputs and worked traces are teaching examples. --- ## TEE · Video companions in your NotebookLM Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/about/scope--videos/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/about/scope/videos NLP · M2 and TEE · edition 3 October 2026 Your M2 notebook contains the exact three slide ranges and checked numerical teaching examples. The requested video topics are N-grams and Laplace smoothing, HMM/Viterbi, and Naive Bayes WSD. Use the written solutions here for the complete calculations. [Open your NLP M2 notebook →](https://notebook.google.com/notebook/a270ef1e-c285-4d31-9db1-302090c5f8b1) The notebook is in your signed-in Google account. Video generation and availability are shown in its Studio panel. Generated narration is a supplementary explanation; the numerical results in this guide have been independently checked. ## Complete video catalog updated\_utc : 2026-10-10T11:17:01.698545+00:00 notebooks : m2 : tee : lesson\_map : m2 : foundations/language : - nlp-m2-01-foundations-counts-probability foundations/notation : - nlp-m2-01-foundations-counts-probability foundations/probability : - nlp-m2-01-foundations-counts-probability m2-language-models/words-counts-probability : - nlp-m2-01-foundations-counts-probability foundations/bayes : - nlp-m2-02-bayes-generative-discriminative m2-pos-tagging/generative-discriminative : - nlp-m2-02-bayes-generative-discriminative m2-language-models/n-grams-construction : - nlp-m2-03-ngrams-language-models m2-language-models/collocations-and-lm : - nlp-m2-03-ngrams-language-models m2-language-models/chain-rule-markov : - nlp-m2-03-ngrams-language-models m2-count-to-probability/unigram-estimation : - nlp-m2-03-ngrams-language-models m2-count-to-probability/bigram-count-table : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/trigram-estimation : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/class-count-table-questions : - nlp-m2-04-bigram-trigram-tables m2-evaluation-smoothing/laplace-basics : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/smoothed-sentence : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/adjusted-count-discount : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/evaluation-perplexity : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/log-probabilities : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/class-perplexity-characters : - nlp-m2-06-perplexity-logs-calculator foundations/calculator : - nlp-m2-06-perplexity-logs-calculator m2-pos-tagging/word-classes-pos : - nlp-m2-07-pos-tags-tagger-methods m2-pos-tagging/tagger-methods : - nlp-m2-07-pos-tags-tagger-methods m2-hmm/markov-chain : - nlp-m2-08-markov-hmm-components m2-hmm/hmm-components : - nlp-m2-08-markov-hmm-components m2-hmm/transition-emission-estimation : - nlp-m2-08-markov-hmm-components m2-hmm/hot-cold-paths : - nlp-m2-09-hmm-joint-paths m2-hmm/tagging-corpus-case : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/joint-vs-posterior : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/sequence-count : - nlp-m2-10-viterbi-decoding m2-hmm-decoding/viterbi-explained : - nlp-m2-10-viterbi-decoding m2-chunks-grammar/chunking-chinking : - nlp-m2-11-chunking-chinking m2-chunks-grammar/cfg-foundation : - nlp-m2-12-cfg-top-down-parsing m2-chunks-grammar/structural-ambiguity : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-theory : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-examples : - nlp-m2-12-cfg-top-down-parsing m2-parsing/shift-reduce-theory : - nlp-m2-13-bottom-up-shift-reduce m2-parsing/shift-reduce-examples : - nlp-m2-13-bottom-up-shift-reduce m2-ner/ner-definition : - nlp-m2-14-named-entity-recognition m2-ner/ner-approaches : - nlp-m2-14-named-entity-recognition m2-ner/ner-worked-examples : - nlp-m2-14-named-entity-recognition u5-meaning/u5-words : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-relations : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-wordnet : - nlp-m2-15-lexical-meaning-wordnet u5-wsd/u5-wsd-basics : - nlp-m2-16-lesk-wsd u5-wsd/u5-lesk : - nlp-m2-16-lesk-wsd u5-wsd/u5-walker : - nlp-m2-17-walker-graph-wsd u5-wsd/u5-graph : - nlp-m2-17-walker-graph-wsd u5-naive-bayes/u5-bayes-meaning : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-steps : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-mam : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-practice : - nlp-m2-18-naive-bayes-wsd tee : foundations/language : - nlp-m2-01-foundations-counts-probability foundations/notation : - nlp-m2-01-foundations-counts-probability foundations/probability : - nlp-m2-01-foundations-counts-probability m2-language-models/words-counts-probability : - nlp-m2-01-foundations-counts-probability foundations/bayes : - nlp-m2-02-bayes-generative-discriminative m2-pos-tagging/generative-discriminative : - nlp-m2-02-bayes-generative-discriminative m2-language-models/n-grams-construction : - nlp-m2-03-ngrams-language-models m2-language-models/collocations-and-lm : - nlp-m2-03-ngrams-language-models m2-language-models/chain-rule-markov : - nlp-m2-03-ngrams-language-models m2-count-to-probability/unigram-estimation : - nlp-m2-03-ngrams-language-models m2-count-to-probability/bigram-count-table : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/trigram-estimation : - nlp-m2-04-bigram-trigram-tables m2-count-to-probability/class-count-table-questions : - nlp-m2-04-bigram-trigram-tables m2-evaluation-smoothing/laplace-basics : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/smoothed-sentence : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/adjusted-count-discount : - nlp-m2-05-laplace-smoothing-discount m2-evaluation-smoothing/evaluation-perplexity : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/log-probabilities : - nlp-m2-06-perplexity-logs-calculator m2-evaluation-smoothing/class-perplexity-characters : - nlp-m2-06-perplexity-logs-calculator foundations/calculator : - nlp-m2-06-perplexity-logs-calculator m2-pos-tagging/word-classes-pos : - nlp-m2-07-pos-tags-tagger-methods m2-pos-tagging/tagger-methods : - nlp-m2-07-pos-tags-tagger-methods m2-hmm/markov-chain : - nlp-m2-08-markov-hmm-components m2-hmm/hmm-components : - nlp-m2-08-markov-hmm-components m2-hmm/transition-emission-estimation : - nlp-m2-08-markov-hmm-components m2-hmm/hot-cold-paths : - nlp-m2-09-hmm-joint-paths m2-hmm/tagging-corpus-case : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/joint-vs-posterior : - nlp-m2-09-hmm-joint-paths m2-hmm-decoding/sequence-count : - nlp-m2-10-viterbi-decoding m2-hmm-decoding/viterbi-explained : - nlp-m2-10-viterbi-decoding m2-chunks-grammar/chunking-chinking : - nlp-m2-11-chunking-chinking m2-chunks-grammar/cfg-foundation : - nlp-m2-12-cfg-top-down-parsing m2-chunks-grammar/structural-ambiguity : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-theory : - nlp-m2-12-cfg-top-down-parsing m2-parsing/top-down-examples : - nlp-m2-12-cfg-top-down-parsing m2-parsing/shift-reduce-theory : - nlp-m2-13-bottom-up-shift-reduce m2-parsing/shift-reduce-examples : - nlp-m2-13-bottom-up-shift-reduce m2-ner/ner-definition : - nlp-m2-14-named-entity-recognition m2-ner/ner-approaches : - nlp-m2-14-named-entity-recognition m2-ner/ner-worked-examples : - nlp-m2-14-named-entity-recognition u5-meaning/u5-words : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-relations : - nlp-m2-15-lexical-meaning-wordnet u5-meaning/u5-wordnet : - nlp-m2-15-lexical-meaning-wordnet u5-wsd/u5-wsd-basics : - nlp-m2-16-lesk-wsd u5-wsd/u5-lesk : - nlp-m2-16-lesk-wsd u5-wsd/u5-walker : - nlp-m2-17-walker-graph-wsd u5-wsd/u5-graph : - nlp-m2-17-walker-graph-wsd u5-naive-bayes/u5-bayes-meaning : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-steps : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-mam : - nlp-m2-18-naive-bayes-wsd u5-naive-bayes/u5-bayes-practice : - nlp-m2-18-naive-bayes-wsd u1-introduction/u1-nlp : - tee-01 u1-introduction/u1-language : - tee-01 u1-introduction/u1-approaches : - tee-01 u1-introduction/u1-nlu-nlg : - tee-02 u1-introduction/u1-pipeline : - tee-02 u1-introduction/u1-challenges : - tee-02 u2-preprocess/u2-tokenize : - tee-03 u2-preprocess/u2-normalization : - tee-03 u2-preprocess/u2-stem-lemma : - tee-03 u2-preprocess/u2-ttr : - tee-03 u2-representation/u2-vocab : - tee-04 u2-representation/u2-label-onehot : - tee-04 u2-representation/u2-bow : - tee-04 u2-representation/u2-tfidf : - tee-05 u2-similarity/u2-cosine : - tee-05 u2-representation/u2-slide-combined : - tee-05 u2-similarity/u2-jaccard : - tee-06 u2-similarity/u2-distances : - tee-06 u2-similarity/u2-choice : - tee-06 u3-corpora/u3-corpus : - tee-07 u3-corpora/u3-types : - tee-07 u3-corpora/u3-access : - tee-07 u3-corpora/u3-resources : - tee-07 u3-corpora/u3-acquisition : - tee-07 u3-corpora/u3-links : - tee-07 u3-regex/regex-language : - tee-08 u3-regex/regex-functions : - tee-08 u3-regex/regex-worked : - tee-08 u3-regex/regex-reference : - tee-08 u5-vectors/u5-distribution : - tee-09 u5-vectors/u5-representation : - tee-09 u5-vectors/u5-sentence : - tee-09 u5-vectors/u5-word2vec : - tee-10 u5-glove/u5-glove-theory : - tee-11 u5-glove/u5-glove-loss : - tee-11 u6/reference : - tee-12 u6/resolution : - tee-12 u6/hobbs : - tee-13 u7/classification : - tee-14 u7/sentiment : - tee-14 u7/evaluation : - tee-14 u7/spelling : - tee-15 u7/translation : - tee-16 u7/qa : - tee-17 u7/extraction : - tee-18 u7/retrieval : - tee-18 written\_only : m2 tee : u3-smoothing-extensions/add-k : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u3-smoothing-extensions/interpolation : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u3-smoothing-extensions/stupid-backoff : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u4-forward-cky/forward-probability : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u4-forward-cky/cnf : This worked lesson was added after the video plan. No matching video has been generated or reviewed. u4-forward-cky/cky : This worked lesson was added after the video plan. No matching video has been generated or reviewed. service\_limit : status : blocked reset\_local : 2026-10-03T22:06:00+05:30 second\_account\_reset\_local : 2026-10-03T22:51:00+05:30 remaining\_not\_submitted : 12 ### M2 01 — NLP basics, counts and conditional probability id : nlp-m2-01-foundations-counts-probability title : M2 01 — NLP basics, counts and conditional probability title\_confirmed : false topic\_title : NLP basics, counts and conditional probability mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Bayes Rule & Models id : nlp-m2-02-bayes-generative-discriminative title : Bayes Rule & Models title\_confirmed : true topic\_title : Bayes rule and generative versus discriminative models mode : m2 kind : focused status : available duration : 10:51 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 03: N-grams & Unigrams id : nlp-m2-03-ngrams-language-models title : NLP M2 03: N-grams & Unigrams title\_confirmed : true topic\_title : N-grams, chain rule and unigram models mode : m2 kind : focused status : available duration : 9:52 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Bigram & Trigram Probs id : nlp-m2-04-bigram-trigram-tables title : Bigram & Trigram Probs title\_confirmed : true topic\_title : Build bigram and trigram probabilities from a corpus mode : m2 kind : focused status : available duration : 10:05 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 05: Laplace Smoothing id : nlp-m2-05-laplace-smoothing-discount title : M2 05: Laplace Smoothing title\_confirmed : true topic\_title : Laplace smoothing, unseen sequences and adjusted counts mode : m2 kind : focused status : available duration : 11:11 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 06 — Perplexity, logarithms and calculator working id : nlp-m2-06-perplexity-logs-calculator title : M2 06 — Perplexity, logarithms and calculator working title\_confirmed : false topic\_title : Perplexity, logarithms and calculator working mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule id : nlp-m2-07-pos-tags-tagger-methods title : NLP M2 — POS tagging and Brill | Reviewed explanation and worked rule title\_confirmed : true topic\_title : POS tags, ambiguity and tagging methods mode : m2 kind : focused status : available duration : 9:06 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Tagger methods, Brill net gain and the full conditional rule; clarified tagging and imperative explanations. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 08: Markov Paths & HMMs id : nlp-m2-08-markov-hmm-components title : M2 08: Markov Paths & HMMs title\_confirmed : true topic\_title : Markov paths, HMM components and training counts mode : m2 kind : focused status : available duration : 9:45 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### HMM Path Probabilities id : nlp-m2-09-hmm-joint-paths title : HMM Path Probabilities title\_confirmed : true topic\_title : Complete HMM path scores and observation probabilities mode : m2 kind : focused status : available duration : 8:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 10 Viterbi Decoding id : nlp-m2-10-viterbi-decoding title : M2 10 Viterbi Decoding title\_confirmed : true topic\_title : Viterbi decoding with backpointers mode : m2 kind : focused status : available duration : 9:34 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Chunking and Chinking | Reviewed rules and worked examples id : nlp-m2-11-chunking-chinking title : NLP M2 — Chunking and Chinking | Reviewed rules and worked examples title\_confirmed : true topic\_title : Chunking and chinking from POS-tagged words mode : m2 kind : focused status : available duration : 8:22 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Correct POS labels including JJ = adjective, full NP rule application and chinking boundaries. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — CFG and top-down parsing | Reviewed derivations and trees id : nlp-m2-12-cfg-top-down-parsing title : NLP M2 — CFG and top-down parsing | Reviewed derivations and trees title\_confirmed : true topic\_title : CFG, ambiguity and top-down parsing mode : m2 kind : focused status : available duration : 8:54 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete grammars, derivations and valid parse trees; corrected the acceptance condition. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 13 — Shift-reduce parsing and backtracking id : nlp-m2-13-bottom-up-shift-reduce title : M2 13 — Shift-reduce parsing and backtracking title\_confirmed : false topic\_title : Shift-reduce parsing and backtracking mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 14 — Named entities, boundaries and recognition approaches id : nlp-m2-14-named-entity-recognition title : M2 14 — Named entities, boundaries and recognition approaches title\_confirmed : false topic\_title : Named entities, boundaries and recognition approaches mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 15 — Lexical meaning, semantic relations and WordNet id : nlp-m2-15-lexical-meaning-wordnet title : M2 15 — Lexical meaning, semantic relations and WordNet title\_confirmed : false topic\_title : Lexical meaning, semantic relations and WordNet mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 16 WSD and Lesk Scoring id : nlp-m2-16-lesk-wsd title : M2 16 WSD and Lesk Scoring title\_confirmed : true topic\_title : Word-sense disambiguation and Lesk overlap scoring mode : m2 kind : focused status : available duration : 10:17 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### M2 17 — Walker category scoring and graph-based WSD id : nlp-m2-17-walker-graph-wsd title : M2 17 — Walker category scoring and graph-based WSD title\_confirmed : false topic\_title : Walker category scoring and graph-based WSD mode : m2 kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Naive Bayes WSD Numerical id : nlp-m2-18-naive-bayes-wsd title : Naive Bayes WSD Numerical title\_confirmed : true topic\_title : Naive Bayes WSD: a complete corpus numerical mode : m2 kind : focused status : available duration : 11:35 reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 01 — NLP foundations: language units, tasks and learning approaches id : tee-01 title : TEE 01 — NLP foundations: language units, tasks and learning approaches title\_confirmed : false topic\_title : NLP foundations: language units, tasks and learning approaches mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLU, NLG, the NLP pipeline and language challenges id : tee-02 title : NLU, NLG, the NLP pipeline and language challenges title\_confirmed : false topic\_title : NLU, NLG, the NLP pipeline and language challenges mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR id : tee-03 title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR title\_confirmed : false topic\_title : Preprocessing and morphology: tokenize, normalize, stem, lemmatize, count TTR mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Vocabulary, label IDs, one-hot encoding and bag of words id : tee-04 title : Vocabulary, label IDs, one-hot encoding and bag of words title\_confirmed : false topic\_title : Vocabulary, label IDs, one-hot encoding and bag of words mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 05 — TF-IDF and cosine: from the corpus to the final similarity id : tee-05 title : TEE 05 — TF-IDF and cosine: from the corpus to the final similarity title\_confirmed : false topic\_title : TF-IDF and cosine: from the corpus to the final similarity mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski id : tee-06 title : TEE 06 — Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski title\_confirmed : false topic\_title : Jaccard and distances: intersection, union, Manhattan, Euclidean, Minkowski mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Corpora and lexical resources: where NLP evidence comes from id : tee-07 title : Corpora and lexical resources: where NLP evidence comes from title\_confirmed : false topic\_title : Corpora and lexical resources: where NLP evidence comes from mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Regular expressions: build and explain exact text patterns id : tee-08 title : Regular expressions: build and explain exact text patterns title\_confirmed : false topic\_title : Regular expressions: build and explain exact text patterns mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors id : tee-09 title : TEE 09 — Distributional semantics, co-occurrence matrices and sentence vectors title\_confirmed : false topic\_title : Distributional semantics, co-occurrence matrices and sentence vectors mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 10 — Word2Vec: CBOW and skip-gram, training pairs and softmax numericals id : tee-10 title : TEE 10 — Word2Vec: CBOW and skip-gram, training pairs and softmax numericals title\_confirmed : false topic\_title : Word2Vec: CBOW and skip-gram, training pairs and softmax numericals mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### GloVe: global counts, probability ratios and weighted loss id : tee-11 title : GloVe: global counts, probability ratios and weighted loss title\_confirmed : false topic\_title : GloVe: global counts, probability ratios and weighted loss mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Reference and coreference: mentions, antecedents and pronoun resolution id : tee-12 title : Reference and coreference: mentions, antecedents and pronoun resolution title\_confirmed : false topic\_title : Reference and coreference: mentions, antecedents and pronoun resolution mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Hobbs algorithm: parse-tree traversal with three complete traces id : tee-13 title : Hobbs algorithm: parse-tree traversal with three complete traces title\_confirmed : false topic\_title : Hobbs algorithm: parse-tree traversal with three complete traces mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Text classification, sentiment and evaluation numericals id : tee-14 title : Text classification, sentiment and evaluation numericals title\_confirmed : false topic\_title : Text classification, sentiment and evaluation numericals mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### TEE 15 — Spelling correction: full edit-distance tables and noisy-channel ranking id : tee-15 title : TEE 15 — Spelling correction: full edit-distance tables and noisy-channel ranking title\_confirmed : false topic\_title : Spelling correction: full edit-distance tables and noisy-channel ranking mode : tee kind : focused status : queued duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Machine translation: meaning preservation and worked processing id : tee-16 title : Machine translation: meaning preservation and worked processing title\_confirmed : false topic\_title : Machine translation: meaning preservation and worked processing mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Question answering: identify, retrieve and justify the answer id : tee-17 title : Question answering: identify, retrieve and justify the answer title\_confirmed : false topic\_title : Question answering: identify, retrieve and justify the answer mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### Information extraction and retrieval: records, indexes and precision/recall id : tee-18 title : Information extraction and retrieval: records, indexes and precision/recall title\_confirmed : false topic\_title : Information extraction and retrieval: records, indexes and precision/recall mode : tee kind : focused status : not\_generated duration reviewed : false notebook\_url : youtube\_url corrected\_visuals : false upload\_pending\_correction : false correction\_note related\_topic\_ids These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — N-grams and Laplace | Reviewed worked numericals id : overview-ngrams-laplace title : NLP M2 — N-grams and Laplace | Reviewed worked numericals title\_confirmed : true topic\_title : NLP M2: N-Grams & Laplace mode : m2 kind : supplemental status : available duration : 7:41 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete counts and four Laplace factors; corrected formulas, conditional-factor wording and rounding. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-03-ngrams-language-models - nlp-m2-04-bigram-trigram-tables - nlp-m2-05-laplace-smoothing-discount These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — HMM and Viterbi | Reviewed worked numericals id : overview-hmm-viterbi title : NLP M2 — HMM and Viterbi | Reviewed worked numericals title\_confirmed : true topic\_title : HMM Prob & Viterbi mode : m2 kind : supplemental status : available duration : 8:30 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Complete HMM factors, all Viterbi candidate scores and backpointers; clarified zero emissions. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-08-markov-hmm-components - nlp-m2-09-hmm-joint-paths - nlp-m2-10-viterbi-decoding These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals id : overview-wsd title : NLP M2 — Word-sense disambiguation | Reviewed Lesk and Bayes numericals title\_confirmed : true topic\_title : WSD: Detective Math Toolkit mode : m2 kind : supplemental status : available duration : 6:11 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Lesk variants, conditional Bayes definitions, smoothing and both worked posteriors; corrected spoken claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-16-lesk-wsd - nlp-m2-17-walker-graph-wsd - nlp-m2-18-naive-bayes-wsd These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Probability and Bayes | Reviewed step-by-step numericals id : overview-probability-bayes title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals title\_confirmed : true topic\_title : NLP M2 — Probability and Bayes | Reviewed step-by-step numericals mode : m2 kind : supplemental status : available duration : 8:53 reviewed : true notebook\_url : youtube\_url corrected\_visuals : true upload\_pending\_correction : true correction\_note : Complete HMM and Naive Bayes calculations; clarified joint scores versus model posteriors. Full transcript and detected slides checked; short voice corrections included. The corrected YouTube upload is still processing. related\_topic\_ids : - nlp-m2-01-foundations-counts-probability - nlp-m2-02-bayes-generative-discriminative These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. ### NLP M2 — Perplexity | Reviewed formulas and solved numericals id : overview-perplexity-logs title : NLP M2 — Perplexity | Reviewed formulas and solved numericals title\_confirmed : true topic\_title : NLP M2 — Perplexity | Reviewed formulas and solved numericals mode : m2 kind : supplemental status : available duration : 9:35 reviewed : true notebook\_url : youtube\_url : corrected\_visuals : true upload\_pending\_correction : false correction\_note : Readable formulas and all worked examples; corrected comparison conditions and context claims. Full transcript and detected slides checked; short voice corrections included. related\_topic\_ids : - nlp-m2-06-perplexity-logs-calculator These are recorded generation statuses, not a live service check. Private YouTube and signed-in NotebookLM links require the owner's account. This export publishes existing catalog metadata, not private video files or an invented transcript. --- ## TEE · Add-k: preserve a normalized vocabulary row Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/about/optional-extensions--add-k/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/about/optional-extensions/add-k Additional explanation; no corresponding question in the two supplied TEE papers. No exam marks are implied. **Optional extension.** Add-k is a supporting generalization of Laplace smoothing. The supplied TEE papers ask add-one bigram sentence probabilities, not an add-k calculation. Complete the actual solved papers first. **Add-k smoothing generalizes add-one smoothing.** Add the same positive pseudo-count k to each allowed outcome. There are V outcomes, so the total pseudo-count is kV. Setting k=1 recovers Laplace smoothing. A smaller k usually changes the original counts less; it is not automatically the best choice for every dataset. $$ P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV},\qquad k>0. $$ Use one fixed vocabulary, including an explicit unknown-word policy if needed. Choose k from training/validation evidence, not by fitting the final test results. Unigram smoothing uses N, the total token count, in place of C(h). **Question. After history h, counts for the complete next-token vocabulary [a,b,c] are [3,1,0]. Calculate add-k probabilities for k=0.5 and check the row sum. [Authored worked example]** ### Answer #### Given C(h)=4; V=3; k=1/2; the zero-count word c is an allowed outcome. #### To find All three probabilities and their interpretation. #### Formula and why it applies $$ P_k(w\mid h)=\frac{C(h,w)+k}{C(h)+kV} $$ Smoothing must adjust the denominator for every pseudo-count added across the row. #### Solution ##### Step 1. Form the new total $$ 4+(0.5)(3)=5.5. $$ ##### Step 2. Calculate a $$ P(a\mid h)=\frac{3+0.5}{5.5}=\frac7{11}\approx0.636364. $$ ##### Step 3. Calculate b and c separately $$ P(b\mid h)=\frac{1+0.5}{5.5}=\frac3{11}\approx0.272727. $$ $$ P(c\mid h)=\frac{0+0.5}{5.5}=\frac1{11}\approx0.090909. $$ ##### Step 4. Check normalization $$ \frac7{11}+\frac3{11}+\frac1{11}=1. $$ **Final answer.** The row is [7/11,3/11,1/11]. The unseen allowed outcome c receives positive probability, and the row remains a probability distribution. This worked lesson was added after the video plan. No matching video has been generated or reviewed. --- ## TEE · Forward HMM: sum paths and normalize posteriors Canonical: https://deep-learning-study-room.vercel.app/read/nlp/tee/about/optional-extensions--forward-probability/ Reader: https://deep-learning-study-room.vercel.app/nlp/#tee/about/optional-extensions/forward-probability Additional explanation; no corresponding question in the two supplied TEE papers. No exam marks are implied. **Optional extension.** Forward totals and tag-path posteriors are optional HMM extensions here. Both supplied TEE papers ask emission and transition matrices, then the joint probability of a given tag sequence. Complete the actual solved papers first. **The forward algorithm finds the total probability of observations when the hidden path is unknown.** Its cell αₜ(j) is the joint probability of the observed prefix o₁…oₜ and ending in hidden state j, summed over every earlier hidden path. It stores two totals per time step in the two-state example; it does not choose one winning predecessor. $$ \alpha_1(j)=\pi_jb_j(o_1) $$ $$ \alpha_t(j)=b_j(o_t)\sum_i\alpha_{t-1}(i)a_{ij} $$ $$ P(O)=\sum_j\alpha_T(j). $$ πⱼ is the first-state probability; aᵢⱼ is transition i→j; bⱼ(oₜ) is the probability that state j emits the current observation. Multiply each earlier total by its transition, **add all incoming alternatives**, then multiply by the current emission. With explicit END transitions, terminate using Σⱼ αT(j)aⱼ,END instead. Do not invent END factors in a fixed-length model that has none. | Question | Operation | Output | | --- | --- | --- | | Given one hidden path and the words | Multiply its start, transitions and emissions | P(path, observations) | | Find the best hidden path | Viterbi: maximum, with backpointers | Best path and its joint score | | Find observation probability | Forward: sum incoming paths | P(observations) | | Find a specified path posterior | Divide its joint score by P(observations) | P(path | observations) | **Question. Use the HMM tables below to compute P(1,3,1) with the forward algorithm. Compare with the Viterbi path H,H,C whose joint score is 0.00672, then obtain that path’s posterior. [Authored worked example]** ### Answer #### Given Use H,C state order and the tables shown. Three observations are 1,3,1. There is no END state. | Quantity | H | C | | --- | --- | --- | | Initial π | 0.8 | 0.2 | | From H → next state | 0.7 | 0.3 | | From C → next state | 0.4 | 0.6 | | State | Emit 1 | Emit 2 | Emit 3 | | --- | --- | --- | --- | | H | 0.2 | 0.4 | 0.4 | | C | 0.5 | 0.4 | 0.1 | #### To find Every forward cell, total observation probability, and P(H,H,C | 1,3,1). #### Formula and why it applies $$ \alpha_t(j)=b_j(o_t)\sum_i\alpha_{t-1}(i)a_{ij} $$ All hidden paths are mutually exclusive alternatives, so their scores must be added. Viterbi’s maximum answers a different question. #### Solution ##### Step 1. Initialize observation 1 $$ \alpha_1(H)=0.8(0.2)=0.16,\qquad\alpha_1(C)=0.2(0.5)=0.10. $$ ##### Step 2. Sum incoming paths to H at observation 3 $$ \alpha_2(H)=[0.16(0.7)+0.10(0.4)](0.4) $$ $$ =[0.112+0.040](0.4)=0.0608. $$ ##### Step 3. Sum incoming paths to C at observation 3 $$ \alpha_2(C)=[0.16(0.3)+0.10(0.6)](0.1) $$ $$ =[0.048+0.060](0.1)=0.0108. $$ ##### Step 4. Calculate final H at observation 1 $$ \alpha_3(H)=[0.0608(0.7)+0.0108(0.4)](0.2) $$ $$ =[0.04256+0.00432](0.2)=0.009376. $$ ##### Step 5. Calculate final C at observation 1 $$ \alpha_3(C)=[0.0608(0.3)+0.0108(0.6)](0.5) $$ $$ =[0.01824+0.00648](0.5)=0.01236. $$ ##### Step 6. Add the final-state alternatives $$ P(1,3,1)=0.009376+0.01236=0.021736. $$ ##### Step 7. Read the table as a trace | State / time | 1: observe 1 | 2: observe 3 | 3: observe 1 | | --- | --- | --- | --- | | H | 0.16 | 0.0608 | 0.009376 | | C | 0.10 | 0.0108 | 0.01236 | | Column total | 0.26 | 0.0716 | 0.021736 | For each new cell, draw arrows from both cells in the previous column. Along each arrow multiply by its transition; add the two contributions and multiply by the destination’s emission. ##### Step 8. Normalize the specified best path $$ P(H,H,C\mid1,3,1)=\frac{0.00672}{0.021736}=\frac{840}{2717}\approx0.309164. $$ The best single path has only part of the total mass. The remaining paths account for the rest. **Final answer.** P(observations)=0.021736. Viterbi selects H,H,C with joint score 0.00672; its posterior is about 0.309164, not 0.00672. **Question. With the same explicitly listed HMM, find the probability of observations 3,1 by forward calculation and by enumerating all four hidden paths. [Authored worked example]** ### Answer #### Given π=(0.8,0.2); transition rows H=(0.7,0.3), C=(0.4,0.6); emissions H=(0.2,0.4,0.4), C=(0.5,0.4,0.1) for labels 1,2,3; no END factor. #### To find P(3,1), and an independent path-sum check. #### Formula and why it applies Forward reuses prefix totals. Exhaustive enumeration is practical for this two-position example and must give the same total. #### Solution ##### Step 1. Initialize observation 3 $$ \alpha_1(H)=0.8(0.4)=0.32,\quad\alpha_1(C)=0.2(0.1)=0.02. $$ ##### Step 2. Calculate the second H cell $$ \alpha_2(H)=[0.32(0.7)+0.02(0.4)](0.2)=0.0464. $$ ##### Step 3. Calculate the second C cell $$ \alpha_2(C)=[0.32(0.3)+0.02(0.6)](0.5)=0.054. $$ ##### Step 4. Sum final cells $$ P(3,1)=0.0464+0.054=0.1004. $$ ##### Step 5. Enumerate each complete path | Path | Full multiplication | Joint score | | --- | --- | --- | | HH | 0.8 × 0.4 × 0.7 × 0.2 | 0.0448 | | HC | 0.8 × 0.4 × 0.3 × 0.5 | 0.048 | | CH | 0.2 × 0.1 × 0.4 × 0.2 | 0.0016 | | CC | 0.2 × 0.1 × 0.6 × 0.5 | 0.006 | $$ 0.0448+0.048+0.0016+0.006=0.1004. $$ **Final answer.** Both methods give 0.1004. The largest single-path score is 0.048, which is smaller than the sum. For K states and T observations, both dense forward and Viterbi recurrences take O(TK²) work. Forward needs only the preceding column for the probability total. Viterbi also stores backpointers to recover the path. Very long sequences need scaling or log-sum-exp for forward; ordinary logarithms alone do not turn a sum into a maximum. **Yaad rakho:** Viterbi = best raasta. Forward = saare raaston ka total. This is a supporting HMM extension, not an assertion that a separate forward-algorithm question appeared in the supplied papers. This worked lesson was added after the video plan. No matching video has been generated or reviewed. # Intelligent Systems II — all public teaching content Exact source JSON: https://deep-learning-study-room.vercel.app/exports/is2.json --- ## M2 · Start here: your IS2 M2 study order Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/m2-start--how-to-use/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/m2-start/how-to-use M2 scope and pattern confirmed in your 8 October class update. Learn **Fuzzy Logic → Genetic Algorithms → Particle Swarm Optimization**, with the required maths explained just before it is used. The answers stay below the questions. ## The confirmed 10-mark pattern | Question | Marks | Attempt rule | Prepare for | | --- | --- | --- | --- | | Q1(a) | 2 | Compulsory | A short numerical or theory question | | Q1(b) | 2 | Compulsory | A short numerical or theory question, including swarm intelligence | | Q2, Q3, Q4 | 3 each | Attempt any two | GA, PSO and fuzzy logic; the mix can include theory and numericals | Your class update specifically highlights fuzzy union/intersection/difference/complement, GA with supplied operators, two-particle PSO minimisation, fuzzy rules, architecture, advantages and disadvantages. Work GA and PSO through **iteration 2**. The mock question numbering is a practice arrangement; the actual exam can rearrange the topics. ## Read in this order 1. [Essential maths](https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--reading-formulas/): candidate versus score, fractions, min/max, binary and two-coordinate movement. 2. [Fuzzy logic](https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--membership-meaning/): membership meaning → graphs → set operations → rule base → Mamdani → defuzzification. Then the complete 32°C / 7-person fan example. 3. [Genetic algorithms](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--meaning/): terms → encoding → selection → crossover/mutation → complete two-generation solutions. 4. [PSO](https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--meaning/): position, velocity and memory → single updates → two particles → two dimensions → class photograph example. 5. [Three 10-mark mocks](https://deep-learning-study-room.vercel.app/read/is2/m2/index/mocks/), followed by [last-day revision](https://deep-learning-study-room.vercel.app/read/is2/m2/index/revision/). Fuzzy relation matrices are kept as additional Unit 2 practice after the three main topics. ACO and GA-for-TSP are in the TEE route; they are not mixed into this M2 reading path. ## How to write a numerical **Full question → Given → To find → Formula and why → numbered calculation steps → final answer.** Read one worked example, then write the same calculation in your notebook. Keep its intermediate table: in GA and PSO that table is the input to the next iteration. ## If the exam is tomorrow First complete fuzzy operations, triangle membership and the Mamdani architecture. Next finish one GA two-generation solution and one PSO two-iteration solution. Then do the remaining variants, theory answers and mocks. Use the formula page for revision after learning the symbols; it is not a substitute for the worked explanations. **Yaad rakho:** Fuzzy = degree aur rules. GA = select, cross, mutate. PSO = velocity, position, best memories. --- ## M2 · Symbols, brackets and iteration labels: how to read a formula Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--reading-formulas/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/reading-formulas Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## Read a formula as a calculation instruction A **variable** is a name for a number that can change. In **f(x)=x²+1**, x is the input, f is the name of the scoring rule, and f(x) is its result. An **equation** says that the expressions on both sides have the same value. A formula tells you how to calculate a quantity. | Read this | Meaning | Concrete example | | --- | --- | --- | | x² | x multiplied by itself | 3² = 3 × 3 = 9 | | 2x or 2(x) | 2 multiplied by x | For x=3: 2x=6, while x²=9 | | f(3) | Apply the rule named f to input 3 | If f(x)=x²+1, then f(3)=10 | | x₁ and x₂ | First and second named entries; the small lower number is a label | If a position is (3,7), x₁=3 and x₂=7 | | x at time t | A version of x at a particular iteration | Old x=3; after a move x=2. The time label is not squaring. | | = and ≈ | Exactly equal; approximately equal after rounding | 1/3 ≈ 0.333333 | | ∈ and { } | Belongs to; a collection of values | x ∈ {2,5} means x may be 2 or 5 | **Evaluate f(x)=2x²+1 at x=−3.** [2 marks · Worked example] #### Given The rule is f(x)=2x²+1. The supplied input is x=−3. #### To find f(−3): the output of this rule for input −3. #### Formula and why it applies $$ f(x)=2x^2+1 $$ The question supplies the rule. Replace x by −3, square that whole input, multiply the square by 2, and add 1. Parentheses keep the negative input together. #### Solution ##### Step 1. Substitute with parentheses $$ f(-3)=2(-3)^2+1 $$ ##### Step 2. Do the square first $$ (-3)^2=(-3)\times(-3)=9 $$ ##### Step 3. Multiply by 2 $$ 2\times9=18 $$ ##### Step 4. Add 1 $$ f(-3)=18+1=19 $$ **Final answer.** f(−3)=19. **Evaluate g(x)=(x−2)/4 at x=10.** [2 marks · Worked example] #### Given The rule is g(x)=(x−2)/4. The supplied input is x=10. #### To find g(10): the output of this rule for input 10. #### Formula and why it applies $$ g(x)=\frac{x-2}{4} $$ The fraction bar groups the numerator x−2. Subtract 2 from the input first, then divide the entire difference by 4. #### Solution ##### Step 1. Replace the input $$ g(10)=\frac{10-2}{4} $$ ##### Step 2. Finish the numerator $$ 10-2=8 $$ ##### Step 3. Divide $$ g(10)=8/4=2 $$ **Final answer.** g(10)=2. **At iteration 0, x=4. The rule is new x = old x − 1. Find x after two iterations.** [2 marks · Worked example] #### Given Initial value x=4 at iteration 0. Every update subtracts 1 from the current value. Perform two updates. #### To find The value after update 1 and after update 2. #### Formula and why it applies $$ x^{(t+1)}=x^{(t)}-1 $$ t labels the current iteration, and t+1 labels the next. The parenthesised upper label records the step; it is not a power. Apply the supplied rule once, then use that new answer as the input to the second update. #### Solution ##### Step 1. Iteration 1 uses the initial value $$ x^{(1)}=4-1=3 $$ ##### Step 2. Iteration 2 uses the updated value $$ x^{(2)}=3-1=2 $$ ##### Step 3. Distinguish a label from a power x at iteration 2 is 2. Squaring the initial value would give 4²=16; that is a different operation. **Final answer.** After update 1: x=3. After update 2: x=2. The sequence is 4 → 3 → 2. **Yaad rakho:** Symbol sirf naam hai. Pehle bolo kaunsa number chahiye, phir uski jagah value likho. --- ## M2 · Position, score, minimisation and maximisation Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--objective/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/objective Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## What are we trying to do in IS2? **An intelligent system** uses information and a decision procedure to choose an output or an action. In M2, you study three different procedures. | Topic | Input | What the procedure does | Output | | --- | --- | --- | --- | | Fuzzy logic | Temperature and occupancy | Applies rules such as IF hot AND crowded THEN fast | A fan-speed setting | | Genetic algorithm (GA) | A population of candidate answers | Selects, combines and changes candidate answers | A good solution and its fitness | | Particle swarm optimization (PSO) | Positions and velocities of particles | Moves candidate answers using individual and shared experience | A best position and its objective value | ## Candidate answer and score are different An **objective function** is the calculation used to score an answer. In **minimisation**, a smaller score is better; in **maximisation**, a larger score is better. The letter **x** names a candidate answer. The notation **f(x)**, read “f of x”, means its score. Parentheses here do not mean f multiplied by x. **For the minimisation problem $$ f(x)=(x-3)^2 $$ , find the score at $$ x=2 $$ .** [2 marks · Practice] #### Given $$ x=2 $$ , $$ f(x)=(x-3)^2 $$ #### To find The objective value f(x). #### Formula and why it applies Replace x by the supplied number. Square means multiply the result by itself. #### Solution ##### Step 1. Subtract 3 $$ x-3=2-3=-1 $$ ##### Step 2. Square the difference $$ f(2)=(-1)^2=(-1)\times(-1)=1 $$ **Final answer.** The position is x = 2; its score is 1. These are two different numbers. **For the minimisation problem $$ f(x)=(x-3)^2 $$ , find the score at $$ x=5 $$ .** [2 marks · Practice] #### Given $$ x=5 $$ , $$ f(x)=(x-3)^2 $$ #### To find The objective value f(x). #### Formula and why it applies Replace x by the supplied number. Square means multiply the result by itself. #### Solution ##### Step 1. Subtract 3 $$ x-3=5-3=2 $$ ##### Step 2. Square the difference $$ f(5)=(2)^2=(2)\times(2)=4 $$ **Final answer.** The position is x = 5; its score is 4. These are two different numbers. **For the minimisation problem $$ f(x)=(x-3)^2 $$ , find the score at $$ x=-1 $$ .** [2 marks · Practice] #### Given $$ x=-1 $$ , $$ f(x)=(x-3)^2 $$ #### To find The objective value f(x). #### Formula and why it applies Replace x by the supplied number. Square means multiply the result by itself. #### Solution ##### Step 1. Subtract 3 $$ x-3=(-1)-3=-4 $$ ##### Step 2. Square the difference $$ f(-1)=(-4)^2=(-4)\times(-4)=16 $$ **Final answer.** The position is x = -1; its score is 16. These are two different numbers. Comparing these candidates: x = 2 has score 1, x = 5 has score 4, and x = −1 has score 16. For minimisation, x = 2 is the best of these three. A negative position is allowed unless the question gives a restriction; a low score, not a low coordinate, decides the winner. **Yaad rakho:** x = answer ki jagah; f(x) = us answer ka score. Min mein chhota score jeetega. --- ## M2 · Fractions, brackets, minimum and maximum Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--arithmetic/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/arithmetic Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## Fractions, min and max A fraction means division: the top number is the **numerator**, and the bottom is the **denominator**. Calculate each bracket first. **min** selects the smallest listed number; **max** selects the largest. Neither word means addition. **Evaluate $$ \frac{32-25}{35-25} $$ .** [2 marks · Practice] #### Given Top: 32 − 25; bottom: 35 − 25. #### To find The quotient. #### Formula and why it applies Subtract separately in the numerator and denominator, then divide. #### Solution ##### Step 1. Calculate the numerator $$ 32-25=7 $$ ##### Step 2. Calculate the denominator $$ 35-25=10 $$ ##### Step 3. Divide $$ \frac{7}{10}=0.7 $$ **Final answer.** 0.7 **Evaluate $$ \frac{35-32}{35-25} $$ .** [2 marks · Practice] #### Given Top: 35 − 32; bottom: 35 − 25. #### To find The quotient. #### Formula and why it applies Subtract separately in the numerator and denominator, then divide. #### Solution ##### Step 1. Calculate the numerator $$ 35-32=3 $$ ##### Step 2. Calculate the denominator $$ 35-25=10 $$ ##### Step 3. Divide $$ \frac{3}{10}=0.3 $$ **Final answer.** 0.3 **Evaluate $$ \frac{7-6}{8-6} $$ .** [2 marks · Practice] #### Given Top: 7 − 6; bottom: 8 − 6. #### To find The quotient. #### Formula and why it applies Subtract separately in the numerator and denominator, then divide. #### Solution ##### Step 1. Calculate the numerator $$ 7-6=1 $$ ##### Step 2. Calculate the denominator $$ 8-6=2 $$ ##### Step 3. Divide $$ \frac{1}{2}=0.5 $$ **Final answer.** 0.5 **Find $$ \max\{\min(0.7,0.2),\min(0.4,0.8)\} $$ .** [2 marks · Practice] #### Given Two inner min operations followed by one outer max. #### To find One number. #### Formula and why it applies Work from the innermost brackets outward. This exact pattern occurs in fuzzy relation composition. #### Solution ##### Step 1. First pair $$ \min(0.7,0.2)=0.2 $$ ##### Step 2. Second pair $$ \min(0.4,0.8)=0.4 $$ ##### Step 3. Choose the larger result $$ \max(0.2,0.4)=0.4 $$ **Final answer.** 0.4. **Yaad rakho:** Pehle bracket, phir divide. AND ka min; OR ka max. --- ## M2 · Percentages, probabilities and cumulative intervals Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--probability/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/probability Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## Probability is a share of a specified total A probability is a number between 0 and 1. In these finite examples, 0 means an event has no probability mass and 1 means it occurs with certainty under the model. In a continuous distribution, a single exact value can have probability zero without being logically impossible; probabilities then describe intervals. 25% means 25 out of 100, so it equals 25÷100=0.25. For equally likely outcomes, probability is favourable outcomes divided by all possible outcomes. For GA selection, it is the candidate’s weight divided by the sum of weights. $$ P(\text{event})=\frac{\text{favourable count}}{\text{total count}} $$ **A bag contains 3 red balls and 7 blue balls. One ball is drawn uniformly at random. Find the probability of red and of blue.** [2 marks · Worked example] #### Given Red=3; blue=7. Each ball is equally likely. #### To find Both probabilities and their percentages. #### Formula and why it applies $$ P(\text{colour})=\frac{\text{balls of that colour}}{\text{all balls}} $$ All balls are equally likely, so counting them gives probability. The denominator counts all balls; the numerator counts the requested colour. #### Solution ##### Step 1. Find the total $$ 3+7=10 $$ ##### Step 2. Red probability $$ P(R)=3/10=0.3=30\% $$ ##### Step 3. Blue probability $$ P(B)=7/10=0.7=70\% $$ **Final answer.** Red 0.3; blue 0.7. Their sum is 1. ## From weights to probabilities to cumulative probabilities A **weight** need not itself be a probability. Weights 1, 2, 3, 4 add to 10. Dividing every weight by 10 makes their total 1: this is **normalisation**. **Cumulative** means “total accumulated so far”. It turns separate shares into boundaries along a line from 0 to 1. **Candidates A, B, C and D have weights 1, 2, 3 and 4. Find probabilities and cumulative probabilities. Which candidate is selected by random draw r=0.55?** [2 marks · Worked example] #### Given Weights [1,2,3,4], draw 0.55. Use intervals including their lower endpoint and excluding their upper endpoint. #### To find Four probabilities, their running totals and the selected candidate. #### Formula and why it applies $$ P_i=F_i/(F_A+F_B+F_C+F_D) $$ F is weight. P is its share. Add P values in row order for the cumulative boundary. #### Solution ##### Step 1. Add all weights $$ 1+2+3+4=10 $$ ##### Step 2. Divide each weight by that same total | Candidate | Division | Probability | | --- | --- | --- | | A | 1÷10 | 0.1 | | B | 2÷10 | 0.2 | | C | 3÷10 | 0.3 | | D | 4÷10 | 0.4 | ##### Step 3. Build the running totals | Candidate | Running addition | Interval | | --- | --- | --- | | A | 0+0.1=0.1 | 0 ≤ r < 0.1 | | B | 0.1+0.2=0.3 | 0.1 ≤ r < 0.3 | | C | 0.3+0.3=0.6 | 0.3 ≤ r < 0.6 | | D | 0.6+0.4=1 | 0.6 ≤ r ≤ 1 | ##### Step 4. Locate the draw 0.55 is at least 0.3 and less than 0.6. It lies in C’s interval. The draw is not compared with C’s weight 3. **Final answer.** Candidate C is selected. Swipe across the diagram to read its labels ↔ ![Each interval width equals its probability. Running totals mark the ends.](https://deep-learning-study-room.vercel.app/is2/fig/depth_roulette.12804497d2.svg) Diagram: Each interval width equals its probability. Running totals mark the ends.. Labels: Probabilities become intervals on the same 0–1 line | A | 0.1 | 0 | B | 0.2 | 0.1 | C | 0.3 | 0.3 | D | 0.4 | 0.6 | 1 | r = 0.55 selects C Each interval width equals its probability. Running totals mark the ends. **Two unnormalised class scores are 0.12 and 0.08. Convert them to probabilities.** [2 marks · Worked example] #### Given Scores A=0.12 and B=0.08; they describe the only two alternatives. #### To find Normalised probabilities. #### Formula and why it applies Divide each score by the sum, preserving their ratio. #### Solution ##### Step 1. Sum the scores $$ 0.12+0.08=0.20 $$ ##### Step 2. Normalise A $$ P(A)=0.12/0.20=0.6 $$ ##### Step 3. Normalise B $$ P(B)=0.08/0.20=0.4 $$ **Final answer.** A=0.6 and B=0.4. The original score 0.12 was not the final probability. **Yaad rakho:** Probability = apna hissa ÷ sabka total. Cumulative = ab tak ke saare hisse jodna. --- ## M2 · Binary from the beginning: positions, place values and bits Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--binary/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/binary Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## 1. Position means the digit's place **A position is simply a place occupied by one digit in a number.** A digit is the symbol written there. The **place value** tells us how much one unit in that position is worth. Start with an ordinary number you already know: **23**. Its two digits are 2 and 3. They do not contribute the same kind of unit. | | LEFT: tens place | RIGHT: ones place | | --- | --- | --- | | Value of this place | 10 | 1 | | Digit written here | 2 | 3 | | Contribution | 2 × 10 = 20 | 3 × 1 = 3 | $$ 23=(2\times 10)+(3\times 1)=20+3 $$ The **2** means two tens because of where it sits. In **32**, the same digit 2 sits on the right and means only two ones. Changing a digit’s position can change its contribution. **Yaad rakho:** Position = digit ki jagah. Place value = us jagah par likhe ONE ki keemat. 23 mein 2 ka contribution 20 hai, kyunki woh tens ki jagah par hai. ## 2. Binary changes the size of the groups Our ordinary number system is **decimal**: it has ten possible digits, 0 to 9. Its places are ones, tens, hundreds, and so on. Moving one place left multiplies the place value by 10. **Binary** has only two possible digits: **0 and 1**. One binary digit is called a **bit**. Therefore, a **4-bit number means four digit boxes**, each holding either 0 or 1. In binary, grouping happens in **twos**. The rightmost box counts single units. After one single, a second single cannot be written as digit “2”, because binary has no digit 2. Instead, the two singles become **one group of two** in the next box on the left. **Binary 10 means one group of two and zero single units.** It represents the ordinary number 2. Read its digits as “one-zero” when discussing binary, so you do not accidentally read it as decimal ten. Two groups of two make **4**, so the next box to the left is the fours place. Two groups of four make **8**, so the next box is the eights place. This is why the values double. | Ordinary number | Binary digits | Meaning | | --- | --- | --- | | 0 | 0 | No single units | | 1 | 1 | One single unit | | 2 | 10 | One two + zero ones = 2 | | 3 | 11 | One two + one one = 3 | | 4 | 100 | One four + zero twos + zero ones = 4 | **Yaad rakho:** Decimal mein 10 ka group; binary mein 2 ka group. Do singles = ek pair; do pairs = ek group of four. ## 3. Build the places from the RIGHT First draw four empty boxes. **Start with the box furthest to the right.** Give that place the value 1. Move left one box at a time, doubling the previous value. 1. **1st box from the right:** value 1. 2. **2nd box from the right:** 1 × 2 = 2. 3. **3rd box from the right:** 2 × 2 = 4. 4. **4th box from the right:** 4 × 2 = 8. Place value: what ONE in this box is worth 4th from right**8** 3rd from right**4** 2nd from right**2** 1st from right**1** ← Double each time you move one box LEFT Start at the RIGHT with 1. Then 1 × 2 = 2, 2 × 2 = 4, and 4 × 2 = 8. We built the values by moving right to left: **1 → 2 → 4 → 8**. Once the boxes are on paper, reading them in the usual left-to-right direction shows **8, 4, 2, 1**. These are the same four places, viewed in the opposite direction. **Position number is not place value.** The third box from the right has position number 3, but its place value is 4. “Third” tells you where the box is; “4” tells you what a 1 in it contributes. If you add a fifth box on the left, its value is 8 × 2 = **16**. The rightmost place stays 1; adding a box on the left does not change the existing places. **Yaad rakho:** Sabse RIGHT se 1 likho. LEFT jaate jaate double karo: 1, 2, 4, 8, 16. Question ke digits baad mein in boxes mein rakho. ## 4. The place value stays fixed; the digit chooses it Keep the digit **1** the same and move it into a different place. Put 0 in every other box. Watch how its position changes the number it represents: | Binary digits | Where the 1 sits | Decimal value | | --- | --- | --- | | 0001 | 1st place from the right: ones | 1 × 1 = 1 | | 0010 | 2nd place from the right: twos | 1 × 2 = 2 | | 0100 | 3rd place from the right: fours | 1 × 4 = 4 | | 1000 | 4th place from the right: eights | 1 × 8 = 8 | In all four rows, the written digit is still 1. Its **position** decides whether that 1 contributes one, two, four or eight. Now put **1 0 1 1** in the four boxes. The values 8, 4, 2 and 1 are labels for the boxes. The written digits tell us which values to include: - **Digit 1:** take one of this place value. - **Digit 0:** take zero of this place value. Place value: what ONE in this box is worth 4th from right**8** 3rd from right**4** 2nd from right**2** 1st from right**1** Digits written in those boxes **1**include **0**skip **1**include **1**include Each box contributes 1 × 8**8** 0 × 4**0** 1 × 2**2** 1 × 1**1** Add the contributions: 8 + 0 + 2 + 1 = 11 in ordinary decimal notation. The 0 above does **not** mean the place value changes from 4 to 0. The place is still worth 4 per unit; the digit says to take zero units of it. Its contribution is therefore 0 × 4 = 0. The three 1s are the same digit, but they sit in different places. They contribute **8, 2 and 1**. Therefore, binary 1011 represents decimal 11. **Yaad rakho:** Box ki value fixed. Andar 1 hai toh value lo; 0 hai toh skip karo. Phir li hui values jodo. ## 5. Binary to decimal: four complete examples **Decode** means convert the binary digits into an ordinary decimal number. In exam notation, a small 2 below a number means binary; a small 10 means decimal. These small labels identify the number system. They are not multipliers. $$ (1011)_2=(11)_{10} $$ This line says: “one-zero-one-one in binary and eleven in decimal represent the same quantity.” **Convert the binary number $$ (1011)_2 $$ to decimal.** [2 marks · Practice] #### Given Binary digits: 1011, in this exact order. #### To find The ordinary decimal number represented by these digits. #### Formula and why it applies Each digit tells us how many of that place value to include. Multiply each digit by its place value, then add. $$ \text{Number}=\text{sum of (digit}\times\text{place value)} $$ #### Solution ##### Step 1. Write the place values above the digits | Place value | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | | Binary digit | 1 | 0 | 1 | 1 | ##### Step 2. Calculate the contribution of the 8-place The digit is 1, so include the value of this place. $$ 1\times 8=8 $$ ##### Step 3. Calculate the contribution of the 4-place The digit is 0, so this place contributes zero. Its place value still stays the same. $$ 0\times 4=0 $$ ##### Step 4. Calculate the contribution of the 2-place The digit is 1, so include the value of this place. $$ 1\times 2=2 $$ ##### Step 5. Calculate the contribution of the 1-place The digit is 1, so include the value of this place. $$ 1\times 1=1 $$ ##### Step 6. Add the contributions $$ 8+0+2+1=11 $$ **Final answer.** $$ (1011)_2=(11)_{10} $$ . **Convert the binary number $$ (0110)_2 $$ to decimal.** [2 marks · Practice] #### Given Binary digits: 0110, in this exact order. #### To find The ordinary decimal number represented by these digits. #### Formula and why it applies Each digit tells us how many of that place value to include. Multiply each digit by its place value, then add. $$ \text{Number}=\text{sum of (digit}\times\text{place value)} $$ #### Solution ##### Step 1. Write the place values above the digits | Place value | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | | Binary digit | 0 | 1 | 1 | 0 | ##### Step 2. Calculate the contribution of the 8-place The digit is 0, so this place contributes zero. Its place value still stays the same. $$ 0\times 8=0 $$ ##### Step 3. Calculate the contribution of the 4-place The digit is 1, so include the value of this place. $$ 1\times 4=4 $$ ##### Step 4. Calculate the contribution of the 2-place The digit is 1, so include the value of this place. $$ 1\times 2=2 $$ ##### Step 5. Calculate the contribution of the 1-place The digit is 0, so this place contributes zero. Its place value still stays the same. $$ 0\times 1=0 $$ ##### Step 6. Add the contributions $$ 0+4+2+0=6 $$ **Final answer.** $$ (0110)_2=(6)_{10} $$ . **Convert the binary number $$ (1001)_2 $$ to decimal.** [2 marks · Practice] #### Given Binary digits: 1001, in this exact order. #### To find The ordinary decimal number represented by these digits. #### Formula and why it applies Each digit tells us how many of that place value to include. Multiply each digit by its place value, then add. $$ \text{Number}=\text{sum of (digit}\times\text{place value)} $$ #### Solution ##### Step 1. Write the place values above the digits | Place value | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | | Binary digit | 1 | 0 | 0 | 1 | ##### Step 2. Calculate the contribution of the 8-place The digit is 1, so include the value of this place. $$ 1\times 8=8 $$ ##### Step 3. Calculate the contribution of the 4-place The digit is 0, so this place contributes zero. Its place value still stays the same. $$ 0\times 4=0 $$ ##### Step 4. Calculate the contribution of the 2-place The digit is 0, so this place contributes zero. Its place value still stays the same. $$ 0\times 2=0 $$ ##### Step 5. Calculate the contribution of the 1-place The digit is 1, so include the value of this place. $$ 1\times 1=1 $$ ##### Step 6. Add the contributions $$ 8+0+0+1=9 $$ **Final answer.** $$ (1001)_2=(9)_{10} $$ . **Convert the binary number $$ (10011)_2 $$ to decimal.** [2 marks · Practice] #### Given Binary digits: 10011, in this exact order. #### To find The ordinary decimal number represented by these digits. #### Formula and why it applies Each digit tells us how many of that place value to include. Multiply each digit by its place value, then add. $$ \text{Number}=\text{sum of (digit}\times\text{place value)} $$ #### Solution ##### Step 1. Write the place values above the digits | Place value | 16 | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | --- | | Binary digit | 1 | 0 | 0 | 1 | 1 | ##### Step 2. Calculate the contribution of the 16-place The digit is 1, so include the value of this place. $$ 1\times 16=16 $$ ##### Step 3. Calculate the contribution of the 8-place The digit is 0, so this place contributes zero. Its place value still stays the same. $$ 0\times 8=0 $$ ##### Step 4. Calculate the contribution of the 4-place The digit is 0, so this place contributes zero. Its place value still stays the same. $$ 0\times 4=0 $$ ##### Step 5. Calculate the contribution of the 2-place The digit is 1, so include the value of this place. $$ 1\times 2=2 $$ ##### Step 6. Calculate the contribution of the 1-place The digit is 1, so include the value of this place. $$ 1\times 1=1 $$ ##### Step 7. Add the contributions $$ 16+0+0+2+1=19 $$ **Final answer.** $$ (10011)_2=(19)_{10} $$ . ## 6. Decimal to binary: choose each digit **Encode** means write an ordinary number using binary digits. The direction of the task is now reversed: the number is known, and you have to choose the 0s and 1s. For four bits, write the places 8, 4, 2, 1. Start with the largest place on the left. Use it only if it does not exceed the amount still needed. Whenever you use a place, subtract it before moving to the next one. **Yaad rakho:** Number banana hai. Badi value se start: jitna bacha hai usmein value fit ho toh 1 aur subtract; fit nahi ho toh 0. **Write the decimal number 6 as an unsigned 4-bit binary number.** [2 marks · Practice] #### Given Decimal number: 6. Required length: 4 bits, meaning exactly 4 digit boxes. #### To find The binary digits, including any zero at the beginning needed to keep the required length. #### Formula and why it applies At each place, compare its value with the amount still needed. If that amount is at least the place value, write 1 and subtract that value. Otherwise write 0 and keep the remaining amount unchanged. This works because each place can be used either once (1) or not at all (0). The selected places must add up to the given number. #### Solution ##### Step 1. Make 4 boxes and label their values | Place value | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | | Digit to choose | □ | □ | □ | □ | Start with the whole number 6 still to make. Work from the largest place on the LEFT to the smallest on the RIGHT. ##### Step 2. Choose the digit in the 8-place We still need 6. Taking 8 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **6**. ##### Step 3. Choose the digit in the 4-place We still need 6. This is at least 4, so take one 4. Write **1** in this box. $$ \text{Remaining}=6-4=2 $$ ##### Step 4. Choose the digit in the 2-place We still need 2. This is at least 2, so take one 2. Write **1** in this box. $$ \text{Remaining}=2-2=0 $$ ##### Step 5. Choose the digit in the 1-place We still need 0. Taking 1 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **0**. ##### Step 6. Read the digits from LEFT to RIGHT The boxes now contain **0 1 1 0**. Join them in the same order: **0110**. The remaining amount is 0, so we have made the required number exactly. ##### Step 7. Check by converting back $$ 0\times 8+1\times 4+1\times 2+0\times 1=6 $$ **Final answer.** $$ (6)_{10}=(0110)_2 $$ . The answer has 4 bits. **Write the decimal number 11 as an unsigned 4-bit binary number.** [2 marks · Practice] #### Given Decimal number: 11. Required length: 4 bits, meaning exactly 4 digit boxes. #### To find The binary digits, including any zero at the beginning needed to keep the required length. #### Formula and why it applies At each place, compare its value with the amount still needed. If that amount is at least the place value, write 1 and subtract that value. Otherwise write 0 and keep the remaining amount unchanged. This works because each place can be used either once (1) or not at all (0). The selected places must add up to the given number. #### Solution ##### Step 1. Make 4 boxes and label their values | Place value | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | | Digit to choose | □ | □ | □ | □ | Start with the whole number 11 still to make. Work from the largest place on the LEFT to the smallest on the RIGHT. ##### Step 2. Choose the digit in the 8-place We still need 11. This is at least 8, so take one 8. Write **1** in this box. $$ \text{Remaining}=11-8=3 $$ ##### Step 3. Choose the digit in the 4-place We still need 3. Taking 4 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **3**. ##### Step 4. Choose the digit in the 2-place We still need 3. This is at least 2, so take one 2. Write **1** in this box. $$ \text{Remaining}=3-2=1 $$ ##### Step 5. Choose the digit in the 1-place We still need 1. This is at least 1, so take one 1. Write **1** in this box. $$ \text{Remaining}=1-1=0 $$ ##### Step 6. Read the digits from LEFT to RIGHT The boxes now contain **1 0 1 1**. Join them in the same order: **1011**. The remaining amount is 0, so we have made the required number exactly. ##### Step 7. Check by converting back $$ 1\times 8+0\times 4+1\times 2+1\times 1=11 $$ **Final answer.** $$ (11)_{10}=(1011)_2 $$ . The answer has 4 bits. **Write the decimal number 19 as an unsigned 5-bit binary number.** [2 marks · Practice] #### Given Decimal number: 19. Required length: 5 bits, meaning exactly 5 digit boxes. #### To find The binary digits, including any zero at the beginning needed to keep the required length. #### Formula and why it applies At each place, compare its value with the amount still needed. If that amount is at least the place value, write 1 and subtract that value. Otherwise write 0 and keep the remaining amount unchanged. This works because each place can be used either once (1) or not at all (0). The selected places must add up to the given number. #### Solution ##### Step 1. Make 5 boxes and label their values | Place value | 16 | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | --- | | Digit to choose | □ | □ | □ | □ | □ | Start with the whole number 19 still to make. Work from the largest place on the LEFT to the smallest on the RIGHT. ##### Step 2. Choose the digit in the 16-place We still need 19. This is at least 16, so take one 16. Write **1** in this box. $$ \text{Remaining}=19-16=3 $$ ##### Step 3. Choose the digit in the 8-place We still need 3. Taking 8 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **3**. ##### Step 4. Choose the digit in the 4-place We still need 3. Taking 4 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **3**. ##### Step 5. Choose the digit in the 2-place We still need 3. This is at least 2, so take one 2. Write **1** in this box. $$ \text{Remaining}=3-2=1 $$ ##### Step 6. Choose the digit in the 1-place We still need 1. This is at least 1, so take one 1. Write **1** in this box. $$ \text{Remaining}=1-1=0 $$ ##### Step 7. Read the digits from LEFT to RIGHT The boxes now contain **1 0 0 1 1**. Join them in the same order: **10011**. The remaining amount is 0, so we have made the required number exactly. ##### Step 8. Check by converting back $$ 1\times 16+0\times 8+0\times 4+1\times 2+1\times 1=19 $$ **Final answer.** $$ (19)_{10}=(10011)_2 $$ . The answer has 5 bits. **Write the decimal number 0 as an unsigned 4-bit binary number.** [2 marks · Practice] #### Given Decimal number: 0. Required length: 4 bits, meaning exactly 4 digit boxes. #### To find The binary digits, including any zero at the beginning needed to keep the required length. #### Formula and why it applies At each place, compare its value with the amount still needed. If that amount is at least the place value, write 1 and subtract that value. Otherwise write 0 and keep the remaining amount unchanged. This works because each place can be used either once (1) or not at all (0). The selected places must add up to the given number. #### Solution ##### Step 1. Make 4 boxes and label their values | Place value | 8 | 4 | 2 | 1 | | --- | --- | --- | --- | --- | | Digit to choose | □ | □ | □ | □ | Start with the whole number 0 still to make. Work from the largest place on the LEFT to the smallest on the RIGHT. ##### Step 2. Choose the digit in the 8-place We still need 0. Taking 8 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **0**. ##### Step 3. Choose the digit in the 4-place We still need 0. Taking 4 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **0**. ##### Step 4. Choose the digit in the 2-place We still need 0. Taking 2 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **0**. ##### Step 5. Choose the digit in the 1-place We still need 0. Taking 1 would be too much, so skip this place. Write **0** in this box. We used nothing from this place. The remaining amount stays **0**. ##### Step 6. Read the digits from LEFT to RIGHT The boxes now contain **0 0 0 0**. Join them in the same order: **0000**. The remaining amount is 0, so we have made the required number exactly. ##### Step 7. Check by converting back $$ 0\times 8+0\times 4+0\times 2+0\times 1=0 $$ **Final answer.** $$ (0)_{10}=(0000)_2 $$ . The answer has 4 bits. ## 7. A zero at the start can keep the required length **Leading zeros** are zeros at the beginning, on the left. Binary 110 and 0110 both represent decimal 6: $$ (110)_2=(1\times4)+(1\times2)+(0\times1)=6 $$ $$ (0110)_2=(0\times8)+(1\times4)+(1\times2)+(0\times1)=6 $$ The added leftmost place contributes zero. It changes the length from three bits to four bits, but not the value. If a GA question requires four bits, write **0110**. This rule is specifically about zeros on the **left**. Appending a zero on the right shifts the existing digits into places worth twice as much. For example, binary 1100 is 8 + 4 = 12, not 6. ## 8. How many bits are enough? For these GA conversions, **unsigned** means the bits represent zero or a positive whole number, with no negative sign. The smallest 4-bit value is 0000. The largest is 1111 because all four places are included. **Find the largest unsigned integer that can be represented using 4 bits. Can decimal 19 fit in 4 bits?** [2 marks · Practice] #### Given Four places with values 8, 4, 2 and 1. Each digit is either 0 or 1. #### To find The largest possible value and whether 19 can be represented. #### Formula and why it applies A digit contributes the most when it is 1. Set every digit to 1 and add all the place values. #### Solution ##### Step 1. Use all four places The largest 4-bit string is **1111**. ##### Step 2. Add their values $$ 8+4+2+1=15 $$ ##### Step 3. Compare with 19 19 is greater than 15, so four bits are insufficient. Add the 16-place on the left. Then 19 = 16 + 2 + 1, giving **10011** in five bits. **Final answer.** The maximum is 15. Decimal 19 needs at least 5 bits. The general rule is $$ \text{largest unsigned value}=2^n-1 $$ . Here n means the number of bits. The power $$ 2^n $$ means multiplying n copies of 2; for four bits, $$ 2^4=2\times2\times2\times2=16 $$ , so the maximum is 16 − 1 = 15. There are 16 different values, from 0 through 15, because zero is also one value. | Bits | Place values from left to right | Unsigned values | | --- | --- | --- | | 3 | 4, 2, 1 | 0 to 7 | | 4 | 8, 4, 2, 1 | 0 to 15 | | 5 | 16, 8, 4, 2, 1 | 0 to 31 | If an exam gives a number outside the capacity of the stated bit length, state that inconsistency. Do not silently cut off a digit: removing the 16-place from 10011 would change 19 into 3. Also retain any separate allowed range stated in the question. ## 9. Where this fits in a genetic algorithm A **chromosome** is the representation of one candidate solution in a genetic algorithm. With the direct unsigned binary encoding used here, decimal 6 is represented by the four-bit chromosome **0110**. During crossover or mutation, the algorithm changes these bits. Afterwards, **decode the resulting chromosome** to find its new value of x. Then use that x in the given objective or fitness function. The binary string, its decimal value, and its fitness score are different things. **Yaad rakho:** Encode: number se bits. Decode: bits se number. GA mein bits badle toh x dobara decode karo, phir uska score nikalo. --- ## M2 · Vectors, coordinates, matrices and transpose from the first entry Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--coordinates/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/coordinates Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## A vector is an ordered list of numbers A **scalar** is one number, such as 5. A **vector** is an ordered list, such as (3,7). Order matters: the first number may be horizontal position and the second vertical position. A **matrix** is a rectangular table of numbers. These words describe how numbers are arranged, not a new kind of arithmetic. | Arrangement | Example | How to read it | | --- | --- | --- | | Scalar | 5 | One value | | Row vector | [3,7] | One row, two entries | | Column vector | $$ \begin{bmatrix}3\\7\end{bmatrix} $$ | Two rows, one entry | | Matrix | $$ \begin{bmatrix}3&7\\4&2\end{bmatrix} $$ | Two rows, two columns | ## Rows go across; columns go down For a matrix A, **A₂₁** means the entry in **row 2, column 1**. The first lower label is the row; the second is the column. Matrix size is written **number of rows × number of columns**. A 2×3 matrix has two rows containing three entries each. Swipe across the diagram to read its labels ↔ ![Read an entry by finding its row first and its column second.](https://deep-learning-study-room.vercel.app/is2/fig/depth_matrix.a8870f4ab6.svg) Diagram: Read an entry by finding its row first and its column second.. Labels: Rows across. Columns down. Entry A₂₃ = 7. | Column 1 | Column 2 | Column 3 | Row 1 | 2 | 5 | 8 | Row 2 | 1 | 4 | 7 | Two rows × three columns = 2 × 3 matrix Read an entry by finding its row first and its column second. **Let A have first row [2,5,8] and second row [1,4,7]. State its size and find A₂₃ and A₁₂.** [2 marks · Worked example] #### Given $$ A=\begin{bmatrix}2&5&8\\1&4&7\end{bmatrix} $$ #### To find Matrix dimensions and two entries. #### Formula and why it applies Read the lower labels in row–column order. #### Solution ##### Step 1. Count the arrangement There are 2 horizontal rows and 3 vertical columns. Size=2×3. ##### Step 2. Find A₂₃ Row 2 is [1,4,7]. Its third entry is 7. ##### Step 3. Find A₁₂ Row 1 is [2,5,8]. Its second entry is 5. **Final answer.** Size 2×3; A₂₃=7; A₁₂=5. **A particle has position x=(3,7), velocity v=(−1,2), and best position p=(1,8). Find p−x, half of that difference, and x+v.** [2 marks · Worked example] #### Given Position x=(3,7); velocity v=(−1,2); personal-best position p=(1,8). First and second coordinates stay in the same order. #### To find The difference p−x, the scaled difference 0.5(p−x), and the updated position x+v. #### Formula and why it applies Match first with first and second with second. A scalar multiplies every coordinate. #### Solution ##### Step 1. Subtract corresponding coordinates $$ p-x=(1-3,\;8-7)=(-2,1) $$ ##### Step 2. Multiply both coordinates by 0.5 $$ 0.5(p-x)=(0.5(-2),\;0.5(1))=(-1,0.5) $$ ##### Step 3. Add the velocity to the position $$ x+v=(3+(-1),\;7+2)=(2,9) $$ **Final answer.** Difference (−2,1); scaled difference (−1,0.5); new position (2,9). ## Transpose exchanges rows and columns The superscript **T** in Aᵀ means **transpose**. It does not mean multiply or raise A to a power. Row 1 of A becomes column 1 of Aᵀ; row 2 becomes column 2. An m×n arrangement becomes n×m. **Find the transpose of A = [[2,5,8],[1,4,7]].** [2 marks · Worked example] #### Given $$ A=\begin{bmatrix}2&5&8\\1&4&7\end{bmatrix} $$ #### To find Aᵀ. #### Formula and why it applies Turn each old row into a new column, keeping its order. #### Solution ##### Step 1. Move the first row [2,5,8] becomes the first column from top to bottom. ##### Step 2. Move the second row [1,4,7] becomes the second column from top to bottom. ##### Step 3. Write the new arrangement $$ A^T=\begin{bmatrix}2&1\\5&4\\8&7\end{bmatrix} $$ **Final answer.** Aᵀ has 3 rows and 2 columns. No number changed. **Calculate the dot product of [2,3] and [4,5].** [2 marks · Worked example] #### Given First vector a=[2,3]; second vector b=[4,5]. Each has two entries. #### To find The dot product a·b, a single number obtained by adding the two matching products. #### Formula and why it applies $$ a\cdot b=a_1b_1+a_2b_2 $$ Multiply corresponding entries, then add the products. Markov matrix multiplication later repeats this for each destination column. #### Solution ##### Step 1. First product $$ 2\times4=8 $$ ##### Step 2. Second product $$ 3\times5=15 $$ ##### Step 3. Add $$ 8+15=23 $$ **Final answer.** The dot product is 23. It is not the vector [8,15]. **Yaad rakho:** Vector = order wali list. Matrix = rows-columns ka table. Entry padhte waqt pehle row, phir column. --- ## M2 · Graph axes, intervals and the fractions inside membership functions Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--graphs/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/graphs Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## Read a graph one input at a time The **horizontal axis** runs left to right and shows input x. The **vertical axis** runs bottom to top and shows the output. A point (3,0.4) means: input 3 gives output 0.4. In a membership graph, the vertical output is a degree between 0 and 1. To read the graph at x=3, start at 3 on the bottom axis, move vertically until you reach the line, then move horizontally to the vertical axis. The height you read is the membership. Swipe across the diagram to read its labels ↔ ![Halfway along a straight rise from height 0 to height 1 gives height 0.5.](https://deep-learning-study-room.vercel.app/is2/fig/depth_slope.c91a1030aa.svg) Diagram: Halfway along a straight rise from height 0 to height 1 gives height 0.5.. Labels: Read x = 4: halfway from 2 to 6, so height = 0.5 | 0 | 0.25 | 0.5 | 1 | 0 | 2 | 3 | 4 | 6 | 8 | μ | Input x | Straight rising membership | x = 4 Halfway along a straight rise from height 0 to height 1 gives height 0.5. | Read this | Meaning | Concrete example | | --- | --- | --- | | x < 5 | Strictly less than 5 | 4.9 qualifies; 5 does not | | x ≤ 5 | Less than or equal to 5 | Both 4.9 and 5 qualify | | 2 ≤ x ≤ 8 | x is between 2 and 8 including both ends | The interval is [2,8] | | [2,8) | Include 2; exclude 8 | Useful for roulette interval boundaries | | Piecewise formula | Choose one calculation according to where x lies | Rising branch, falling branch or outside | **A straight membership line rises from (2,0) to (6,1). Find its height at x=3.** [2 marks · Worked example] #### Given Left input=2; peak input=6; required input=3. #### To find Membership height. #### Formula and why it applies $$ \text{height}=\frac{\text{distance from left}}{\text{total rising width}} $$ A straight line changes by the same proportion of its height as the proportion travelled along its width. #### Solution ##### Step 1. Find distance travelled $$ 3-2=1 $$ ##### Step 2. Find total width $$ 6-2=4 $$ ##### Step 3. Divide distance by width $$ \mu(3)=\frac{1}{4}=0.25 $$ **Final answer.** Membership=0.25. **A straight membership line rises from (2,0) to (6,1). Find its height at x=4.** [2 marks · Worked example] #### Given Left input=2; peak input=6; required input=4. #### To find Membership height. #### Formula and why it applies $$ \text{height}=\frac{\text{distance from left}}{\text{total rising width}} $$ A straight line changes by the same proportion of its height as the proportion travelled along its width. #### Solution ##### Step 1. Find distance travelled $$ 4-2=2 $$ ##### Step 2. Find total width $$ 6-2=4 $$ ##### Step 3. Divide distance by width $$ \mu(4)=\frac{2}{4}=0.5 $$ **Final answer.** Membership=0.5. **A straight membership line rises from (10,0) to (30,1). Find its height at x=25.** [2 marks · Worked example] #### Given Left input=10; peak input=30; required input=25. #### To find Membership height. #### Formula and why it applies $$ \text{height}=\frac{\text{distance from left}}{\text{total rising width}} $$ A straight line changes by the same proportion of its height as the proportion travelled along its width. #### Solution ##### Step 1. Find distance travelled $$ 25-10=15 $$ ##### Step 2. Find total width $$ 30-10=20 $$ ##### Step 3. Divide distance by width $$ \mu(25)=\frac{15}{20}=0.75 $$ **Final answer.** Membership=0.75. **A membership line falls from (5,1) to (8,0). Find its height at x=7.** [2 marks · Worked example] #### Given Peak=5; right zero=8; x=7. #### To find Membership. #### Formula and why it applies $$ \mu(x)=\frac{\text{distance still remaining to right zero}}{\text{total falling width}} $$ #### Solution ##### Step 1. Remaining distance $$ 8-7=1 $$ ##### Step 2. Total falling width $$ 8-5=3 $$ ##### Step 3. Divide $$ \mu(7)=1/3\approx0.333333 $$ **Final answer.** Membership≈0.333333. Using distance already travelled would reverse the falling line. **Yaad rakho:** Chadhti line: left se kitna aage? Utarti line: right tak kitna baaki? Dono mein poori width se divide. --- ## M2 · Sum notation, ordinary averages and weighted averages Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--sums-averages/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/sums-averages Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## Σ means add a list of terms The capital Greek letter **Σ** is read “sum”. The lower label says where to start and the upper label where to stop. The small index i is just a counter: first item, second item, third item. $$ \sum_{i=1}^{3}a_i=a_1+a_2+a_3 $$ **For a₁=2, a₂=5 and a₃=8, calculate Σaᵢ and their mean.** [2 marks · Worked example] #### Given Three numbers: 2,5,8. #### To find Sum and ordinary average. #### Formula and why it applies $$ \bar a=\frac{a_1+a_2+a_3}{3} $$ The bar means average. Each number has equal weight. #### Solution ##### Step 1. Expand the sum symbol $$ \sum_{i=1}^3 a_i=2+5+8 $$ ##### Step 2. Add $$ 2+5=7\quad7+8=15 $$ ##### Step 3. Divide by the count of numbers $$ \bar a=15/3=5 $$ **Final answer.** Sum=15; mean=5. ## A weighted average gives unequal influence If values 20 and 80 have weights 3 and 1, this is equivalent to averaging [20,20,20,80]. Therefore you divide by the **sum of weights**, not by the number of distinct values. Weights may also be decimals. Their total need not initially be 1. $$ \bar z=\frac{w_1z_1+w_2z_2+\cdots}{w_1+w_2+\cdots} $$ **Find the weighted average of [20, 80] with corresponding weights [3, 1].** [2 marks · Worked example] #### Given | Value | Its weight | | --- | --- | | 20 | 3 | | 80 | 1 | #### To find The weighted average of the listed values. #### Formula and why it applies $$ \bar z=\frac{\sum_i w_i z_i}{\sum_i w_i} $$ zᵢ is a value and wᵢ is its corresponding weight. Multiply each value by its own weight, add, then divide by total weight. Here all weights are nonnegative and their total is positive. #### Solution ##### Step 1. Multiply value 1 by its weight $$ 20\times3=60 $$ ##### Step 2. Multiply value 2 by its weight $$ 80\times1=80 $$ ##### Step 3. Add the weighted products $$ 60+80=140 $$ ##### Step 4. Add the weights $$ 3+1=4 $$ ##### Step 5. Divide the two totals $$ \bar z=\frac{140}{4}=35 $$ **Final answer.** 35. **Find the weighted average of [20, 50, 80] with corresponding weights [0.2, 0.6, 0.2].** [2 marks · Worked example] #### Given | Value | Its weight | | --- | --- | | 20 | 0.2 | | 50 | 0.6 | | 80 | 0.2 | #### To find The weighted average of the listed values. #### Formula and why it applies $$ \bar z=\frac{\sum_i w_i z_i}{\sum_i w_i} $$ zᵢ is a value and wᵢ is its corresponding weight. Multiply each value by its own weight, add, then divide by total weight. Here all weights are nonnegative and their total is positive. #### Solution ##### Step 1. Multiply value 1 by its weight $$ 20\times0.2=4 $$ ##### Step 2. Multiply value 2 by its weight $$ 50\times0.6=30 $$ ##### Step 3. Multiply value 3 by its weight $$ 80\times0.2=16 $$ ##### Step 4. Add the weighted products $$ 4+30+16=50 $$ ##### Step 5. Add the weights $$ 0.2+0.6+0.2=1 $$ ##### Step 6. Divide the two totals $$ \bar z=\frac{50}{1}=50 $$ **Final answer.** 50. **Find the weighted average of [65, 85] with corresponding weights [0.3, 0.5].** [2 marks · Worked example] #### Given | Value | Its weight | | --- | --- | | 65 | 0.3 | | 85 | 0.5 | #### To find The weighted average of the listed values. #### Formula and why it applies $$ \bar z=\frac{\sum_i w_i z_i}{\sum_i w_i} $$ zᵢ is a value and wᵢ is its corresponding weight. Multiply each value by its own weight, add, then divide by total weight. Here all weights are nonnegative and their total is positive. #### Solution ##### Step 1. Multiply value 1 by its weight $$ 65\times0.3=19.5 $$ ##### Step 2. Multiply value 2 by its weight $$ 85\times0.5=42.5 $$ ##### Step 3. Add the weighted products $$ 19.5+42.5=62 $$ ##### Step 4. Add the weights $$ 0.3+0.5=0.8 $$ ##### Step 5. Divide the two totals $$ \bar z=\frac{62}{0.8}=77.5 $$ **Final answer.** 77.5. ## Π means multiply a list of terms $$ \prod_{i=1}^{3}a_i=a_1a_2a_3 $$ For [0.5,0.2,0.4], first 0.5×0.2=0.1, then 0.1×0.4=0.04. Σ would instead give 1.1. Naive Bayes multiplies likelihood factors; it does not add them. **Yaad rakho:** Sigma = sab jodo. Product Pi = sab multiply. Weighted mean = value×weight ke total ko weights ke total se divide. --- ## M2 · Powers, roots, absolute values and exp on the calculator Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--powers-exponentials/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/core-maths/powers-exponentials Short prerequisites, with worked examples. Read once before Fuzzy → GA → PSO. ## Powers, roots and exponentials used later A positive integer power repeats multiplication. 2³=2×2×2=8; it does not mean 2×3. A nonzero number raised to power 0 equals 1. A negative power means a reciprocal: 2⁻²=1/(2²)=1/4. A square root reverses a square for a nonnegative result: √9=3. | Read this | Meaning | Concrete example | | --- | --- | --- | | 2⁰, 2¹, 2², 2³ | Binary place values | 1,2,4,8 | | |x| | Absolute value: distance from zero | |−3|=3 and |3|=3 | | e ≈ 2.71828 | A fixed mathematical constant | e⁰=1 | | exp(u) | Another way of writing eᵘ | exp(−2)=e⁻²≈0.135335 | | ln(x) | The natural logarithm; the exponent giving x from e | ln(e²)=2; ln(1)=0 | **Calculate exp(−x²/2) at x=-2.** [2 marks · Worked example] #### Given x=-2; exp means e raised to the bracketed exponent. #### To find The exponential value. #### Formula and why it applies Calculate the complete exponent first. Then use the calculator’s e-to-the-power function. #### Solution ##### Step 1. Square x $$ (-2)^2=4 $$ ##### Step 2. Negate and divide by 2 $$ u=-\frac{4}{2}=-2 $$ ##### Step 3. Evaluate the exponential $$ e^u=e^{-2}\approx 0.135335283 $$ **Final answer.** 0.135335283. **Calculate exp(−x²/2) at x=0.** [2 marks · Worked example] #### Given x=0; exp means e raised to the bracketed exponent. #### To find The exponential value. #### Formula and why it applies Calculate the complete exponent first. Then use the calculator’s e-to-the-power function. #### Solution ##### Step 1. Square x $$ (0)^2=0 $$ ##### Step 2. Negate and divide by 2 $$ u=-\frac{0}{2}=0 $$ ##### Step 3. Evaluate the exponential $$ e^u=e^{0}\approx 1 $$ **Final answer.** 1. **Calculate exp(−x²/2) at x=2.** [2 marks · Worked example] #### Given x=2; exp means e raised to the bracketed exponent. #### To find The exponential value. #### Formula and why it applies Calculate the complete exponent first. Then use the calculator’s e-to-the-power function. #### Solution ##### Step 1. Square x $$ (2)^2=4 $$ ##### Step 2. Negate and divide by 2 $$ u=-\frac{4}{2}=-2 $$ ##### Step 3. Evaluate the exponential $$ e^u=e^{-2}\approx 0.135335283 $$ **Final answer.** 0.135335283. On an fx-991ES-style calculator, eˣ is normally the shifted function above ln. Enter the complete exponent in brackets. Use the negative-number key for −2 where required. Confirm against e⁰=1 before a longer calculation. The exact key labels depend on the model. **Yaad rakho:** exp ka matlab e ki power. Pehle exponent ka poora hisaab, phir eˣ. --- ## M2 · Fuzzy sets: what membership actually means Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--membership-meaning/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/membership-meaning Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Start with an input and a label Suppose the input is **temperature 32°C** and the fuzzy label is **Hot**. A membership function is the rule that converts that temperature into a degree of matching the label. If μHot(32)=0.7, read it aloud as “the membership of 32 degrees in Hot is 0.7”. The lower word Hot names the set; 32 is the input; 0.7 is the output. The same temperature can have membership 0.3 in Medium and 0.7 in Hot. These are two descriptions of one input. Fuzzy memberships do not generally have to sum to 1. By contrast, probabilities of mutually exclusive, exhaustive outcomes do. | Read this | Meaning | Concrete example | | --- | --- | --- | | Universe U | All input values being considered | Temperatures from 0 to 40°C | | Crisp value | One definite measured number | 32°C | | Linguistic term | A word naming a fuzzy category | Low, Medium, High | | μA(x) | Degree to which input x fits category A | μHot(32)=0.7 | **Yaad rakho:** 32 temperature hai; 0.7 us temperature ka Hot label mein degree hai. Dono ko mix mat karo. ## Why “true or false” is sometimes too rigid Suppose a controller says a room is hot only when its temperature is at least 30°C. At 29.9°C it says “not hot”; at 30°C it says “hot”. That sharp boundary is a **crisp** rule. Fuzzy logic allows a gradual degree of belonging to the category “Hot”. **Definition.** A fuzzy set A on a universe U assigns every element x a membership degree μA(x) between 0 and 1. The symbol μ is the Greek letter “mu”; read μA(x) as “the degree to which x belongs to set A”. $$ \mu_A:U\rightarrow[0,1] $$ | Term | Meaning | Temperature example | | --- | --- | --- | | Universe U | All allowed input values | Temperatures from 0°C to 40°C | | Element x | The particular input being considered | 32°C | | Fuzzy set A | A category described by a membership function | Hot | | Membership μA(x) | How strongly x fits that category | μHot(32) = 0.7 | ## Read membership correctly Membership 0 means no belonging; 1 means full belonging; 0.7 means partial belonging under the chosen definition. **It does not mean a 70% chance that the measured temperature is 32°C.** Nor is it a 70% chance that the sensor is correct. Fuzziness concerns the gradual meaning of “Hot”; probability concerns uncertainty about which event occurs. | Input | Warm membership | Hot membership | Interpretation | | --- | --- | --- | --- | | 28 | 0.8 | 0.1 | Fits Warm strongly and Hot weakly | | 32 | 0.3 | 0.7 | Fits Hot more than Warm | | 35 | 0 | 1 | Fully Hot in this chosen model | These memberships need not add to 1. Overlapping categories are allowed. A crisp set is the special case in which membership can only be 0 or 1. A crisp set can also contain infinitely many elements; “crisp” does not mean “finite”. **On U = {20, 25, 30, 35}, the Hot memberships are [0, 0.2, 0.6, 1]. Write the fuzzy set Hot and identify the fully and partly belonging inputs.** [2 marks · Practice] #### Given Ordered inputs [20, 25, 30, 35]; memberships [0, 0.2, 0.6, 1]. #### To find The set notation and interpretation. #### Formula and why it applies In the common fuzzy-set notation μ/x, the slash separates membership from element; it is not division. #### Solution ##### Step 1. Pair each input with its membership | x | μHot(x) | | --- | --- | | 20 | 0 | | 25 | 0.2 | | 30 | 0.6 | | 35 | 1 | ##### Step 2. Write the set $$ Hot=\{0/20,\;0.2/25,\;0.6/30,\;1/35\} $$ ##### Step 3. Interpret 35 fully belongs; 25 and 30 partly belong; 20 does not belong. **Final answer.** Hot = {0/20, 0.2/25, 0.6/30, 1/35}. **Distinguish crisp and fuzzy sets.** [2 marks · Practice] #### Answer | Crisp set | Fuzzy set | | --- | --- | | Membership is 0 or 1 | Membership is any value in [0,1] | | A sharp boundary separates members | A membership function permits gradual boundaries | | Example: temperature ≥ 30°C | Example: degree of being Hot increases gradually | **Yaad rakho:** μ ka matlab: is category mein kitna fit hota hai. Chance nahi. --- ## M2 · Triangular membership: choose the side, then calculate Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--triangle/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/triangle Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Why the triangle has two different fractions Tri(2,5,8) means membership starts at zero at x=2, rises to 1 at x=5, then falls to zero at x=8. The first three numbers are **input positions**, not three membership degrees. | Where is x? | Meaning | Calculation | | --- | --- | --- | | x≤2 or x≥8 | Outside, or at a zero-height endpoint | 0 | | 20; core: degree=1; boundary: 0 0. **Core:** inputs with μ = 1. **Boundary:** inputs with 0 < μ < 1. **Height:** largest membership (more generally the supremum). A set is normal when its height is 1; it is subnormal when its height is below 1. A crossover point has membership 0.5. An α-cut contains inputs with μ ≥ α. For Tri(2,5,8), support is (2,8), core is {5}, height is 1, and the 0.5-cut is [3.5,6.5]. Solve (x−2)/3=0.5 → x=3.5 and (8−x)/3=0.5 → x=6.5. A one-dimensional fuzzy set is **convex** if every α-cut is an interval (or empty). The membership between two inputs must not dip below the smaller endpoint membership. A single triangle is convex; two separated peaks with a lower valley may not be. **Yaad rakho:** Fuzzy A aur NOT A dono partial ho sakte hain. Isliye crisp waali 1 aur 0 identities blindly nahi lagani. --- ## M2 · Mamdani architecture: what every block does Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--mamdani-architecture/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/mamdani-architecture Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Follow one measurement through the controller A **controller** turns measurements into an action setting. Consider temperature 32°C and occupancy 7 people. The input numbers first become membership degrees. The rules then combine those degrees. Only the final defuzzification returns one fan-speed number. | Stage | What goes in | Operation | What comes out | | --- | --- | --- | --- | | Fuzzifier | 32°C and 7 people | Use the supplied membership graphs | Temperature Medium=.3, High=.7; occupancy High=.5 | | Knowledge base | Chosen labels and their shapes | Store graphs and IF–THEN rules | A reusable definition of how the controller works | | Inference | Medium=.3 AND High=.5 | Rule strength=min(.3,.5)=.3 | “Fan High” fires at .3 | | Aggregation | All clipped output sets | Take the largest output degree at each speed | One combined fuzzy output shape | | Defuzzifier | Combined shape | Apply the specified MoM, centroid or representative method | One numerical speed | The **antecedent** is the IF part (“temperature is Medium AND occupancy is High”). The **consequent** is the THEN part (“fan speed is High”). A rule strength of 0.3 is a membership height; it does not mean the fan runs at 30%. ## Definition: a fuzzy inference system A **fuzzy inference system (FIS)** maps inputs to an output by evaluating fuzzy IF–THEN rules. A **Mamdani FIS** uses fuzzy sets in the rule consequents, combines their inferred output sets, and defuzzifies the result. Crisp inputs32°C; 7 people → Fuzzificationdegrees of Warm/Hot etc. → Rules + inferencerule strengths and clipped outputs → Aggregationone combined output shape → Defuzzificationone fan-speed number ## Architecture: what each block contributes | Block | What it stores or calculates | Concrete meaning | | --- | --- | --- | | Fuzzifier | Membership values of the measured inputs | 32°C may be Medium to degree 0.3 and High to degree 0.7 | | Knowledge base: database | Ranges and membership functions | Medium = Tri(15,25,35) | | Knowledge base: rule base | IF–THEN rules | IF temperature High AND occupancy High THEN fan Very High | | Inference engine | Combines conditions and applies consequents | min for AND; clips the output triangle at the rule strength | | Aggregator | Combines all rule outputs | At every output location, take the maximum clipped membership | | Defuzzifier | Converts the aggregate to one crisp value | Use centroid or mean of maxima, as requested | ## Parts of an IF–THEN rule In “IF temperature is High AND occupancy is High THEN fan speed is Very High”, the IF part is the **antecedent** (condition), and the THEN part is the **consequent** (output category). The input variable temperature, the label High, and the numerical membership μHigh(32) are different things. ## Why a rule strength is not yet the fan speed A strength such as 0.3 says how strongly a rule fires. It is on the membership scale 0–1. Fan speed might be on a 0–100% scale. Defuzzification is the step that produces a number on that output scale. **Explain the architecture of a Mamdani fuzzy inference system.** [3 marks · Practice] #### Answer A Mamdani FIS contains a fuzzifier, knowledge base, inference engine and defuzzifier. The database specifies membership functions and the rule base stores IF–THEN rules. Crisp inputs are fuzzified. The inference engine calculates rule firing strengths and constructs output fuzzy sets. These sets are aggregated, then defuzzified into a crisp action. For a fan controller, temperature and occupancy are inputs and fan speed is the output. **Yaad rakho:** Input ka degree → rule ka strength → output ka shape → final number. --- ## M2 · Rule firing, clipping and aggregation: four solved cases Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--rules-clipping/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/rules-clipping Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Clipping means limiting a height A rule firing at strength 0.3 cannot give its consequent a membership above 0.3. At every candidate speed z, keep the smaller of 0.3 and the original output-set membership. The speed axis is unchanged. **At speeds 40, 60 and 80, output set A has memberships [0.1,0.8,0.2] and set B has [0,0.4,1]. Rule A fires at 0.3 and rule B at 0.5. Clip and max-aggregate the sets.** [2 marks · Worked example] #### Given | Speed | Original A membership | Original B membership | | --- | --- | --- | | 40 | 0.1 | 0 | | 60 | 0.8 | 0.4 | | 80 | 0.2 | 1 | Rule A strength=0.3; rule B strength=0.5. Compare memberships at the same speed. #### To find The combined membership at each speed. #### Formula and why it applies $$ \mu_{out}(z)=\max(\min(0.3,\mu_A(z)),\min(0.5,\mu_B(z))) $$ Inner min limits a rule’s height; outer max keeps the stronger support at that speed. #### Solution ##### Step 1. Clip A At 40: min(.3,.1)=.1. At 60: min(.3,.8)=.3. At 80: min(.3,.2)=.2. ##### Step 2. Clip B At 40: min(.5,0)=0. At 60: min(.5,.4)=.4. At 80: min(.5,1)=.5. ##### Step 3. Aggregate at each speed | Speed | Clipped A | Clipped B | max(A,B) | | --- | --- | --- | --- | | 40 | 0.1 | 0 | 0.1 | | 60 | 0.3 | 0.4 | 0.4 | | 80 | 0.2 | 0.5 | 0.5 | **Final answer.** Combined degrees [0.1,0.4,0.5] at speeds [40,60,80]. A separate defuzzification step is still required. **Yaad rakho:** Clip = height ki ceiling. Aggregate = har same x par sabse upar wala height. ## Four different calculations happen in inference **AND/OR** combine conditions within a rule. **Implication** uses the resulting strength to modify that rule’s output membership function. **Aggregation** merges different rule outputs. Here we use min-AND, max-OR, min-implication and max-aggregation. $$ \alpha_{AND}=\min(a,b)\qquad\alpha_{OR}=\max(a,b) $$ $$ \mu_{\mathrm{rule}}(z)=\min(\alpha,\mu_{\mathrm{consequent}}(z)) $$ $$ \mu_{\mathrm{aggregate}}(z)=\max_r\mu_{\mathrm{rule}\ r}(z) $$ α (“alpha”) is the rule firing strength. z names a possible output value. The subscript r labels a rule, not a multiplication. **A rule has two condition degrees 0.7 and 0.5, joined by AND. Find its firing strength.** [2 marks · Practice] #### Given a=0.7; b=0.5; connective=AND. #### To find The strength α. #### Formula and why it applies Use the smaller degree because both conditions must hold. #### Solution ##### Step 1. Select the operator $$ \alpha=\min(0.7,0.5) $$ ##### Step 2. Evaluate $$ \alpha=0.5 $$ **Final answer.** 0.5. **A rule has two condition degrees 0.3 and 0, joined by AND. Find its firing strength.** [2 marks · Practice] #### Given a=0.3; b=0; connective=AND. #### To find The strength α. #### Formula and why it applies Use the smaller degree because both conditions must hold. #### Solution ##### Step 1. Select the operator $$ \alpha=\min(0.3,0) $$ ##### Step 2. Evaluate $$ \alpha=0 $$ **Final answer.** 0. **A rule has two condition degrees 0.2 and 0.8, joined by OR. Find its firing strength.** [2 marks · Practice] #### Given a=0.2; b=0.8; connective=OR. #### To find The strength α. #### Formula and why it applies Use the larger degree because either condition is sufficient. #### Solution ##### Step 1. Select the operator $$ \alpha=\max(0.2,0.8) $$ ##### Step 2. Evaluate $$ \alpha=0.8 $$ **Final answer.** 0.8. **Rule 1 fires at 0.3 and its original output membership at z=60 is 0.8. Rule 2 fires at 0.6 and its original membership at z=60 is 0.2. Find the aggregate membership at z=60 using min-implication and max-aggregation.** [3 marks · Practice] #### Given Rule 1: α₁=0.3, μ₁(60)=0.8. Rule 2: α₂=0.6, μ₂(60)=0.2. #### To find Membership at the single output location z=60 after implication and aggregation. #### Formula and why it applies Clip each original degree at its rule strength; then take the larger clipped degree. #### Solution ##### Step 1. Clip rule 1 $$ \mu_{R1}(60)=\min(0.3,0.8)=0.3 $$ ##### Step 2. Clip rule 2 $$ \mu_{R2}(60)=\min(0.6,0.2)=0.2 $$ ##### Step 3. Aggregate at this output location $$ \mu_{out}(60)=\max(0.3,0.2)=0.3 $$ **Final answer.** The aggregate membership at z=60 is 0.3. This is not a crisp fan-speed answer. Swipe across the diagram to read its labels ↔ ![Above the 0.3 level, the original triangle is cut off. Below 0.3, the original slope remains.](https://deep-learning-study-room.vercel.app/is2/fig/v2_clip.f101b5cfa9.svg) Diagram: Above the 0.3 level, the original triangle is cut off. Below 0.3, the original slope remains.. Labels: Min-implication clips the triangle at strength 0.3 | 0 | 0.3 | 1 | 50 | 54.5 | 65 | 75.5 | 80 | μ | Output z | Original High | clip 0.3 Above the 0.3 level, the original triangle is cut off. Below 0.3, the original slope remains. **Yaad rakho:** AND ke liye min; output clip karne ke liye bhi min; rules ko jodne ke liye max. Har min ka role alag hai. --- ## M2 · Mean of maxima: discrete peaks and clipped plateaus Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--mom/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/mom Definitions → membership graphs → operations → rules → defuzzification → complete controller. **Maximum membership** is a height. **Mean of maxima** averages the horizontal locations attaining that height. If heights at speeds [20,50,80] are [.2,.6,.6], the maximum height is .6 and the qualifying speeds are 50 and 80; MoM=(50+80)/2=65. Do not average [.6,.6], which would return a degree rather than a speed. A continuous flat maximum from 50 to 80 also has midpoint 65. ## Mean of maxima (MoM) **Defuzzification** converts the aggregate membership function into one crisp output. For MoM, first find the largest membership height, then average the output locations attaining that height. Do not average the membership values themselves. $$ z^*_{MoM}=\frac{z_1+\cdots+z_m}{m} $$ For a continuous flat maximum from a to b, the mean is its midpoint: $$ z^*_{MoM}=\frac{a+b}{2} $$ **For the following discrete aggregate, find the mean-of-maxima output.** [2 marks · Practice] #### Given | Output z | Membership μ | | --- | --- | | 10 | 0.2 | | 20 | 0.8 | | 30 | 0.8 | | 40 | 0.3 | #### To find MoM crisp output. #### Formula and why it applies Find all locations at the largest height, then average only those locations. #### Solution ##### Step 1. Find the maximum height $$ \mu_{max}=0.8 $$ ##### Step 2. List the locations attaining it 20, 30 ##### Step 3. Average these locations $$ z^*=\frac{20+30}{2}=25 $$ **Final answer.** 25. **For the following discrete aggregate, find the mean-of-maxima output.** [2 marks · Practice] #### Given | Output z | Membership μ | | --- | --- | | 1 | 0.1 | | 2 | 0.4 | | 3 | 0.9 | | 4 | 0.2 | #### To find MoM crisp output. #### Formula and why it applies Find all locations at the largest height, then average only those locations. #### Solution ##### Step 1. Find the maximum height $$ \mu_{max}=0.9 $$ ##### Step 2. List the locations attaining it 3 ##### Step 3. Average these locations $$ z^*=\frac{3}{1}=3 $$ **Final answer.** 3. **A single triangular output Tri(50,65,80) is clipped at α=0.3. Find its crisp value by mean of maxima.** [3 marks · Practice] #### Given a=50, b=65, c=80; clip height α=0.3. #### To find Both plateau endpoints and their mean. #### Formula and why it applies Set the rising and falling membership fractions equal to 0.3. Every point between those intersections has the maximum height. #### Solution ##### Step 1. Find the left endpoint $$ \frac{z_L-50}{65-50}=0.3 $$ $$ z_L-50=0.3\times15=4.5 $$ $$ z_L=54.5 $$ ##### Step 2. Find the right endpoint $$ \frac{80-z_R}{80-65}=0.3 $$ $$ 80-z_R=4.5 $$ $$ z_R=75.5 $$ ##### Step 3. Average endpoints $$ z^*=(54.5+75.5)/2=65 $$ **Final answer.** MoM = 65. Swipe across the diagram to read its labels ↔ ![The maximum extends over [54.5,75.5], whose midpoint is 65.](https://deep-learning-study-room.vercel.app/is2/fig/v2_clip.f101b5cfa9.svg) Diagram: The maximum extends over [54.5,75.5], whose midpoint is 65.. Labels: Min-implication clips the triangle at strength 0.3 | 0 | 0.3 | 1 | 50 | 54.5 | 65 | 75.5 | 80 | μ | Output z | Original High | clip 0.3 The maximum extends over [54.5,75.5], whose midpoint is 65. For several disjoint continuous maximum plateaus, a continuous MoM uses a length-weighted mean over all maximum intervals; simply averaging two outside endpoints is not generally valid. In tabulated questions, average the listed locations at the maximum. **Yaad rakho:** Sabse upar ka level dekho; us level par x-values ka mean lo. --- ## M2 · Centroid: three table sums and one area calculation Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--centroid/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/centroid Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Centroid means the balance point of the whole shape Imagine the area under the output graph as a thin sheet of uniform material. The centroid is its horizontal balance point. More area on one side pulls the answer toward that side. In a discrete problem, each membership is a weight. In a continuous problem, the entire area supplies the weights. The integral sign ∫ means adding very thin strips across a continuous interval. The denominator ∫μ(z)dz is the total area. The numerator ∫zμ(z)dz is the **first moment**: position multiplied by area, accumulated across the shape. Their ratio has units of the output, such as fan-speed percent. **A non-overlapping aggregate consists of a rectangle of width 4 and height 0.5 centred at z=2, plus a triangle of base 4 and height 0.5 centred horizontally at z=6. Find the area centroid.** [2 marks · Worked example] #### Given Rectangle: width 4, height 0.5, horizontal centre 2. Triangle: base 4, height 0.5, horizontal centroid 6. Their areas do not overlap. #### To find The combined horizontal centroid. #### Formula and why it applies $$ z^*=\frac{A_1c_1+A_2c_2}{A_1+A_2} $$ Each piece contributes area times its own centre. Use this only for a valid decomposition of the aggregate. #### Solution ##### Step 1. Rectangle area $$ A_1=4\times0.5=2 $$ ##### Step 2. Triangle area $$ A_2=\tfrac12\times4\times0.5=1 $$ ##### Step 3. First moments $$ A_1c_1=2\times2=4 $$ $$ A_2c_2=1\times6=6 $$ ##### Step 4. Totals $$ A=2+1=3\qquad M=4+6=10 $$ ##### Step 5. Divide $$ z^*=10/3\approx3.333333 $$ **Final answer.** Centroid≈3.333333, closer to the larger-area piece. For overlapping clipped sets, first form their max-aggregate. Adding the areas of the original overlapping sets would count the overlap twice. ## Centroid: the balance point of the output The centroid uses the whole aggregate membership distribution. For a discrete output table, multiply each output location by its membership, add the products, then divide by total membership. $$ z^*=\frac{\sum_i z_i\mu(z_i)}{\sum_i\mu(z_i)} $$ The index i simply numbers table entries. Σ (“sigma”) means add all listed terms. The denominator must be nonzero. **Find the discrete centroid of the aggregate membership table.** [3 marks · Practice] #### Given | z | μ(z) | | --- | --- | | 10 | 0.2 | | 20 | 0.6 | | 30 | 0.4 | #### To find One centroid value on the output scale. #### Formula and why it applies This question supplies memberships at discrete output locations; use the discrete centroid, not a peak-only approximation. $$ z^*=\frac{\sum z_i\mu_i}{\sum\mu_i} $$ #### Solution ##### Step 1. Multiply every location by its membership | z | μ | z × μ | | --- | --- | --- | | 10 | 0.2 | 2 | | 20 | 0.6 | 12 | | 30 | 0.4 | 12 | ##### Step 2. Add the numerator $$ 2+12+12=26 $$ ##### Step 3. Add the denominator $$ 0.2+0.6+0.4=1.2 $$ ##### Step 4. Divide $$ z^*=\frac{26}{1.2}=21.666666667 $$ **Final answer.** 21.666666667. **Find the discrete centroid of the aggregate membership table.** [3 marks · Practice] #### Given | z | μ(z) | | --- | --- | | 0 | 0.5 | | 5 | 1 | | 10 | 0.5 | #### To find One centroid value on the output scale. #### Formula and why it applies This question supplies memberships at discrete output locations; use the discrete centroid, not a peak-only approximation. $$ z^*=\frac{\sum z_i\mu_i}{\sum\mu_i} $$ #### Solution ##### Step 1. Multiply every location by its membership | z | μ | z × μ | | --- | --- | --- | | 0 | 0.5 | 0 | | 5 | 1 | 5 | | 10 | 0.5 | 5 | ##### Step 2. Add the numerator $$ 0+5+5=10 $$ ##### Step 3. Add the denominator $$ 0.5+1+0.5=2 $$ ##### Step 4. Divide $$ z^*=\frac{10}{2}=5 $$ **Final answer.** 5. **Find the discrete centroid of the aggregate membership table.** [3 marks · Practice] #### Given | z | μ(z) | | --- | --- | | 20 | 0 | | 40 | 0.3 | | 60 | 0.6 | | 80 | 0.1 | #### To find One centroid value on the output scale. #### Formula and why it applies This question supplies memberships at discrete output locations; use the discrete centroid, not a peak-only approximation. $$ z^*=\frac{\sum z_i\mu_i}{\sum\mu_i} $$ #### Solution ##### Step 1. Multiply every location by its membership | z | μ | z × μ | | --- | --- | --- | | 20 | 0 | 0 | | 40 | 0.3 | 12 | | 60 | 0.6 | 36 | | 80 | 0.1 | 8 | ##### Step 2. Add the numerator $$ 0+12+36+8=56 $$ ##### Step 3. Add the denominator $$ 0+0.3+0.6+0.1=1 $$ ##### Step 4. Divide $$ z^*=\frac{56}{1}=56 $$ **Final answer.** 56. ## For a continuous shape, integrate the area $$ z^*=\frac{\int z\mu(z)\,dz}{\int\mu(z)\,dz} $$ The denominator is the area under the aggregate. The numerator is its first moment about zero: area weighted by horizontal location. For a shape made of non-overlapping rectangles and triangles, compute each part’s area A and its own centroid c, then use $$ z^*=(\sum A_jc_j)/(\sum A_j) $$ . Overlapping regions of a max-aggregate must not be counted twice. **An aggregate is a rectangle of height 0.5 from z=0 to z=4, followed by a right triangle decreasing from height 0.5 at z=4 to 0 at z=6. Find its area centroid.** [3 marks · Practice] #### Given Rectangle [0,4], height 0.5; triangle with vertices (4,0),(4,0.5),(6,0). #### To find Centroid of the combined area. #### Formula and why it applies The parts do not overlap except at their boundary. Weight each part’s horizontal centroid by its area. #### Solution ##### Step 1. Rectangle area and centre $$ A_1=4\times0.5=2\qquad c_1=(0+4)/2=2 $$ ##### Step 2. Triangle area and centre $$ A_2=\tfrac12\times2\times0.5=0.5 $$ $$ c_2=(4+4+6)/3=14/3 $$ ##### Step 3. Add first moments $$ A_1c_1+A_2c_2=2(2)+0.5(14/3)=19/3 $$ ##### Step 4. Divide by total area $$ z^*=(19/3)/(2+0.5)=38/15\approx2.533333 $$ **Final answer.** Centroid ≈ 2.533333. The plateau MoM would be 2, so different methods can legitimately give different results. **Yaad rakho:** Centroid = har location × uska weight, total karke total weight se divide. --- ## M2 · Weighted average and centroid are not automatically the same Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--weighted-average/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/weighted-average Definitions → membership graphs → operations → rules → defuzzification → complete controller. This is the same weighted-average calculation taught in Essential maths. Here a representative output value c is weighted by a rule strength α. Keep the pair together: strength .3 belonging to speed 65 contributes .3×65. The denominator is the total strength. It is not automatically the number of rules, and the representative is not automatically the area-centre of a clipped set. ## Weighted average of output representatives Some classroom solutions use one representative (often the peak) per output label and weight it by the firing strength. That is a **weighted-average or height-based calculation**. It is exact for a specified weighted combination of singleton outputs, but is not generally the centroid of a max-aggregated, clipped Mamdani shape. Always name the method. A table of memberships at actual output locations supports the discrete centroid. A list of category peaks and strengths supports the specified representative-weighted calculation. Full continuous output shapes support area centroid. **Using the weighted-average method, combine output representatives [25, 75] mph with respective strengths [0.1, 0.7].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 25 | 0.1 | | 75 | 0.7 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0.1 × 25 | 2.5 | | 0.7 × 75 | 52.5 | ##### Step 2. Add the products $$ 2.5+52.5=55 $$ ##### Step 3. Add the strengths $$ 0.1+0.7=0.8 $$ ##### Step 4. Divide $$ z^*=\frac{55}{0.8}=68.75 $$ **Final answer.** 68.75 mph. **Using the weighted-average method, combine output representatives [40, 65, 85] % fan speed with respective strengths [0, 0.3, 0.5].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 40 | 0 | | 65 | 0.3 | | 85 | 0.5 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0 × 40 | 0 | | 0.3 × 65 | 19.5 | | 0.5 × 85 | 42.5 | ##### Step 2. Add the products $$ 0+19.5+42.5=62 $$ ##### Step 3. Add the strengths $$ 0+0.3+0.5=0.8 $$ ##### Step 4. Divide $$ z^*=\frac{62}{0.8}=77.5 $$ **Final answer.** 77.5 % fan speed. **Using the weighted-average method, combine output representatives [25, 40, 60] minutes with respective strengths [0.6, 0.4, 0.2].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 25 | 0.6 | | 40 | 0.4 | | 60 | 0.2 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0.6 × 25 | 15 | | 0.4 × 40 | 16 | | 0.2 × 60 | 12 | ##### Step 2. Add the products $$ 15+16+12=43 $$ ##### Step 3. Add the strengths $$ 0.6+0.4+0.2=1.2 $$ ##### Step 4. Divide $$ z^*=\frac{43}{1.2}=35.833333333 $$ **Final answer.** 35.833333333 minutes. | Method | What it uses | Limitation | | --- | --- | --- | | MoM | Locations at maximum membership | Ignores lower portions of the shape; can change abruptly | | Area centroid | Entire aggregated output area | Needs an area/moment calculation or numerical integration | | Representative weighted average | Supplied representative values and strengths | Not interchangeable with a clipped Mamdani area centroid | No method is universally “most accurate”. The correct exam calculation is the method specified by the question. If all memberships or all weights are zero, division by zero is undefined; a controller needs an explicit fallback rule. **Yaad rakho:** Formula similar dikhega, par weight kya hai? Membership, area, ya rule strength—pehle woh identify karo. ## Faculty slide example: one temperature input This is a different example from the two-input fan design. With Warm=Tri(20,30,40) and Hot=Tri(30,40,50), temperature 32 gives Warm=(40−32)/10=0.8 and Hot=(32−30)/10=0.2. The slide uses output representatives 50 and 80. **Using the weighted-average method, combine output representatives [50, 80] % fan speed with respective strengths [0.8, 0.2].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 50 | 0.8 | | 80 | 0.2 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0.8 × 50 | 40 | | 0.2 × 80 | 16 | ##### Step 2. Add the products $$ 40+16=56 $$ ##### Step 3. Add the strengths $$ 0.8+0.2=1 $$ ##### Step 4. Divide $$ z^*=\frac{56}{1}=56 $$ **Final answer.** 56 % fan speed. --- ## M2 · Class fan question: 32°C, 7 people, every rule and step Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--class-fan/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/class-fan Definitions → membership graphs → operations → rules → defuzzification → complete controller. There are 3 temperature labels and 3 occupancy labels. Pair each temperature label with each occupancy label: 3×3=9 possible AND rules. At the specified input, any rule containing a zero input membership has strength min(0,other)=0 and contributes no output. The rules with nonzero strength remain; their output labels come from the supplied rule table, not from the magnitude of the strength. This is the fan-controller design question shared in class: temperature 32°C and occupancy 7. The photos contain inconsistent graph breakpoints and arithmetic. A design question permits a stated design; below, every membership function is specified explicitly. These choices make the solution reproducible and should not be read as a uniquely recoverable answer from the photos. **Design a Mamdani fan controller for temperature 0–40°C and occupancy 0–10, with three input labels each and five fan-speed labels on 0–100%. For T=32°C and occupancy=7, use the membership functions and nine rules below. Calculate all memberships, firing strengths, and active output sets. Find fan speed by (a) mean of maxima and (b) the specified representative-weighted average.** [10 marks · Class-question adaptation; full design] #### Given | Variable | Low / Very Low | Medium / Low | High / Medium | High output | Very High output | | --- | --- | --- | --- | --- | --- | | Temperature | Tri(0,10,20) | Tri(15,25,35) | Tri(25,35,40) | — | — | | Occupancy | Tri(0,2,4) | Tri(3,5,7) | Tri(6,8,10) | — | — | | Fan speed | Tri(0,10,20) | Tri(10,25,40) | Tri(20,40,60) | Tri(50,65,80) | Tri(70,85,100) | Input T=32°C; occupancy=7. Use min-AND, min-clipping and max-aggregation. | Temperature | Occupancy | Fan output | | --- | --- | --- | | Low | Low | Very Low | | Low | Medium | Low | | Low | High | Medium | | Medium | Low | Low | | Medium | Medium | Medium | | Medium | High | High | | High | Low | Medium | | High | Medium | High | | High | High | Very High | #### To find Input memberships → nine rule strengths → clipped output → two named crisp outputs. #### Formula and why it applies Use the appropriate triangle branch for membership. AND takes the smaller degree. Each output is clipped at that strength; rules with the same consequent combine by max. MoM uses only the highest plateau. The requested representative weighted average uses category peaks and strengths. $$ \alpha_r=\min(\mu_{T,r},\mu_{O,r})\quad\mu_{out}(z)=\max_r\min(\alpha_r,\mu_{C_r}(z)) $$ #### Solution ##### Step 1. Read the temperature graph Swipe across the diagram to read its labels ↔ ![At 32, Medium is on its falling side; High is on its rising side.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fan_temp.434c9b6635.svg) Diagram: At 32, Medium is on its falling side; High is on its rising side.. Labels: Chosen temperature membership functions | 0 | 0.5 | 1 | 0 | 10 | 15 | 20 | 25 | 32 | 35 | 40 | μ | Temperature in °C | Low | Medium | High | 32°C At 32, Medium is on its falling side; High is on its rising side. $$ \mu_L(32)=0 $$ $$ \mu_M(32)=\frac{35-32}{35-25}=\frac3{10}=0.3 $$ $$ \mu_H(32)=\frac{32-25}{35-25}=\frac7{10}=0.7 $$ ##### Step 2. Read the occupancy graph Swipe across the diagram to read its labels ↔ ![At 7, Medium reaches its right zero and High is halfway up its rising edge.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fan_people.1861531dd1.svg) Diagram: At 7, Medium reaches its right zero and High is halfway up its rising edge.. Labels: Chosen occupancy membership functions | 0 | 0.5 | 1 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | μ | Number of people | Low | Medium | High | 7 people At 7, Medium reaches its right zero and High is halfway up its rising edge. $$ \mu_L(7)=0\qquad\mu_M(7)=0 $$ $$ \mu_H(7)=\frac{7-6}{8-6}=\frac12=0.5 $$ ##### Step 3. Evaluate all nine rules | T label / degree | O label / degree | min = strength | Output | | --- | --- | --- | --- | | Low / 0 | Low / 0 | 0 | Very Low | | Low / 0 | Medium / 0 | 0 | Low | | Low / 0 | High / 0.5 | 0 | Medium | | Medium / 0.3 | Low / 0 | 0 | Low | | Medium / 0.3 | Medium / 0 | 0 | Medium | | Medium / 0.3 | High / 0.5 | 0.3 | High | | High / 0.7 | Low / 0 | 0 | Medium | | High / 0.7 | Medium / 0 | 0 | High | | High / 0.7 | High / 0.5 | 0.5 | Very High | ##### Step 4. Clip and aggregate $$ \mu_{out}(z)=\max\{\min(0.3,\mu_H(z)),\min(0.5,\mu_{VH}(z))\} $$ Swipe across the diagram to read its labels ↔ ![High is clipped at 0.3; Very High is clipped at 0.5. Use the upper envelope where the two shapes overlap.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fan_output.8708d0ea77.svg) Diagram: High is clipped at 0.3; Very High is clipped at 0.5. Use the upper envelope where the two shapes overlap.. Labels: Two active output sets | 0 | 0.5 | 1 | 50 | 54.5 | 65 | 70 | 75 | 77.5 | 85 | 92.5 | 100 | μ | Fan speed (%) | High | Very High | clip 0.3 | clip 0.5 High is clipped at 0.3; Very High is clipped at 0.5. Use the upper envelope where the two shapes overlap. ##### Step 5. Find the highest plateau for MoM The maximum is 0.5, belonging to Very High. Its triangle is (70,85,100). $$ \frac{z_L-70}{15}=0.5\Rightarrow z_L=77.5 $$ $$ \frac{100-z_R}{15}=0.5\Rightarrow z_R=92.5 $$ $$ z^*_{MoM}=\frac{77.5+92.5}{2}=85 $$ ##### Step 6. Multiply the representatives by strengths $$ 0.3\times65=19.5 $$ $$ 0.5\times85=42.5 $$ ##### Step 7. Add, then divide for the weighted average $$ z^*_{WA}=\frac{19.5+42.5}{0.3+0.5}=\frac{62}{0.8}=77.5 $$ **Final answer.** For this explicitly stated design: MoM = 85%; representative-weighted output = 77.5%. These are different methods. 77.5% is not claimed to be the true continuous Mamdani area centroid. **How to adapt it in the exam:** if breakpoints or output representatives differ, recompute the memberships from those data. The sequence of operations stays unchanged. If the question explicitly asks for the area centroid, integrate the max-aggregated shape; do not silently substitute the weighted average above. **Yaad rakho:** Temperature aur people ka degree nikalo → 9 min → active outputs clip → jo method poocha hai woh lagao. ## Why the Very High plateau has midpoint 85 For the stated Very High triangle (70,85,100) clipped at height 0.5, solve the rising edge: (z−70)/(85−70)=0.5, so z−70=0.5×15=7.5 and z=77.5. On the falling edge, (100−z)/(100−85)=0.5, so 100−z=7.5 and z=92.5. Thus the highest flat interval is [77.5,92.5]. Its midpoint is (77.5+92.5)/2=170/2=85. Continue with the [complete area-centroid calculation for this same fan design](https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--centroid-comparison/): 77.086956522%, with all overlap breakpoints and area/moment arithmetic shown. --- ## M2 · The same fan controller: true area centroid versus MoM and weighted average Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--centroid-comparison/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/centroid-comparison Definitions → membership graphs → operations → rules → defuzzification → complete controller. The preceding class-fan lesson makes the input shapes and nine rules explicit because the photographs do not specify one internally consistent design. Keep that exact stated design here: temperature 32°C, occupancy 7, High output Tri(50,65,80) firing at 0.3 and Very High output Tri(70,85,100) firing at 0.5. This continuation calculates the actual area centroid of their max-aggregate. It does not reinterpret 77.5 as a centroid. **For the stated two-input class fan controller at T=32°C and occupancy=7, calculate the true continuous Mamdani area centroid of High=Tri(50,65,80) clipped at 0.3 and Very High=Tri(70,85,100) clipped at 0.5. Compare it with mean of maxima and the specified representative-weighted average.** [6 marks] Worked teaching continuation of the class-design adaptation; the mark allocation here is for practice. #### Given All other output rules have strength zero. Use min for implication, max for aggregation and speed z on 0–100%. The two active rule strengths were calculated from input memberships: min(0.3,0.5)=0.3 and min(0.7,0.5)=0.5. #### To find Every visible segment of the aggregate, its area and first moment, the centroid, and the differences between the three crisp answers. #### Formula and why it applies $$ \mu_{out}(z)=\max\{\min(0.3,\mu_H(z)),\min(0.5,\mu_{VH}(z))\} $$ $$ z^*_{COA}=\frac{\int z\mu_{out}(z)\,dz}{\int\mu_{out}(z)\,dz} $$ Centroid balances all the area under the upper envelope. A straight strip with endpoints (a,u) and (b,v) has the following area A and first moment M; adding only disjoint strips counts overlap once. $$ A=\frac{(b-a)(u+v)}2,\qquad M=\frac{b-a}{6}[(2a+b)u+(a+2b)v] $$ #### Solution ##### Step 1. Locate the two clipped plateaus $$ H:\quad z_L=50+0.3(65-50)=54.5,\quad z_R=80-0.3(80-65)=75.5 $$ $$ VH:\quad z_L=70+0.5(85-70)=77.5,\quad z_R=100-0.5(100-85)=92.5 $$ High has a 0.3 plateau from 54.5 to 75.5. Very High has a 0.5 plateau from 77.5 to 92.5. ##### Step 2. Find where the upper envelope changes owner Very High rises through the High plateau at height 0.3. Solve its rising equation: $$ (z-70)/15=0.3\quad\Rightarrow\quad z=74.5 $$ Before 74.5, clipped High supplies the upper envelope. After 74.5, Very High is above it. The descending High edge from 75.5 to 80 lies underneath and contributes no extra aggregate area. ##### Step 3. Trace the aggregate graph Swipe across the diagram to read its labels ↔ ![Fan output: trace the upper envelope before finding its centroid](https://deep-learning-study-room.vercel.app/is2/fig/deep_fan_centroid.98bf1c217d.svg) Diagram: Fan output: trace the upper envelope before finding its centroid. Labels: Fan output: trace the upper envelope before finding its centroid | High clipped at 0.3; Very High clipped at 0.5. Hidden overlap is counted once. | 0 | 0.3 | 0.5 | 50 | 54.5 | 74.5 | 77.5 | 92.5 | 100 | Centroid ≈77.087% | Fan speed z (%) | Area = 17.25; first moment = 1329.75; centroid = moment ÷ area. Read left to right: (50,0) → (54.5,0.3) → (74.5,0.3) → (77.5,0.5) → (92.5,0.5) → (100,0). Membership is zero outside this support. Each adjacent pair is one straight strip in the table. ##### Step 4. Work the rising first strip explicitly $$ A_1=(54.5-50)(0+0.3)/2=0.675 $$ $$ M_1=\frac{4.5}{6}[(100+54.5)0+(50+109)0.3]=35.775 $$ The first moment has units of speed multiplied by area. Dividing total moment by total area returns speed. ##### Step 5. Calculate all disjoint strips | a | b | u | v | Area A | First moment M | | --- | --- | --- | --- | --- | --- | | 50 | 54.5 | 0 | 0.3 | 0.675 | 35.775 | | 54.5 | 74.5 | 0.3 | 0.3 | 6 | 387 | | 74.5 | 77.5 | 0.3 | 0.5 | 1.2 | 91.35 | | 77.5 | 92.5 | 0.5 | 0.5 | 7.5 | 637.5 | | 92.5 | 100 | 0.5 | 0 | 1.875 | 178.125 | $$ \sum A=0.675+6+1.2+7.5+1.875=17.25 $$ $$ \sum M=35.775+387+91.35+637.5+178.125=1329.75 $$ ##### Step 6. Divide for the actual centroid $$ z^*_{COA}=1329.75/17.25=1773/23\approx77.086956522\% $$ ##### Step 7. Compare methods on the same aggregate | Method | Calculation | Answer | | --- | --- | --- | | Area centroid | 1329.75 ÷ 17.25 | 77.086956522% | | Mean of maxima | (77.5+92.5) ÷ 2 | 85% | | Specified representative weighted average | (0.3×65+0.5×85) ÷ (0.3+0.5) | 77.5% | MoM uses only the highest plateau. The representative method uses two chosen peaks and two rule strengths. The area centroid uses the full max-aggregate, so these are three legitimate but different calculations. **Final answer.** Area centroid ≈77.086956522%; MoM=85%; representative-weighted output=77.5%. Report the method with the value. The centroid is inside the support [50,100], as a balance point must be. **Boundary coverage of this teaching design.** With the ordinary triangles stated in the class adaptation, every temperature degree is zero at T=0 or T=40, and every occupancy degree is zero at occupancy=0 or 10. Then no rule fires and an area centroid is undefined. To make a controller cover the entire declared input range, explicitly redesign outer sets as shoulders or specify an out-of-coverage rule; do not silently divide by zero. The worked T=32, occupancy=7 case has positive area and is unaffected. **Yaad rakho:** Centroid mein poore shaded area ka balance chahiye. Peak × strength wala shortcut same calculation nahi hai. --- ## M2 · Fuzzy theory: definitions, architecture, advantages and applications Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--exam-theory/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/exam-theory Definitions → membership graphs → operations → rules → defuzzification → complete controller. **Explain fuzzy logic with one application.** [2 marks · Practice] #### Answer Fuzzy logic represents gradual truth or category membership using values between 0 and 1. A temperature controller can use Low, Medium and High temperature sets. It evaluates rules such as IF temperature is High THEN fan speed is High, combines the rule outputs and defuzzifies them into a numerical fan speed. This permits gradual control around category boundaries. **Give the advantages and limitations of a Mamdani fuzzy controller.** [3 marks · Practice] #### Answer | Advantages | Limitations | | --- | --- | | Human-readable rules express expert knowledge | Membership functions and rules require careful design | | Can model gradual boundaries without an exact analytic plant model | Rule count can grow rapidly with more inputs and categories | | Handles overlapping linguistic descriptions | A poorly chosen rule base can produce poor or uncovered outputs | | Suitable for many control applications | Does not automatically learn or guarantee optimal control | **Describe the design of a fuzzy fan controller.** [3 marks · Practice] #### Answer Choose temperature and occupancy as inputs and fan speed as output. Define their ranges and Low/Medium/High membership functions. Construct a rule table linking input labels to output labels. Fuzzify measured inputs, use min to obtain AND-rule strengths, clip the consequent sets and max-aggregate them. Apply the specified defuzzification method to obtain a fan speed. Test the design at normal inputs and range boundaries. ## Other applications you can explain | Application | Inputs | Example rule | Output | | --- | --- | --- | --- | | Washing machine | Dirtiness, load | IF dirt High AND load High THEN wash time Long | Wash duration | | Loan decision support | Income, credit score | IF income High AND credit Good THEN approval suitability High | Suitability score (not an automatic legal lending decision) | | Fire warning | Temperature, smoke | IF temperature High OR smoke High THEN risk High | Alarm-risk score | | Vehicle control | Distance, relative speed | IF distance Small AND closing speed High THEN braking Strong | Braking command | **Yaad rakho:** Theory answer mein definition → blocks/steps → ek concrete example. Sirf headings se marks nahi milte. --- ## M2 · Gaussian, singleton and bell memberships Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--smooth-memberships/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-core/smooth-memberships Definitions → membership graphs → operations → rules → defuzzification → complete controller. For a Gaussian membership, c is the input at the centre and σ controls the width. Compute x−c, square it, divide by 2σ², put a minus sign in front, and apply eˣ. The maximum membership is 1 at x=c because the exponent becomes 0. Increasing σ makes the same distance from the centre count as a smaller relative distance, so the curve becomes wider. A Gaussian membership is not the same formula as a normal probability density: there is no 1/(√(2π)σ) multiplier here. ## A smooth membership function A Gaussian membership function reaches 1 at its centre c and decreases smoothly on both sides. Its width σ must be positive. This is a fuzzy membership curve, not a normalised probability density: there is no 1/(√(2π)σ) factor. $$ \mu(x)=\exp\left(-\frac{(x-c)^2}{2\sigma^2}\right) $$ exp(u) means eᵘ. The calculator evaluates it using eˣ; calculate the entire negative exponent first. A squared distance treats equal left and right offsets equally. **For Gaussian membership with c=10 and σ=2, find μ(10).** [2 marks · Practice] #### Given x=10; centre c=10; width σ=2. #### To find Membership degree. #### Formula and why it applies Use squared distance from the centre in the Gaussian membership formula. #### Solution ##### Step 1. Distance from the centre $$ 10-10=0 $$ ##### Step 2. Square the distance and width $$ (0)^2=0\qquad2\sigma^2=2(2^2)=8 $$ ##### Step 3. Exponent $$ -0/8=0 $$ ##### Step 4. Evaluate e to this power $$ \mu(10)=e^{0}\approx1 $$ **Final answer.** 1. **For Gaussian membership with c=10 and σ=2, find μ(12).** [2 marks · Practice] #### Given x=12; centre c=10; width σ=2. #### To find Membership degree. #### Formula and why it applies Use squared distance from the centre in the Gaussian membership formula. #### Solution ##### Step 1. Distance from the centre $$ 12-10=2 $$ ##### Step 2. Square the distance and width $$ (2)^2=4\qquad2\sigma^2=2(2^2)=8 $$ ##### Step 3. Exponent $$ -4/8=-0.5 $$ ##### Step 4. Evaluate e to this power $$ \mu(12)=e^{-0.5}\approx0.60653066 $$ **Final answer.** 0.60653066. **For Gaussian membership with c=10 and σ=2, find μ(14).** [2 marks · Practice] #### Given x=14; centre c=10; width σ=2. #### To find Membership degree. #### Formula and why it applies Use squared distance from the centre in the Gaussian membership formula. #### Solution ##### Step 1. Distance from the centre $$ 14-10=4 $$ ##### Step 2. Square the distance and width $$ (4)^2=16\qquad2\sigma^2=2(2^2)=8 $$ ##### Step 3. Exponent $$ -16/8=-2 $$ ##### Step 4. Evaluate e to this power $$ \mu(14)=e^{-2}\approx0.135335283 $$ **Final answer.** 0.135335283. ## Singleton A singleton fuzzy set has membership 1 at exactly one specified point and 0 elsewhere. For a singleton at c=50: μ(50)=1, μ(49)=0 and μ(51)=0. A singleton consequent is represented by one numerical output; this is why a specified weighted average of singleton outputs is a legitimate calculation. ## Generalised bell (when its formula is supplied) $$ \mu(x)=\frac1{1+\left|\frac{x-c}{a}\right|^{2b}} $$ c is the centre, a>0 controls width and b>0 controls the slope. Absolute value |u| is the nonnegative magnitude: |−2|=2. At x=c membership is 1; at x=c±a membership is 1/2. **For a=2, b=1 and c=10, find generalised-bell membership at x=14.** [2 marks · Practice] #### Given a=2, b=1, c=10; x=14. #### To find μ(14). #### Formula and why it applies Subtract the centre, divide by a, take the absolute value, raise to 2b, then use the reciprocal. #### Solution ##### Step 1. Scaled distance $$ |(14-10)/2|=|2|=2 $$ ##### Step 2. Power $$ 2^{2(1)}=4 $$ ##### Step 3. Final reciprocal $$ \mu(14)=1/(1+4)=0.2 $$ **Final answer.** 0.2. **Yaad rakho:** Gaussian MF ka centre 1 hota hai. GMM density ke normalising factor ko yahan mat lagao. --- ## M2 · GA meaning, biological idea and every basic term Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--meaning/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/meaning Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. | Object | Concrete value | Do not confuse it with | | --- | --- | --- | | Individual number | Row 2 | The candidate’s value x | | Candidate value x | 11 | Its chromosome 1011 | | Chromosome | 1011 | Its fitness | | Fitness when maximising x² | 11×11=121 | Its row number or bit string | In this guide, “gene 3” counts the third bit from the **left** starting at 1. Binary place values were built from the **right** starting with weight 1. These are two labelling conventions, not contradictory place-value rules. Read the end specified in an exam. ## What a genetic algorithm does A **genetic algorithm (GA)** is a population-based optimisation method inspired by biological evolution. It maintains candidate solutions, evaluates their fitness, selects parents, generates children through crossover and mutation, and repeats the process. In biology, genes carry inherited information. In a GA, an encoding plays the role of inherited information. The analogy motivates the operators; it does not mean that a computer chromosome is literally DNA. | Term | Meaning | Four-bit example | | --- | --- | --- | | Individual | One candidate solution | x=11 | | Chromosome | The complete encoded candidate | 1011 | | Gene | One component of an encoding | The third bit | | Allele | The value at a gene | 0 or 1 for binary encoding | | Population | The collection of current individuals | [3,5,10,12] | | Fitness | A score used to favour useful candidates | For max x², fitness of 11 is 121 | | Generation | One cycle producing a new population | Select → cross → mutate → replace | | Search space | All candidate solutions permitted | 4-bit integers: 0–15 | | Elitism | Preserving best individuals unchanged | Copy the best into the next population | An objective describes what the problem wants. Fitness describes what the selection mechanism favours. For maximising a nonnegative objective, they may be the same. For minimising cost, directly selecting in proportion to cost would favour bad candidates; use a specified transformation or a comparison-based selection method. **For a maximisation GA with objective f(x)=x², find the fitness of x=3.** [2 marks · Practice] #### Given x=3; fitness equals the nonnegative objective x². #### To find Fitness, not the encoded chromosome. #### Formula and why it applies Square x to evaluate its objective; binary encoding is a separate representation step. #### Solution ##### Step 1. Substitute x $$ f(3)=3^2 $$ ##### Step 2. Multiply $$ 3\times3=9 $$ **Final answer.** Fitness = 9. **For a maximisation GA with objective f(x)=x², find the fitness of x=7.** [2 marks · Practice] #### Given x=7; fitness equals the nonnegative objective x². #### To find Fitness, not the encoded chromosome. #### Formula and why it applies Square x to evaluate its objective; binary encoding is a separate representation step. #### Solution ##### Step 1. Substitute x $$ f(7)=7^2 $$ ##### Step 2. Multiply $$ 7\times7=49 $$ **Final answer.** Fitness = 49. **For a maximisation GA with objective f(x)=x², find the fitness of x=12.** [2 marks · Practice] #### Given x=12; fitness equals the nonnegative objective x². #### To find Fitness, not the encoded chromosome. #### Formula and why it applies Square x to evaluate its objective; binary encoding is a separate representation step. #### Solution ##### Step 1. Substitute x $$ f(12)=12^2 $$ ##### Step 2. Multiply $$ 12\times12=144 $$ **Final answer.** Fitness = 144. **Yaad rakho:** Chromosome answer ka code; fitness us code wale answer ka score. --- ## M2 · Encoding: binary, real values and permutations Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--encoding/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/encoding Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. ## Expand the compact decoding formula before using it $$ x=b_1 2^3+b_2 2^2+b_3 2^1+b_4 2^0 $$ For four bits, b₁ is the leftmost bit and receives weight 8; b₄ is the rightmost and receives weight 1. In general n−i gives the power belonging to bit i. For 1011, the four terms are 1×8, 0×4, 1×2, 1×1. Add 8+0+2+1=11. For real-range mapping, d/(2ⁿ−1) is the fraction of the way through the possible codes. Multiplying by U−L travels that fraction of the physical range. Adding L shifts the result to start at the lower limit. | 3-bit code | Decoded d | Fraction d/7 | Mapped to [−2,5] | | --- | --- | --- | --- | | 000 | 0 | 0/7=0 | −2+0×7=−2 | | 011 | 3 | 3/7 | −2+(3/7)×7=1 | | 111 | 7 | 7/7=1 | −2+1×7=5 | ## Choose an encoding that can express legal solutions **Binary encoding** uses bit strings; **real-valued encoding** stores real numbers directly; **permutation encoding** stores an order, such as a TSP city tour. An operator valid for bits may create invalid city tours, so encoding and operators must fit the problem. | Representation | Candidate | Appropriate use | | --- | --- | --- | | Binary | 1011 | An unsigned integer or a coded real value | | Real-valued | [0.2, 1.7] | Continuous parameters | | Permutation | [A,C,B,D] | Visit order in TSP | ## Decode an n-bit integer $$ x=b_1 2^{n-1}+b_2 2^{n-2}+\cdots+b_n2^0 $$ b₁ is the leftmost bit, not a coefficient you have to learn. In four bits the weights are 8,4,2,1. The maximum is $$ 2^n-1 $$ . **Decode the 4-bit chromosome 0011.** [2 marks · Practice] #### Given Bits 0011; place values 8,4,2,1. #### To find Integer x. #### Formula and why it applies Multiply each bit by its matching place value, then add. #### Solution ##### Step 1. List contributions | Bit | Place value | Contribution | | --- | --- | --- | | 0 | 8 | 0 | | 0 | 4 | 0 | | 1 | 2 | 2 | | 1 | 1 | 1 | ##### Step 2. Add $$ 0+0+2+1=3 $$ **Final answer.** x=3. **Decode the 4-bit chromosome 1010.** [2 marks · Practice] #### Given Bits 1010; place values 8,4,2,1. #### To find Integer x. #### Formula and why it applies Multiply each bit by its matching place value, then add. #### Solution ##### Step 1. List contributions | Bit | Place value | Contribution | | --- | --- | --- | | 1 | 8 | 8 | | 0 | 4 | 0 | | 1 | 2 | 2 | | 0 | 1 | 0 | ##### Step 2. Add $$ 8+0+2+0=10 $$ **Final answer.** x=10. **Decode the 4-bit chromosome 1111.** [2 marks · Practice] #### Given Bits 1111; place values 8,4,2,1. #### To find Integer x. #### Formula and why it applies Multiply each bit by its matching place value, then add. #### Solution ##### Step 1. List contributions | Bit | Place value | Contribution | | --- | --- | --- | | 1 | 8 | 8 | | 1 | 4 | 4 | | 1 | 2 | 2 | | 1 | 1 | 1 | ##### Step 2. Add $$ 8+4+2+1=15 $$ **Final answer.** x=15. ## Encoding a real interval If the question maps an n-bit integer d to a real interval [L,U], first decode d, then use: $$ x=L+\frac{d}{2^n-1}(U-L) $$ **Map the 3-bit string 101 to the real interval [−2,5] using linear binary decoding.** [3 marks · Practice] #### Given n=3, L=−2, U=5; string 101. #### To find Real x. #### Formula and why it applies The smallest bit code maps to L and the largest to U; the remaining codes are equally spaced. $$ x=L+\frac d{2^n-1}(U-L) $$ #### Solution ##### Step 1. Decode integer d $$ d=1(4)+0(2)+1(1)=5 $$ ##### Step 2. Calculate interval width and maximum code $$ U-L=5-(-2)=7\qquad2^3-1=7 $$ ##### Step 3. Map to the interval $$ x=-2+(5/7)(7)=3 $$ **Final answer.** x=3. **Yaad rakho:** Pehle bits se integer. Real range mapping explicitly diya ho tabhi next formula lagao. --- ## M2 · Roulette selection, cumulative intervals and alternatives Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--selection/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/selection Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. Before the three complete sums below, read the running-total example in [Percentages, probabilities and cumulative intervals](https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--probability/). For weights [1,2,3,4], candidate C’s own probability is 0.3, but its interval is [0.3,0.6). A draw of 0.55 selects C because it lies inside that interval; it is not required to be less than C’s individual probability. ## Selection chooses parents, usually with replacement **Roulette-wheel selection** gives each candidate a probability proportional to a nonnegative fitness weight. A high weight occupies a larger interval on [0,1]. A supplied random number selects the interval containing it. $$ P_i=F_i/\sum_jF_j $$ Fᵢ is the fitness weight of individual i. The sum must be positive. Probabilities add to 1. Cumulative probability is the running sum and supplies the interval boundaries. **Fitness weights of successive individuals are [1, 2, 3, 4]. Select parents using random numbers [0.05, 0.25, 0.55, 0.9].** [3 marks · Practice] #### Given Fitness weights [1, 2, 3, 4]; draws [0.05, 0.25, 0.55, 0.9]. #### To find Parent row number for each draw. #### Formula and why it applies Divide by the fitness sum, form cumulative intervals, then locate each random draw. Use half-open intervals [lower,upper); the final upper endpoint 1 is included. #### Solution ##### Step 1. Sum fitness $$ \sum F=1+2+3+4=10 $$ ##### Step 2. Find probabilities and cumulative bounds | Individual | Fitness | Probability | Cumulative | | --- | --- | --- | --- | | 1 | 1 | 0.1 | 0.1 | | 2 | 2 | 0.2 | 0.3 | | 3 | 3 | 0.3 | 0.6 | | 4 | 4 | 0.4 | 1 | ##### Step 3. Locate each draw | Draw | Selected individual | | --- | --- | | 0.05 | 1 | | 0.25 | 2 | | 0.55 | 3 | | 0.9 | 4 | **Final answer.** Selected row numbers: [1, 2, 3, 4]. Repeated selection is permitted. **Fitness weights of successive individuals are [4, 4, 2]. Select parents using random numbers [0.2, 0.6, 0.85].** [3 marks · Practice] #### Given Fitness weights [4, 4, 2]; draws [0.2, 0.6, 0.85]. #### To find Parent row number for each draw. #### Formula and why it applies Divide by the fitness sum, form cumulative intervals, then locate each random draw. Use half-open intervals [lower,upper); the final upper endpoint 1 is included. #### Solution ##### Step 1. Sum fitness $$ \sum F=4+4+2=10 $$ ##### Step 2. Find probabilities and cumulative bounds | Individual | Fitness | Probability | Cumulative | | --- | --- | --- | --- | | 1 | 4 | 0.4 | 0.4 | | 2 | 4 | 0.4 | 0.8 | | 3 | 2 | 0.2 | 1 | ##### Step 3. Locate each draw | Draw | Selected individual | | --- | --- | | 0.2 | 1 | | 0.6 | 2 | | 0.85 | 3 | **Final answer.** Selected row numbers: [1, 2, 3]. Repeated selection is permitted. **Fitness weights of successive individuals are [0, 3, 1]. Select parents using random numbers [0.2, 0.8].** [3 marks · Practice] #### Given Fitness weights [0, 3, 1]; draws [0.2, 0.8]. #### To find Parent row number for each draw. #### Formula and why it applies Divide by the fitness sum, form cumulative intervals, then locate each random draw. Use half-open intervals [lower,upper); the final upper endpoint 1 is included. #### Solution ##### Step 1. Sum fitness $$ \sum F=0+3+1=4 $$ ##### Step 2. Find probabilities and cumulative bounds | Individual | Fitness | Probability | Cumulative | | --- | --- | --- | --- | | 1 | 0 | 0 | 0 | | 2 | 3 | 0.75 | 0.75 | | 3 | 1 | 0.25 | 1 | ##### Step 3. Locate each draw | Draw | Selected individual | | --- | --- | | 0.2 | 2 | | 0.8 | 3 | **Final answer.** Selected row numbers: [2, 3]. Repeated selection is permitted. ## Other selection methods | Method | Procedure | Why it is used | | --- | --- | --- | | Tournament | Sample a group and take its best candidate | Only objective comparisons are needed | | Rank | Sort candidates; assign weights based on rank | Reduces sensitivity to a single huge raw fitness | | Elitist survival | Preserve top individuals unchanged into the next population | Protects good solutions from destructive variation | Selection is not automatically “replace the weakest with the fittest”. That is one possible classroom reproduction rule. If a problem gives selected parents, a rank rule, tournament contestants or roulette draws, follow that exact instruction. **Yaad rakho:** Probability nikalo → cumulative banao → random number ka interval dekho. --- ## M2 · Crossover and mutation: four complete worked variants Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--operators/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/operators Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. ## Cutting means splitting the written string The **prefix** is the part before the cut; the **suffix** is the part after it. “10 joined with 00 gives 1000” means writing strings next to each other. It is not the arithmetic addition 10+00. | Operation | Parents / instruction | Construction | Children | | --- | --- | --- | --- | | One-point crossover | 10|11 and 01|00 | First prefix + second tail; second prefix + first tail | 1000 and 0111 | | Two-point crossover | 1|01|10 and 0|10|01 | Exchange only the middle blocks | 11010 and 00101 | | Uniform crossover | A=1010, B=0101, mask=1100 | Child 1 takes A where mask=1 and B where mask=0; child 2 reverses this | 1001 and 0110 | | Mutation | 1000; flip position 3 from left | The third bit is 0; replace it by 1; keep every other bit | 1010 | These crossover operators are alternatives. Apply the operator stated in the question; do not apply all three in one generation. Mutation is a separate step after crossover in the worked exam workflow. ## Crossover combines; mutation changes In **single-point crossover**, cut both parents after the specified bit. Child 1 keeps parent 1’s prefix and takes parent 2’s suffix; child 2 does the reverse. A cut after bit 2 in a four-bit string looks like 10|11. **Cross parents 1011 and 0100 after bit 2. Then flip bit 4 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 1011, 0100; cut after bit 2; mutation bit 4. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 10 | 11 and 01 | 00 ##### Step 2. Exchange tails 10 + 00 → 1000 01 + 11 → 0111 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 1000 | 0 | 1 | 1001 | | 0111 | 1 | 0 | 0110 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 1001 | 8 + 0 + 0 + 1 | 9 | | 0110 | 0 + 4 + 2 + 0 | 6 | **Final answer.** Final children 1001 (9) and 0110 (6). **Cross parents 0011 and 1100 after bit 1. Then flip bit 2 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 0011, 1100; cut after bit 1; mutation bit 2. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 0 | 011 and 1 | 100 ##### Step 2. Exchange tails 0 + 100 → 0100 1 + 011 → 1011 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 0100 | 1 | 0 | 0000 | | 1011 | 0 | 1 | 1111 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 0000 | 0 + 0 + 0 + 0 | 0 | | 1111 | 8 + 4 + 2 + 1 | 15 | **Final answer.** Final children 0000 (0) and 1111 (15). **Cross parents 1110 and 0001 after bit 3. Then flip bit 1 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 1110, 0001; cut after bit 3; mutation bit 1. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 111 | 0 and 000 | 1 ##### Step 2. Exchange tails 111 + 1 → 1111 000 + 0 → 0000 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 1111 | 1 | 0 | 0111 | | 0000 | 0 | 1 | 1000 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 0111 | 0 + 4 + 2 + 1 | 7 | | 1000 | 8 + 0 + 0 + 0 | 8 | **Final answer.** Final children 0111 (7) and 1000 (8). **Cross parents 1010 and 1010 after bit 2. Then flip bit 4 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 1010, 1010; cut after bit 2; mutation bit 4. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 10 | 10 and 10 | 10 ##### Step 2. Exchange tails 10 + 10 → 1010 10 + 10 → 1010 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 1010 | 0 | 1 | 1011 | | 1010 | 0 | 1 | 1011 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 1011 | 8 + 0 + 2 + 1 | 11 | | 1011 | 8 + 0 + 2 + 1 | 11 | **Final answer.** Final children 1011 (11) and 1011 (11). **Two-point crossover** exchanges the segment between two cuts. **Uniform crossover** uses a given mask to choose, at each position, which parent contributes a gene. Mutation may flip a bit, perturb a real value, or swap two positions in a permutation. These are different operators; apply the one named in the question. **Yaad rakho:** Crossover = tail exchange. Mutation = specified bit flip. Flip ka matlab 1 bhi 0 banega. --- ## M2 · GA architecture, two-generation flow, constraints and theory Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--algorithm/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/algorithm Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. | Read this | Meaning | Concrete example | | --- | --- | --- | | Feasible | Obeys every stated constraint | x≤10: x=9 qualifies; x=12 does not | | Penalty | A deduction for violating a constraint | For maximisation, subtract 100×excess | | Elitism | Explicitly preserve a best old individual | Only apply if the question specifies it | | Replacement | Choose what population the next generation uses | Our full-replacement examples use the new children | | Stopping condition | A rule ending the repeated process | Stop after two complete generations | ## A complete GA generation Encode + evaluate → Select parents → Crossover → Mutation → Evaluate childrenreplace and repeat A **simple generational GA** replaces the population in generations using a chosen encoding, selection rule and variation operators. A more general GA may use steady-state replacement, different encodings, adaptive operators or explicit elitism. Always state the choices used in your worked answer. ## What “iteration 2” means here Initial population → first generation’s selection/crossover/mutation → new population → second generation’s selection/crossover/mutation → final population. Stopping after the first children and labelling them iteration 2 is not two full generations. ## Stopping conditions Stop after a fixed generation budget, after reaching a specified acceptable objective, or after little improvement for a stated number of generations. A finite run does not prove the global optimum. ## Constraints A **constraint** restricts allowed solutions, such as 0≤x≤15 or “visit every city once”. Common approaches are valid-by-design encoding, repairing an invalid child, rejecting it, or using a penalty. A penalty must reduce attractiveness: subtract it in maximisation or add it to a minimised cost. Do not silently remove the original constraint because you used more bits. **A maximisation problem has objective f(x)=x² and constraint x≤10. The given penalised fitness is F(x)=x²−100 max(0,x−10). Compare x=9 and x=12.** [3 marks · Practice] #### Given Penalty coefficient 100; bound 10; candidates 9 and 12. #### To find Penalised fitness of both candidates. #### Formula and why it applies Only excess above the bound receives a penalty. #### Solution ##### Step 1. Feasible candidate 9 $$ \max(0,9-10)=0 $$ $$ F(9)=81-100(0)=81 $$ ##### Step 2. Infeasible candidate 12 $$ \max(0,12-10)=2 $$ $$ F(12)=144-100(2)=-56 $$ ##### Step 3. Compare For this maximisation fitness, 81 > −56, so x=9 is preferred. Raw objective alone would misleadingly favour 12. **Final answer.** F(9)=81; F(12)=−56. | Advantages | Limitations | | --- | --- | | Derivative-free; supports many encodings | Many objective evaluations may be costly | | Population explores several candidates | Premature convergence is possible | | Crossover and mutation can create new candidates | Parameter and encoding choices affect results | | Applicable to discontinuous or combinatorial searches | No guaranteed global optimum; variation can worsen fitness | **Yaad rakho:** Do generation = do baar poora cycle. Sirf children banana aur decode karna do alag iterations nahi. --- ## M2 · Two generations with given selection, cut and mutation Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--two-generations/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/two-generations Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. Swipe across the diagram to read its labels ↔ ![This diagram traces the first pair in generation 1. The full solution also processes the second pair and the entire second generation.](https://deep-learning-study-room.vercel.app/is2/fig/v2_ga_generation.fa7bf86c05.svg) Diagram: This diagram traces the first pair in generation 1. The full solution also processes the second pair and the entire second generation.. Labels: One complete GA generation: read from top to bottom | Selected parents | 1010 × 1100 | Cut after bit 2 | Mark the cut | 10 | 10 11 | 00 | Keep prefix, exchange suffix | After crossover | 1000 1110 | Children are 8 and 14 | Flip the last bit | 1001 1111 | 0 becomes 1 in each child | Decode and evaluate | 9² = 81 15² = 225 | These are the children’s fitness values This diagram traces the first pair in generation 1. The full solution also processes the second pair and the entire second generation. **Maximise $$ f(x)=x^2 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [3, 5, 10, 12]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 4 of children [1, 2, 3, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Current individual row numbers [3, 4, 4, 3] | | 2 | Current individual row numbers [2, 3, 2, 3] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Use exactly the supplied selection rule; replacing the weakest by the strongest is not a universal GA step. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 3 | 0011 | 3 × 3 = 9 | | 2 | 5 | 0101 | 5 × 5 = 25 | | 3 | 10 | 1010 | 10 × 10 = 100 | | 4 | 12 | 1100 | 12 × 12 = 144 | ##### Step 2. Generation 1: apply the supplied selection order The question directly supplies individual numbers [3, 4, 4, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 3 | 10 | 1010 | | 2 | 4 | 12 | 1100 | | 3 | 4 | 12 | 1100 | | 4 | 3 | 10 | 1010 | ##### Step 3. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 10 | 10 | 11 | 00 | 1000 | 1110 | | 3 & 4 | 11 | 00 | 10 | 10 | 1110 | 1000 | ##### Step 4. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1000 | Flip bit 4 | 1001 | | 2 | 1110 | Flip bit 4 | 1111 | | 3 | 1110 | Flip bit 4 | 1111 | | 4 | 1000 | Flip bit 4 | 1001 | ##### Step 5. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1001 | 8 + 0 + 0 + 1 = 9 | 9² = 81 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1001 | 8 + 0 + 0 + 1 = 9 | 9² = 81 | ##### Step 6. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{9+25+100+144}{4}=\frac{278}{4}=69.5 $$ $$ \overline f_{new}=\frac{81+225+225+81}{4}=\frac{612}{4}=153 $$ Best objective: 144 → 225. Higher is better. The next population is [9, 15, 15, 9]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 7. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 9 | 1001 | 9 × 9 = 81 | | 2 | 15 | 1111 | 15 × 15 = 225 | | 3 | 15 | 1111 | 15 × 15 = 225 | | 4 | 9 | 1001 | 9 × 9 = 81 | ##### Step 8. Generation 2: apply the supplied selection order The question directly supplies individual numbers [2, 3, 2, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 2 | 15 | 1111 | | 2 | 3 | 15 | 1111 | | 3 | 2 | 15 | 1111 | | 4 | 3 | 15 | 1111 | ##### Step 9. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 11 | 11 | 11 | 11 | 1111 | 1111 | | 3 & 4 | 11 | 11 | 11 | 11 | 1111 | 1111 | ##### Step 10. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1111 | Flip bit 4 | 1110 | | 2 | 1111 | Flip bit 4 | 1110 | | 3 | 1111 | Flip bit 4 | 1110 | | 4 | 1111 | Flip bit 4 | 1110 | ##### Step 11. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | ##### Step 12. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{81+225+225+81}{4}=\frac{612}{4}=153 $$ $$ \overline f_{new}=\frac{196+196+196+196}{4}=\frac{784}{4}=196 $$ Best objective: 225 → 196. Higher is better. The next population is [14, 14, 14, 14]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [14, 14, 14, 14]. Best final objective = 196 at x = [14]. --- ## M2 · Two generations including roulette selection Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--roulette-generations/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/roulette-generations Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. **Maximise $$ f(x)=x(15-x)+1 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [2, 5, 9, 12]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 4 of children [2, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Random draws [0.1, 0.4, 0.65, 0.9] | | 2 | Random draws [0.2, 0.4, 0.6, 0.95] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Use exactly the supplied selection rule; replacing the weakest by the strongest is not a universal GA step. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 2 | 0010 | 2 × (15 − 2) + 1 = 27 | | 2 | 5 | 0101 | 5 × (15 − 5) + 1 = 51 | | 3 | 9 | 1001 | 9 × (15 − 9) + 1 = 55 | | 4 | 12 | 1100 | 12 × (15 − 12) + 1 = 37 | ##### Step 2. Generation 1: calculate roulette intervals Fitness weights are nonnegative. The nonnegative objective itself is the fitness in this maximisation example. $$ \sum F=27+51+55+37\approx170 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 27 ÷ 170 ≈ 0.158823529 | 0 + 0.158823529 ≈ 0.158823529 | | 2 | 51 ÷ 170 ≈ 0.3 | 0.158823529 + 0.3 ≈ 0.458823529 | | 3 | 55 ÷ 170 ≈ 0.323529412 | 0.458823529 + 0.323529412 ≈ 0.782352941 | | 4 | 37 ÷ 170 ≈ 0.217647059 | 0.782352941 + 0.217647059 ≈ 1 | ##### Step 3. Generation 1: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.1 | 0 ≤ 0.1 < 0.158823529 | 1 | 2 | | 0.4 | 0.158823529 ≤ 0.4 < 0.458823529 | 2 | 5 | | 0.65 | 0.458823529 ≤ 0.65 < 0.782352941 | 3 | 9 | | 0.9 | 0.782352941 ≤ 0.9 < 1 | 4 | 12 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 4. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 00 | 10 | 01 | 01 | 0001 | 0110 | | 3 & 4 | 10 | 01 | 11 | 00 | 1000 | 1101 | ##### Step 5. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 0001 | No mutation | 0001 | | 2 | 0110 | Flip bit 4 | 0111 | | 3 | 1000 | No mutation | 1000 | | 4 | 1101 | Flip bit 4 | 1100 | ##### Step 6. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 0001 | 0 + 0 + 0 + 1 = 1 | 1(15−1)+1 = 15 | | 0111 | 0 + 4 + 2 + 1 = 7 | 7(15−7)+1 = 57 | | 1000 | 8 + 0 + 0 + 0 = 8 | 8(15−8)+1 = 57 | | 1100 | 8 + 4 + 0 + 0 = 12 | 12(15−12)+1 = 37 | ##### Step 7. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{27+51+55+37}{4}=\frac{170}{4}=42.5 $$ $$ \overline f_{new}=\frac{15+57+57+37}{4}=\frac{166}{4}=41.5 $$ Best objective: 55 → 57. Higher is better. The next population is [1, 7, 8, 12]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 8. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 1 | 0001 | 1 × (15 − 1) + 1 = 15 | | 2 | 7 | 0111 | 7 × (15 − 7) + 1 = 57 | | 3 | 8 | 1000 | 8 × (15 − 8) + 1 = 57 | | 4 | 12 | 1100 | 12 × (15 − 12) + 1 = 37 | ##### Step 9. Generation 2: calculate roulette intervals Fitness weights are nonnegative. The nonnegative objective itself is the fitness in this maximisation example. $$ \sum F=15+57+57+37\approx166 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 15 ÷ 166 ≈ 0.090361446 | 0 + 0.090361446 ≈ 0.090361446 | | 2 | 57 ÷ 166 ≈ 0.343373494 | 0.090361446 + 0.343373494 ≈ 0.43373494 | | 3 | 57 ÷ 166 ≈ 0.343373494 | 0.43373494 + 0.343373494 ≈ 0.777108434 | | 4 | 37 ÷ 166 ≈ 0.222891566 | 0.777108434 + 0.222891566 ≈ 1 | ##### Step 10. Generation 2: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.2 | 0.090361446 ≤ 0.2 < 0.43373494 | 2 | 7 | | 0.4 | 0.090361446 ≤ 0.4 < 0.43373494 | 2 | 7 | | 0.6 | 0.43373494 ≤ 0.6 < 0.777108434 | 3 | 8 | | 0.95 | 0.777108434 ≤ 0.95 < 1 | 4 | 12 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 11. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 01 | 11 | 01 | 11 | 0111 | 0111 | | 3 & 4 | 10 | 00 | 11 | 00 | 1000 | 1100 | ##### Step 12. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 0111 | No mutation | 0111 | | 2 | 0111 | Flip bit 4 | 0110 | | 3 | 1000 | No mutation | 1000 | | 4 | 1100 | Flip bit 4 | 1101 | ##### Step 13. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 0111 | 0 + 4 + 2 + 1 = 7 | 7(15−7)+1 = 57 | | 0110 | 0 + 4 + 2 + 0 = 6 | 6(15−6)+1 = 55 | | 1000 | 8 + 0 + 0 + 0 = 8 | 8(15−8)+1 = 57 | | 1101 | 8 + 4 + 0 + 1 = 13 | 13(15−13)+1 = 27 | ##### Step 14. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{15+57+57+37}{4}=\frac{166}{4}=41.5 $$ $$ \overline f_{new}=\frac{57+55+57+27}{4}=\frac{196}{4}=49 $$ Best objective: 57 → 57. Higher is better. The next population is [7, 6, 8, 13]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [7, 6, 8, 13]. Best final objective = 57 at x = [7, 8]. --- ## M2 · GA minimisation: transform cost before roulette selection Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--minimisation/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/ga-core/minimisation Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. Roulette favours a larger weight. A minimisation objective instead favours a smaller cost. The stated transformation F=1/(1+cost) reverses their order for nonnegative costs: cost 0 → weight 1/1=1; cost 3 → 1/4=.25; cost 24 → 1/25=.04. The +1 prevents division by zero. This is one specified transformation, not a universal rule for every minimisation problem. **Minimise $$ f(x)=(x-7)^2 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [2, 4, 8, 12]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 4 of children [2, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Random draws [0.1, 0.4, 0.65, 0.9] | | 2 | Random draws [0.2, 0.4, 0.6, 0.95] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Roulette uses Fᵢ/ΣF. For minimisation here, use Fᵢ=1/(1+f(xᵢ)); do not directly favour large costs. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 2 | 0010 | (2 − 7)² = 25 | | 2 | 4 | 0100 | (4 − 7)² = 9 | | 3 | 8 | 1000 | (8 − 7)² = 1 | | 4 | 12 | 1100 | (12 − 7)² = 25 | ##### Step 2. Generation 1: calculate roulette intervals Fitness weights are nonnegative. For this minimisation example use F=1/(1+objective), so a smaller cost receives a larger weight. | Individual | Cost | Fitness calculation | | --- | --- | --- | | 1 | 25 | 1 / (1 + 25) = 1 / 26 ≈ 0.038461538 | | 2 | 9 | 1 / (1 + 9) = 1 / 10 ≈ 0.1 | | 3 | 1 | 1 / (1 + 1) = 1 / 2 ≈ 0.5 | | 4 | 25 | 1 / (1 + 25) = 1 / 26 ≈ 0.038461538 | $$ \sum F=0.038461538+0.1+0.5+0.038461538\approx0.676923077 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 0.038461538 ÷ 0.676923077 ≈ 0.056818182 | 0 + 0.056818182 ≈ 0.056818182 | | 2 | 0.1 ÷ 0.676923077 ≈ 0.147727273 | 0.056818182 + 0.147727273 ≈ 0.204545455 | | 3 | 0.5 ÷ 0.676923077 ≈ 0.738636364 | 0.204545455 + 0.738636364 ≈ 0.943181818 | | 4 | 0.038461538 ÷ 0.676923077 ≈ 0.056818182 | 0.943181818 + 0.056818182 ≈ 1 | ##### Step 3. Generation 1: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.1 | 0.056818182 ≤ 0.1 < 0.204545455 | 2 | 4 | | 0.4 | 0.204545455 ≤ 0.4 < 0.943181818 | 3 | 8 | | 0.65 | 0.204545455 ≤ 0.65 < 0.943181818 | 3 | 8 | | 0.9 | 0.204545455 ≤ 0.9 < 0.943181818 | 3 | 8 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 4. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 01 | 00 | 10 | 00 | 0100 | 1000 | | 3 & 4 | 10 | 00 | 10 | 00 | 1000 | 1000 | ##### Step 5. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 0100 | No mutation | 0100 | | 2 | 1000 | Flip bit 4 | 1001 | | 3 | 1000 | No mutation | 1000 | | 4 | 1000 | Flip bit 4 | 1001 | ##### Step 6. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 0100 | 0 + 4 + 0 + 0 = 4 | (4−7)² = 9 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | | 1000 | 8 + 0 + 0 + 0 = 8 | (8−7)² = 1 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | ##### Step 7. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{25+9+1+25}{4}=\frac{60}{4}=15 $$ $$ \overline f_{new}=\frac{9+4+1+4}{4}=\frac{18}{4}=4.5 $$ Best objective: 1 → 1. Lower is better. The next population is [4, 9, 8, 9]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 8. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 4 | 0100 | (4 − 7)² = 9 | | 2 | 9 | 1001 | (9 − 7)² = 4 | | 3 | 8 | 1000 | (8 − 7)² = 1 | | 4 | 9 | 1001 | (9 − 7)² = 4 | ##### Step 9. Generation 2: calculate roulette intervals Fitness weights are nonnegative. For this minimisation example use F=1/(1+objective), so a smaller cost receives a larger weight. | Individual | Cost | Fitness calculation | | --- | --- | --- | | 1 | 9 | 1 / (1 + 9) = 1 / 10 ≈ 0.1 | | 2 | 4 | 1 / (1 + 4) = 1 / 5 ≈ 0.2 | | 3 | 1 | 1 / (1 + 1) = 1 / 2 ≈ 0.5 | | 4 | 4 | 1 / (1 + 4) = 1 / 5 ≈ 0.2 | $$ \sum F=0.1+0.2+0.5+0.2\approx1 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 0.1 ÷ 1 ≈ 0.1 | 0 + 0.1 ≈ 0.1 | | 2 | 0.2 ÷ 1 ≈ 0.2 | 0.1 + 0.2 ≈ 0.3 | | 3 | 0.5 ÷ 1 ≈ 0.5 | 0.3 + 0.5 ≈ 0.8 | | 4 | 0.2 ÷ 1 ≈ 0.2 | 0.8 + 0.2 ≈ 1 | ##### Step 10. Generation 2: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.2 | 0.1 ≤ 0.2 < 0.3 | 2 | 9 | | 0.4 | 0.3 ≤ 0.4 < 0.8 | 3 | 8 | | 0.6 | 0.3 ≤ 0.6 < 0.8 | 3 | 8 | | 0.95 | 0.8 ≤ 0.95 < 1 | 4 | 9 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 11. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 10 | 01 | 10 | 00 | 1000 | 1001 | | 3 & 4 | 10 | 00 | 10 | 01 | 1001 | 1000 | ##### Step 12. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1000 | No mutation | 1000 | | 2 | 1001 | Flip bit 4 | 1000 | | 3 | 1001 | No mutation | 1001 | | 4 | 1000 | Flip bit 4 | 1001 | ##### Step 13. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1000 | 8 + 0 + 0 + 0 = 8 | (8−7)² = 1 | | 1000 | 8 + 0 + 0 + 0 = 8 | (8−7)² = 1 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | ##### Step 14. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{9+4+1+4}{4}=\frac{18}{4}=4.5 $$ $$ \overline f_{new}=\frac{1+1+4+4}{4}=\frac{10}{4}=2.5 $$ Best objective: 1 → 1. Lower is better. The next population is [8, 8, 9, 9]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [8, 8, 9, 9]. Best final objective = 1 at x = [8]. --- ## M2 · Swarm intelligence, particles, velocity, pbest and gbest Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--meaning/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/pso-core/meaning First understand position, velocity and memory; then solve complete two-iteration minimisation examples. ## Swarm intelligence **Swarm intelligence** is collective problem-solving behaviour that emerges from interactions among many relatively simple agents and their environment. It is inspired by systems such as bird flocks, fish schools and ant colonies. **Particle swarm optimization (PSO)** is a population-based search method inspired by social movement. Each particle represents a candidate solution. It moves through the search space using a velocity influenced by its own best previous position and the best position known to the swarm. ## Translate every PSO word into a concrete object | Term | Meaning | Example | | --- | --- | --- | | Particle | One candidate solution | Particle A | | Position x | The candidate’s coordinates | xA = 3 | | Objective f(x) | The problem’s score for that candidate | f(3) = 3² = 9 | | Velocity v | Proposed change of position | vA = −1 means move left by 1 | | pbest | Best position that this particle has visited | A once visited x=1 with score 1 | | gbest | Best position found by any particle | Among all personal bests, x=0.5 scores 0.25 | Swipe across the diagram to read its labels ↔ ![A best memory changes only when the new score improves.](https://deep-learning-study-room.vercel.app/is2/fig/v2_pso_memory.fcc9783a95.svg) Diagram: A best memory changes only when the new score improves.. Labels: Best memory survives a worse move | Visit 1 | Position x = 3 | Score x² = 9 | Remember pbest = 3 | Visit 2 | Position x = 1 | Score x² = 1 | Remember pbest = 1 | Visit 3 | Position x = 2 | Score x² = 4 | Remember pbest = 1 | Minimise x²: after visiting x = 2, the best past position is still x = 1. | Current position and remembered best position are separate records. A best memory changes only when the new score improves. ## Current position is not necessarily personal best For minimising x², suppose a particle visits x=3, then x=1, then x=2. Its current position is 2, but its personal best stays 1 because 1² is less than 2². The particle remembers a better past result even after a bad move. **A particle visits x=3, x=1, x=2 in that order while minimising f(x)=x². Find pbest after each visit.** [2 marks · Practice] #### Given Positions in chronological order: 3 → 1 → 2. #### To find The best-so-far position and score. #### Formula and why it applies Keep the location with the lowest objective encountered so far. #### Solution ##### Step 1. First visit $$ f(3)=9 $$ No history: pbest=3, best score=9. ##### Step 2. Second visit $$ f(1)=1<9 $$ Replace pbest by 1. ##### Step 3. Third visit $$ f(2)=4>1 $$ Keep pbest=1 even though the current position is 2. **Final answer.** Final current position=2; pbest=1; personal-best score=1. **Define swarm intelligence and briefly explain one nature-inspired algorithm.** [2 marks · Practice] #### Answer Swarm intelligence is collective problem-solving that emerges from interactions among simple agents. Particle swarm optimization is inspired by flocking behaviour. Each particle is a candidate solution with a position and velocity. It remembers its own best position and uses the swarm’s best position to guide movement. Repeated evaluation and movement search for a low-cost or high-fitness solution. **Yaad rakho:** Position abhi kahan ho; pbest apna best past; gbest sabka best past. --- ## M2 · Read the PSO equation and work three individual updates Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--equation/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/pso-core/equation First understand position, velocity and memory; then solve complete two-iteration minimisation examples. ## First read the update in words **New velocity = retained old velocity + pull toward personal best + pull toward the group best.** After finding that velocity, add it to the old position. You are calculating a movement, then applying it. Swipe across the diagram to read its labels ↔ ![A negative difference pulls toward a best position on the left.](https://deep-learning-study-room.vercel.app/is2/fig/depth_pso.88e9c2a672.svg) Diagram: A negative difference pulls toward a best position on the left.. Labels: Read the direction before multiplying the pull | Best = 1 | Current = 4 | difference = 1 − 4 = −3 | coefficient 0.5 → pull 0.5 × (−3) = −1.5 (left) A negative difference pulls toward a best position on the left. Example: current x=4 and personal best=1 gives the difference 1−4=−3. A coefficient of .5 gives personal pull .5×(−3)=−1.5. The negative sign says “move left”; it does not mean a negative probability. If current x=−2 and best=1, the difference is 1−(−2)=1+2=3, which pulls right. In vi,jt, i chooses the particle, j chooses its coordinate, and t chooses the recorded iteration. For particle B, coordinate 2, iteration 0, look at row B and the second velocity entry in the initial table. The letters identify a stored number; they are not extra multiplication factors. ## The velocity equation is three contributions added together $$ v_{i,j}^{t+1}=w v_{i,j}^{t}+c_1r_1(pbest_{i,j}^{t}-x_{i,j}^{t})+c_2r_2(gbest_j^{t}-x_{i,j}^{t}) $$ $$ x_{i,j}^{t+1}=x_{i,j}^{t}+v_{i,j}^{t+1} $$ | Symbol | Meaning | | --- | --- | | i | Which particle | | j | Which coordinate of that particle | | t / t+1 | Before / after this iteration; these are labels, not powers | | w | Inertia weight: how much old velocity is kept | | c₁r₁ | Combined strength of the pull to personal best | | c₂r₂ | Combined strength of the pull to global best | | r₁,r₂ | Random factors, normally in [0,1]; use the values supplied in an exam | A best position to the left gives a negative difference and pulls left; a best position to the right gives a positive difference and pulls right. The multiplication sets the strength of each pull. **For one coordinate, x=4, v=0, pbest=4, gbest=2. With w=0.5, c₁=c₂=1 and r₁=r₂=0.5, find the next velocity and position.** [3 marks · Practice] #### Given x=4, v=0, pbest=4, gbest=2; w=.5; c₁r₁=.5; c₂r₂=.5. #### To find New velocity followed by new position. #### Formula and why it applies Use the three-term velocity equation, then add the new velocity to x. $$ v_{new}=0.5v+0.5(pbest-x)+0.5(gbest-x) $$ #### Solution ##### Step 1. Inertia $$ 0.5\times0=0 $$ ##### Step 2. Personal pull $$ 0.5(4-(4))=0 $$ ##### Step 3. Social pull $$ 0.5(2-(4))=-1 $$ ##### Step 4. New velocity $$ v_{new}=0+0+(-1)=-1 $$ ##### Step 5. New position $$ x_{new}=4+(-1)=3 $$ **Final answer.** vnew=-1; xnew=3. **For one coordinate, x=4, v=-2, pbest=3, gbest=1. With w=0.5, c₁=c₂=1 and r₁=r₂=0.5, find the next velocity and position.** [3 marks · Practice] #### Given x=4, v=-2, pbest=3, gbest=1; w=.5; c₁r₁=.5; c₂r₂=.5. #### To find New velocity followed by new position. #### Formula and why it applies Use the three-term velocity equation, then add the new velocity to x. $$ v_{new}=0.5v+0.5(pbest-x)+0.5(gbest-x) $$ #### Solution ##### Step 1. Inertia $$ 0.5\times(-2)=-1 $$ ##### Step 2. Personal pull $$ 0.5(3-(4))=-0.5 $$ ##### Step 3. Social pull $$ 0.5(1-(4))=-1.5 $$ ##### Step 4. New velocity $$ v_{new}=(-1)+(-0.5)+(-1.5)=-3 $$ ##### Step 5. New position $$ x_{new}=4+(-3)=1 $$ **Final answer.** vnew=-3; xnew=1. **For one coordinate, x=-2, v=1, pbest=-1, gbest=1. With w=0.5, c₁=c₂=1 and r₁=r₂=0.5, find the next velocity and position.** [3 marks · Practice] #### Given x=-2, v=1, pbest=-1, gbest=1; w=.5; c₁r₁=.5; c₂r₂=.5. #### To find New velocity followed by new position. #### Formula and why it applies Use the three-term velocity equation, then add the new velocity to x. $$ v_{new}=0.5v+0.5(pbest-x)+0.5(gbest-x) $$ #### Solution ##### Step 1. Inertia $$ 0.5\times1=0.5 $$ ##### Step 2. Personal pull $$ 0.5(-1-(-2))=0.5 $$ ##### Step 3. Social pull $$ 0.5(1-(-2))=1.5 $$ ##### Step 4. New velocity $$ v_{new}=0.5+0.5+1.5=2.5 $$ ##### Step 5. New position $$ x_{new}=(-2)+2.5=0.5 $$ **Final answer.** vnew=2.5; xnew=0.5. **First-iteration shortcut:** if pbest was just initialised to x, the personal difference is zero. This follows from the initialisation; it is not a permanent property of PSO. A gbest particle can still move because its inertia term can be nonzero. **Yaad rakho:** Old speed ka hissa + apne best ki pull + group best ki pull; phir new speed ko position mein add. --- ## M2 · Two particles, two iterations: start with the simplest minimisation Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--two-particles/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/pso-core/two-particles First understand position, velocity and memory; then solve complete two-iteration minimisation examples. A synchronous iteration is a class photograph of the old swarm: every particle uses that same snapshot of gbest. First calculate all new positions, then compare all new scores and update memories. If particle A discovers a better position halfway through the calculations, B still uses the previous snapshot during this iteration. B can use the improved gbest in the next iteration. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. **Minimise $$ f(x)=x^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (-2) | (0) | | B | (4) | (0) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(-2)=(-2)^2=4=4 $$ **Particle B** $$ f(4)=4^2=16=16 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(-2) $$ Its score is 4. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (-2). pbest values are A: (-2), B: (4). ##### Step 4. Iteration 1: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=-2-(-2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-2-(-2)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=(-2)+0=-2 $$ ##### Step 5. Iteration 1: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=4-(4)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-2-(4)=-6 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-6)=-3 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-3)=-3 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=4+(-3)=1 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(-2)=(-2)^2=4=4 $$ **Particle B** $$ f(1)=1^2=1=1 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 4 | 4 | Keep old: tie | (-2) | | B | 16 | 1 | Replace: new score is lower | (1) | Compare the retained pbest scores. gbest = **(1)**, with objective **1**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (-2) | (0) | (-2) | | B | (1) | (-3) | (1) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (1). pbest values are A: (-2), B: (1). ##### Step 10. Iteration 2: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=-2-(-2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(-2)=3 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(3)=1.5 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+1.5=1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=(-2)+1.5=-0.5 $$ ##### Step 11. Iteration 2: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-3)=-1.5 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=(-1.5)+0+0=-1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1+(-1.5)=-0.5 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(-0.5)=(-0.5)^2=0.25=0.25 $$ **Particle B** $$ f(-0.5)=(-0.5)^2=0.25=0.25 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 4 | 0.25 | Replace: new score is lower | (-0.5) | | B | 1 | 0.25 | Replace: new score is lower | (-0.5) | Compare the retained pbest scores. gbest = **(-0.5)**, with objective **0.25**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (-0.5) | (1.5) | (-0.5) | | B | (-0.5) | (-1.5) | (-0.5) | **Final answer.** After iteration 2, gbest = (-0.5), objective = 0.25. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. --- ## M2 · A worse move: pbest stays in the past Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--memory-matters/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/pso-core/memory-matters First understand position, velocity and memory; then solve complete two-iteration minimisation examples. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. **Minimise $$ f(x)=x^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (1) | (4) | | B | (3) | (-1) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(1)=1^2=1=1 $$ **Particle B** $$ f(3)=3^2=9=9 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(1) $$ Its score is 1. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (1). pbest values are A: (1), B: (3). ##### Step 4. Iteration 1: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times4=2 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=2+0+0=2 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=1+2=3 $$ ##### Step 5. Iteration 1: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-1)=-0.5 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3-(3)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(3)=-2 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-2)=-1 $$ Add the three velocity contributions. $$ v^{(1)}=(-0.5)+0+(-1)=-1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=3+(-1.5)=1.5 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(3)=3^2=9=9 $$ **Particle B** $$ f(1.5)=1.5^2=2.25=2.25 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 1 | 9 | Keep old: new score is worse | (1) | | B | 9 | 2.25 | Replace: new score is lower | (1.5) | Compare the retained pbest scores. gbest = **(1)**, with objective **1**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (3) | (2) | (1) | | B | (1.5) | (-1.5) | (1.5) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (1). pbest values are A: (1), B: (1.5). ##### Step 10. Iteration 2: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times2=1 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(3)=-2 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(-2)=-1 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(3)=-2 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-2)=-1 $$ Add the three velocity contributions. $$ v^{(2)}=1+(-1)+(-1)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=3+(-1)=2 $$ ##### Step 11. Iteration 2: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-1.5)=-0.75 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1.5-(1.5)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1.5)=-0.5 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-0.5)=-0.25 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.75)+0+(-0.25)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1.5+(-1)=0.5 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(2)=2^2=4=4 $$ **Particle B** $$ f(0.5)=0.5^2=0.25=0.25 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 1 | 4 | Keep old: new score is worse | (1) | | B | 2.25 | 0.25 | Replace: new score is lower | (0.5) | Compare the retained pbest scores. gbest = **(0.5)**, with objective **0.25**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (2) | (-1) | (1) | | B | (0.5) | (-1) | (0.5) | **Final answer.** After iteration 2, gbest = (0.5), objective = 0.25. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. **Yaad rakho:** Naya result kharab ho toh position toh badlegi, pbest nahi. --- ## M2 · Two-dimensional PSO: keep the coordinates separate Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--two-coordinates/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/pso-core/two-coordinates First understand position, velocity and memory; then solve complete two-iteration minimisation examples. A two-dimensional particle has two entries in its position and two in its velocity. Apply the scalar velocity calculation once to coordinate 1 and once to coordinate 2. Keep them in the same order to form the new pair. Only after both entries are ready do you insert the pair into f(x₁,x₂). Adding x₁ and x₂ into a single position would lose information. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. **Minimise $$ f(x)=x_1^2+x_2^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (1, 1) | (0, 0) | | B | (3, 2) | (0, 0) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(1, 1)=1^2+1^2=1+1=2 $$ **Particle B** $$ f(3, 2)=3^2+2^2=9+4=13 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(1, 1) $$ Its score is 2. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (1, 1). pbest values are A: (1, 1), B: (3, 2). ##### Step 4. Iteration 1: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=1+0=1 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=1+0=1 $$ ##### Step 5. Iteration 1: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3-(3)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(3)=-2 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-2)=-1 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-1)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=3+(-1)=2 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=2-(2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(2)=-1 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-1)=-0.5 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-0.5)=-0.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=2+(-0.5)=1.5 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(1, 1)=1^2+1^2=1+1=2 $$ **Particle B** $$ f(2, 1.5)=2^2+1.5^2=4+2.25=6.25 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 2 | 2 | Keep old: tie | (1, 1) | | B | 13 | 6.25 | Replace: new score is lower | (2, 1.5) | Compare the retained pbest scores. gbest = **(1, 1)**, with objective **2**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (1, 1) | (0, 0) | (1, 1) | | B | (2, 1.5) | (-1, -0.5) | (2, 1.5) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (1, 1). pbest values are A: (1, 1), B: (2, 1.5). ##### Step 10. Iteration 2: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1+0=1 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1+0=1 $$ ##### Step 11. Iteration 2: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-1)=-0.5 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=2-(2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(2)=-1 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-1)=-0.5 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.5)+0+(-0.5)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=2+(-1)=1 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-0.5)=-0.25 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1.5-(1.5)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1.5)=-0.5 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-0.5)=-0.25 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.25)+0+(-0.25)=-0.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1.5+(-0.5)=1 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(1, 1)=1^2+1^2=1+1=2 $$ **Particle B** $$ f(1, 1)=1^2+1^2=1+1=2 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 2 | 2 | Keep old: tie | (1, 1) | | B | 6.25 | 2 | Replace: new score is lower | (1, 1) | Compare the retained pbest scores. gbest = **(1, 1)**, with objective **2**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (1, 1) | (0, 0) | (1, 1) | | B | (1, 1) | (-1, -0.5) | (1, 1) | **Final answer.** After iteration 2, gbest = (1, 1), objective = 2. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. --- ## M2 · Class PSO photograph: all three particles through iteration 2 Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--class-three-particles/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/pso-core/class-three-particles First understand position, velocity and memory; then solve complete two-iteration minimisation examples. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. The positions, velocities and coefficients are transcribed from the class photo. We use f(x₁,x₂)=x₁²+x₂²−x₁x₂, minimisation, initial pbest=x, fixed supplied random factors and synchronous updates. The photo does not itself specify all these conventions; they are stated here to make the two-iteration extension unambiguous. **Minimise $$ f(x)=x_1^2+x_2^2-x_1x_2 $$ using 3 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [10 marks · Class data; two-iteration minimisation extension] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (3.14, 7.21) | (0.1, 0.3) | | B | (4.15, 7.31) | (0.5, 0.1) | | C | (6.17, 8.27) | (0.2, 0) | w=0.5, c₁=1.2, c₂=1.2, r₁=0.5, r₂=0.1. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(3.14, 7.21)=3.14^2+7.21^2-(3.14)(7.21) $$ $$ =9.8596+51.9841-22.6394=39.2043 $$ **Particle B** $$ f(4.15, 7.31)=4.15^2+7.31^2-(4.15)(7.31) $$ $$ =17.2225+53.4361-30.3365=40.3221 $$ **Particle C** $$ f(6.17, 8.27)=6.17^2+8.27^2-(6.17)(8.27) $$ $$ =38.0689+68.3929-51.0259=55.4359 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(3.14, 7.21) $$ Its score is 39.2043. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (3.14, 7.21). pbest values are A: (3.14, 7.21), B: (4.15, 7.31), C: (6.17, 8.27). ##### Step 4. Iteration 1: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.1=0.05 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3.14-(3.14)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(3.14)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0.05+0+0=0.05 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=3.14+0.05=3.19 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.3=0.15 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.21-(7.21)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.21)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0.15+0+0=0.15 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=7.21+0.15=7.36 $$ ##### Step 5. Iteration 1: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.5=0.25 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=4.15-(4.15)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(4.15)=-1.01 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-1.01)=-0.1212 $$ Add the three velocity contributions. $$ v^{(1)}=0.25+0+(-0.1212)=0.1288 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=4.15+0.1288=4.2788 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.1=0.05 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.31-(7.31)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.31)=-0.1 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.1)=-0.012 $$ Add the three velocity contributions. $$ v^{(1)}=0.05+0+(-0.012)=0.038 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=7.31+0.038=7.348 $$ ##### Step 6. Iteration 1: update particle C **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.2=0.1 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=6.17-(6.17)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(6.17)=-3.03 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-3.03)=-0.3636 $$ Add the three velocity contributions. $$ v^{(1)}=0.1+0+(-0.3636)=-0.2636 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=6.17+(-0.2636)=5.9064 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=8.27-(8.27)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(8.27)=-1.06 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-1.06)=-0.1272 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-0.1272)=-0.1272 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=8.27+(-0.1272)=8.1428 $$ ##### Step 7. Iteration 1: evaluate the new positions **Particle A** $$ f(3.19, 7.36)=3.19^2+7.36^2-(3.19)(7.36) $$ $$ =10.1761+54.1696-23.4784=40.8673 $$ **Particle B** $$ f(4.2788, 7.348)=4.2788^2+7.348^2-(4.2788)(7.348) $$ $$ =18.30812944+53.993104-31.4406224=40.86061104 $$ **Particle C** $$ f(5.9064, 8.1428)=5.9064^2+8.1428^2-(5.9064)(8.1428) $$ $$ =34.88556096+66.30519184-48.09463392=53.09611888 $$ ##### Step 8. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 39.2043 | 40.8673 | Keep old: new score is worse | (3.14, 7.21) | | B | 40.3221 | 40.86061104 | Keep old: new score is worse | (4.15, 7.31) | | C | 55.4359 | 53.09611888 | Replace: new score is lower | (5.9064, 8.1428) | Compare the retained pbest scores. gbest = **(3.14, 7.21)**, with objective **39.2043**. ##### Step 9. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (3.19, 7.36) | (0.05, 0.15) | (3.14, 7.21) | | B | (4.2788, 7.348) | (0.1288, 0.038) | (4.15, 7.31) | | C | (5.9064, 8.1428) | (-0.2636, -0.1272) | (5.9064, 8.1428) | ##### Step 10. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (3.14, 7.21). pbest values are A: (3.14, 7.21), B: (4.15, 7.31), C: (5.9064, 8.1428). ##### Step 11. Iteration 2: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.05=0.025 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3.14-(3.19)=-0.05 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.05)=-0.03 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(3.19)=-0.05 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.05)=-0.006 $$ Add the three velocity contributions. $$ v^{(2)}=0.025+(-0.03)+(-0.006)=-0.011 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=3.19+(-0.011)=3.179 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.15=0.075 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.21-(7.36)=-0.15 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.15)=-0.09 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.36)=-0.15 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.15)=-0.018 $$ Add the three velocity contributions. $$ v^{(2)}=0.075+(-0.09)+(-0.018)=-0.033 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=7.36+(-0.033)=7.327 $$ ##### Step 12. Iteration 2: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.1288=0.0644 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=4.15-(4.2788)=-0.1288 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.1288)=-0.07728 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(4.2788)=-1.1388 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-1.1388)=-0.136656 $$ Add the three velocity contributions. $$ v^{(2)}=0.0644+(-0.07728)+(-0.136656)=-0.149536 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=4.2788+(-0.149536)=4.129264 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.038=0.019 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.31-(7.348)=-0.038 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.038)=-0.0228 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.348)=-0.138 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.138)=-0.01656 $$ Add the three velocity contributions. $$ v^{(2)}=0.019+(-0.0228)+(-0.01656)=-0.02036 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=7.348+(-0.02036)=7.32764 $$ ##### Step 13. Iteration 2: update particle C **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-0.2636)=-0.1318 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=5.9064-(5.9064)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(5.9064)=-2.7664 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-2.7664)=-0.331968 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.1318)+0+(-0.331968)=-0.463768 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=5.9064+(-0.463768)=5.442632 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-0.1272)=-0.0636 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=8.1428-(8.1428)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(8.1428)=-0.9328 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.9328)=-0.111936 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.0636)+0+(-0.111936)=-0.175536 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=8.1428+(-0.175536)=7.967264 $$ ##### Step 14. Iteration 2: evaluate the new positions **Particle A** $$ f(3.179, 7.327)=3.179^2+7.327^2-(3.179)(7.327) $$ $$ =10.106041+53.684929-23.292533=40.498437 $$ **Particle B** $$ f(4.129264, 7.32764)=4.129264^2+7.32764^2-(4.129264)(7.32764) $$ $$ =17.050821182+53.69430797-30.257760057=40.487369094 $$ **Particle C** $$ f(5.442632, 7.967264)=5.442632^2+7.967264^2-(5.442632)(7.967264) $$ $$ =29.622243087+63.477295646-43.362885999=49.736652734 $$ ##### Step 15. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 39.2043 | 40.498437 | Keep old: new score is worse | (3.14, 7.21) | | B | 40.3221 | 40.487369094 | Keep old: new score is worse | (4.15, 7.31) | | C | 53.09611888 | 49.736652734 | Replace: new score is lower | (5.442632, 7.967264) | Compare the retained pbest scores. gbest = **(3.14, 7.21)**, with objective **39.2043**. ##### Step 16. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (3.179, 7.327) | (-0.011, -0.033) | (3.14, 7.21) | | B | (4.129264, 7.32764) | (-0.149536, -0.02036) | (4.15, 7.31) | | C | (5.442632, 7.967264) | (-0.463768, -0.175536) | (5.442632, 7.967264) | **Final answer.** After iteration 2, gbest = (3.14, 7.21), objective = 39.2043. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. Swipe across the diagram to read its labels ↔ ![Motion alone does not imply improvement. Evaluate the objective at the new location before updating a best memory.](https://deep-learning-study-room.vercel.app/is2/fig/v2_pso_class_paths.961831cda1.svg) Diagram: Motion alone does not imply improvement. Evaluate the objective at the new location before updating a best memory.. Labels: 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | 8 | 8.5 | Class PSO: positions after each iteration | 0 = initial, 1 = iteration 1, 2 = iteration 2 | A0 | A1 | A2 | B0 | B1 | B2 | C0 | C1 | C2 | First coordinate x₁ | The best-so-far location remains A0 = (3.14, 7.21) after both iterations. Motion alone does not imply improvement. Evaluate the objective at the new location before updating a best memory. --- ## M2 · PSO algorithm, architecture, advantages and limitations Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--theory/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/pso-core/theory First understand position, velocity and memory; then solve complete two-iteration minimisation examples. ## Algorithm you can write in an exam 1. Define the objective and whether to minimise or maximise it. Initialise particle positions and velocities. 2. Evaluate every particle. Set each pbest to its initial position and gbest to the best among them. 3. Use the previous state to calculate each new velocity. 4. Update each position using that new velocity. Apply bounds or a velocity limit only according to the stated policy. 5. Evaluate the new positions. Update a pbest only when the new objective is better; choose gbest from the personal bests. 6. Repeat until the iteration budget or another stopping criterion is reached. Initialise → Evaluate + memories → Velocity → Position → Evaluate + update memoriesrepeat until stopping | Advantages | Limitations | | --- | --- | | Simple position/velocity updates; does not require objective derivatives | Can converge prematurely to a local region | | Uses both individual and shared experience | Results depend on coefficients, initialisation and random factors | | Useful for many continuous optimisation problems | Constraints and discrete encodings need suitable handling | | Keeps best-so-far memories | Does not guarantee the global optimum in a finite run | **Explain one application of PSO.** [3 marks · Practice] #### Answer In controller tuning, a particle can represent the gain values of a controller. The objective is a performance cost, such as tracking error plus an overshoot penalty. Each proposed gain vector is evaluated in the model. PSO updates candidate gains using individual and swarm bests. The best feasible gains found are selected and then validated on the actual system. A lower training cost alone does not prove safe real-world behaviour. | Exam situation | Action | | --- | --- | | Minimisation | Lower f is better | | Maximisation | Higher f is better; do not reverse velocity differences | | Tie in best score | Keep the previous best unless a tie rule is supplied | | Different random factors each iteration | Read the relevant row; do not reuse old factors | | Synchronous update | Freeze gbest for all particles within one iteration | | Position/velocity outside a limit | Use the limit-handling rule explicitly supplied; state an assumption if absent | **Yaad rakho:** Velocity → position → score → pbest → gbest. Ye order har iteration mein same. --- ## M2 · Cartesian product: every cell explained Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-relations--cartesian/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-relations/cartesian Longer Unit 2 numerical practice, also useful for TEE. ## How a row membership and a column membership meet Let A₁=.6 and A₂=.2 label two rows. Let B₁=.3, B₂=.8 and B₃=.5 label three columns. Cell R₂₃ belongs to row 2 and column 3, so use A₂=.2 and B₃=.5. Their min is .2. You do not multiply 2×3; those subscripts only locate the cell. $$ R_{23}=\min(A_2,B_3)=\min(0.2,0.5)=0.2 $$ There are two choices of row and three of column, so fill six cells in a 2×3 rectangle. Keep the row and column labels when reporting a relation. ## What is a fuzzy relation? A fuzzy set gives a membership to one element. A **fuzzy relation** gives a degree to an ordered pair. A matrix is a convenient table: rows name the first element and columns name the second. For the standard min-based Cartesian product R=A×B, put the A memberships down the rows and the B memberships across the columns. Each cell is the smaller of its row and column memberships. $$ R_{ij}=\min(\mu_A(a_i),\mu_B(b_j)) $$ **Find the fuzzy Cartesian product A×B for memberships A=[0.6, 0.2] and B=[0.3, 0.8, 0.5].** [3 marks · Practice] #### Given Ordered A memberships [0.6, 0.2]; B memberships [0.3, 0.8, 0.5]. #### To find A 2×3 relation matrix. #### Formula and why it applies Every row corresponds to an A element and every column to a B element; apply min in every cell. $$ R_{ij}=\min(A_i,B_j) $$ #### Solution ##### Step 1. Determine dimensions 2 row elements × 3 column elements. ##### Step 2. Compute each cell | A ↓ / B → | 0.3 | 0.8 | 0.5 | | --- | --- | --- | --- | | 0.6 | min(0.6,0.3) = 0.3 | min(0.6,0.8) = 0.6 | min(0.6,0.5) = 0.5 | | 0.2 | min(0.2,0.3) = 0.2 | min(0.2,0.8) = 0.2 | min(0.2,0.5) = 0.2 | ##### Step 3. Write the matrix $$ R=\begin{bmatrix}0.3&0.6&0.5\\0.2&0.2&0.2\end{bmatrix} $$ **Final answer.** R is the matrix in Step 3. **Find the fuzzy Cartesian product A×B for memberships A=[1, 0.4] and B=[0.7, 0].** [3 marks · Practice] #### Given Ordered A memberships [1, 0.4]; B memberships [0.7, 0]. #### To find A 2×2 relation matrix. #### Formula and why it applies Every row corresponds to an A element and every column to a B element; apply min in every cell. $$ R_{ij}=\min(A_i,B_j) $$ #### Solution ##### Step 1. Determine dimensions 2 row elements × 2 column elements. ##### Step 2. Compute each cell | A ↓ / B → | 0.7 | 0 | | --- | --- | --- | | 1 | min(1,0.7) = 0.7 | min(1,0) = 0 | | 0.4 | min(0.4,0.7) = 0.4 | min(0.4,0) = 0 | ##### Step 3. Write the matrix $$ R=\begin{bmatrix}0.7&0\\0.4&0\end{bmatrix} $$ **Final answer.** R is the matrix in Step 3. **Find the fuzzy Cartesian product A×B for memberships A=[0.2, 0.9, 0.5] and B=[0.6, 0.1].** [3 marks · Practice] #### Given Ordered A memberships [0.2, 0.9, 0.5]; B memberships [0.6, 0.1]. #### To find A 3×2 relation matrix. #### Formula and why it applies Every row corresponds to an A element and every column to a B element; apply min in every cell. $$ R_{ij}=\min(A_i,B_j) $$ #### Solution ##### Step 1. Determine dimensions 3 row elements × 2 column elements. ##### Step 2. Compute each cell | A ↓ / B → | 0.6 | 0.1 | | --- | --- | --- | | 0.2 | min(0.2,0.6) = 0.2 | min(0.2,0.1) = 0.1 | | 0.9 | min(0.9,0.6) = 0.6 | min(0.9,0.1) = 0.1 | | 0.5 | min(0.5,0.6) = 0.5 | min(0.5,0.1) = 0.1 | ##### Step 3. Write the matrix $$ R=\begin{bmatrix}0.2&0.1\\0.6&0.1\\0.5&0.1\end{bmatrix} $$ **Final answer.** R is the matrix in Step 3. **Yaad rakho:** Row ka degree aur column ka degree—dono ka min us cell mein. --- ## M2 · Max–min composition: three complete matrix examples Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-relations--composition/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/read/fuzzy-relations/composition Longer Unit 2 numerical practice, also useful for TEE. ## One result cell combines alternative two-link paths For T₁₁, start at u₁ and end at w₁. If the middle domain has v₁ and v₂, there are two candidate paths. Their strengths come from matching row 1 of R with column 1 of S. | Path | First link from R | Second link from S | Path strength | | --- | --- | --- | --- | | u₁ → v₁ → w₁ | .2 | .5 | min(.2,.5)=.2 | | u₁ → v₂ → w₁ | .8 | .7 | min(.8,.7)=.7 | Choose the stronger path: max(.2,.7)=.7, so T₁₁=.7. The middle label j runs through v₁ and v₂. The labels i and k stay fixed at 1 while this one cell is calculated. Then move to the next result cell. **Yaad rakho:** Ek path mein dono links chahiye, isliye min. Alag paths mein best chalega, isliye max. ## Composition connects through a middle element Suppose R relates U to V and S relates V to W. To find how strongly u connects to w through each middle v, take min of the two links. Then choose the strongest such path using max. $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ We call this matrix operation R composed with S here. Some books write the symbolic order in reverse. State the formula and row/column labels, especially when a question says S∘R. Composition requires the intermediate domains and their ordering to match; having two 3×3 matrices alone does not prove their labels match. **Calculate the max–min composition using $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ .** [3 marks · Practice] #### Given $$ R=\begin{bmatrix}0.2&0.8\\0.6&0.3\end{bmatrix} $$ $$ S=\begin{bmatrix}0.5&0.9\\0.7&0.1\end{bmatrix} $$ #### To find Every entry of T. #### Formula and why it applies Pair each row entry with the corresponding column entry; min for a two-link path, max across the possible middle elements. This is not ordinary matrix multiplication. #### Solution ##### Step 1. Cell (1,1): row 1 of R with column 1 of S $$ T_{11}=\max\{\min(0.2,0.5),\min(0.8,0.7)\} $$ $$ =\max\{0.2,0.7\}=0.7 $$ ##### Step 2. Cell (1,2): row 1 of R with column 2 of S $$ T_{12}=\max\{\min(0.2,0.9),\min(0.8,0.1)\} $$ $$ =\max\{0.2,0.1\}=0.2 $$ ##### Step 3. Cell (2,1): row 2 of R with column 1 of S $$ T_{21}=\max\{\min(0.6,0.5),\min(0.3,0.7)\} $$ $$ =\max\{0.5,0.3\}=0.5 $$ ##### Step 4. Cell (2,2): row 2 of R with column 2 of S $$ T_{22}=\max\{\min(0.6,0.9),\min(0.3,0.1)\} $$ $$ =\max\{0.6,0.1\}=0.6 $$ **Final answer.** $$ T=\begin{bmatrix}0.7&0.2\\0.5&0.6\end{bmatrix} $$ . **Calculate the max–min composition using $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ .** [3 marks · Practice] #### Given $$ R=\begin{bmatrix}1&0\\0.4&0.7\end{bmatrix} $$ $$ S=\begin{bmatrix}0.2&0.6\\0.8&0.3\end{bmatrix} $$ #### To find Every entry of T. #### Formula and why it applies Pair each row entry with the corresponding column entry; min for a two-link path, max across the possible middle elements. This is not ordinary matrix multiplication. #### Solution ##### Step 1. Cell (1,1): row 1 of R with column 1 of S $$ T_{11}=\max\{\min(1,0.2),\min(0,0.8)\} $$ $$ =\max\{0.2,0\}=0.2 $$ ##### Step 2. Cell (1,2): row 1 of R with column 2 of S $$ T_{12}=\max\{\min(1,0.6),\min(0,0.3)\} $$ $$ =\max\{0.6,0\}=0.6 $$ ##### Step 3. Cell (2,1): row 2 of R with column 1 of S $$ T_{21}=\max\{\min(0.4,0.2),\min(0.7,0.8)\} $$ $$ =\max\{0.2,0.7\}=0.7 $$ ##### Step 4. Cell (2,2): row 2 of R with column 2 of S $$ T_{22}=\max\{\min(0.4,0.6),\min(0.7,0.3)\} $$ $$ =\max\{0.4,0.3\}=0.4 $$ **Final answer.** $$ T=\begin{bmatrix}0.2&0.6\\0.7&0.4\end{bmatrix} $$ . **Calculate the max–min composition using $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ .** [3 marks · Practice] #### Given $$ R=\begin{bmatrix}0.3&0.6&0.9\end{bmatrix} $$ $$ S=\begin{bmatrix}0.8&0.1\\0.2&0.7\\0.5&0.4\end{bmatrix} $$ #### To find Every entry of T. #### Formula and why it applies Pair each row entry with the corresponding column entry; min for a two-link path, max across the possible middle elements. This is not ordinary matrix multiplication. #### Solution ##### Step 1. Cell (1,1): row 1 of R with column 1 of S $$ T_{11}=\max\{\min(0.3,0.8),\min(0.6,0.2),\min(0.9,0.5)\} $$ $$ =\max\{0.3,0.2,0.5\}=0.5 $$ ##### Step 2. Cell (1,2): row 1 of R with column 2 of S $$ T_{12}=\max\{\min(0.3,0.1),\min(0.6,0.7),\min(0.9,0.4)\} $$ $$ =\max\{0.1,0.6,0.4\}=0.6 $$ **Final answer.** $$ T=\begin{bmatrix}0.5&0.6\end{bmatrix} $$ . **Yaad rakho:** Har path mein min, phir saare paths mein max. Multiply-and-add nahi. --- ## M2 · Fuzzy formulas, with meanings and worked-example links Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/formulas/m2-formulas-v2--fuzzy/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/formulas/m2-formulas-v2/fuzzy One place for the formulas, meanings, conditions and links to full worked examples. ## Fuzzy formulas ### Triangular membership $$ \mu(x)=\max\left(0,\min\left(\frac{x-a}{b-a},\frac{c-x}{c-b}\right)\right) $$ a=left zero, b=peak, c=right zero, with a0 sets the width. At x=c the exponent is zero and membership is 1. [Worked exponential calculations →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--smooth-memberships/) ### Generalised bell membership $$ \mu(x)=\frac{1}{1+|(x-c)/a|^{2b}} $$ a>0 controls width; b>0 controls steepness; c is the centre. First subtract c, divide by a, take absolute value, apply the power, add 1, then take its reciprocal. ### Alpha-cut $$ A_\alpha=\{x:\mu_A(x)\ge\alpha\} $$ α is the chosen membership threshold. For 0<α≤1 keep inputs meeting or exceeding it, including threshold-equality endpoints. [Discrete and triangular examples →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--properties/) --- ## M2 · GA and PSO formulas, symbols and memory rules Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/formulas/m2-formulas-v2--ga-pso/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/formulas/m2-formulas-v2/ga-pso One place for the formulas, meanings, conditions and links to full worked examples. ### Binary decoding $$ x=\sum_{i=1}^{n}b_i2^{n-i} $$ bᵢ is a bit; the maximum n-bit unsigned integer is 2ⁿ−1. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--encoding/) ### Roulette selection $$ P_i=F_i/\sum_jF_j\quad C_i=\sum_{j=1}^iP_j $$ Fitness weights must be nonnegative with a positive total. For minimisation use the specified transformation or selection method. C is cumulative probability. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--selection/) ### Population average $$ \bar f=\frac{f_1+\cdots+f_N}{N} $$ N is the number of individuals. Compare both best and average; mutation need not improve either. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--two-generations/) ### PSO velocity $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ w weights inertia; c₁r₁ weights personal pull; c₂r₂ weights social pull. Apply the same equation separately to each coordinate. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--equation/) ### PSO position $$ x^{t+1}=x^t+v^{t+1} $$ Use the new velocity, not the old one. Then reevaluate the objective. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--two-particles/) **Memory updates:** minimise → replace a pbest only for a lower score; maximise → only for a higher score. gbest is a position selected from the personal bests. Store the corresponding objective score alongside it. Keep previous best on ties unless instructed otherwise. **Yaad rakho:** Formula ke pehle objective direction aur old state likho. Iteration 2 mein fresh positions aur retained memories use karo. ## Encoding and minimisation conventions $$ x=L+\frac{d}{2^n-1}(U-L) $$ For the specified uniform real-range encoding: decode n bits into unsigned integer d first. L and U are the physical limits. [Endpoint examples →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--encoding/) $$ F=\frac{1}{1+f(x)} $$ This is the stated roulette transformation in our nonnegative-cost minimisation example. It is not required for every GA. It converts a smaller cost into a larger selection weight. [Complete minimisation example →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--minimisation/) --- ## M2 · Q1(c) Fuzzy set operations [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-final-2025--q1c/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-final-2025/q1c Longer past-paper questions on Fuzzy, GA and PSO. Original TEE marks retained; not the M2 paper pattern. **Question. Consider two fuzzy sets X = {(c1, 0.6), (c2, 0.8), (c3, 0.3), (c4, 0.5), (c5, 0.7)} and Y = {(c1, 0.4), (c2, 0.5), (c3, 0.9), (c4, 0.6), (c5, 0.2)}. Find 1) X ∪ Y 2) X ∩ Y 3) ¬X ∪ ¬Y 4) ¬X ∩ ¬Y.** [5 marks] ### Answer #### Given The aligned X and Y memberships printed in this question. #### To find The four requested membership lists in the original element order. #### Formula and why it applies Use max for OR, min for AND, and 1−μ for NOT. $$ \mu_{X\cup Y}=\max(\mu_X,\mu_Y),\quad\mu_{X\cap Y}=\min(\mu_X,\mu_Y) $$ Complement both operands before applying the operator in ¬X∪¬Y or ¬X∩¬Y. #### Solution ##### Step 1. Complements ¬X = {0.4, 0.2, 0.7, 0.5, 0.3}; ¬Y = {0.6, 0.5, 0.1, 0.4, 0.8}. ##### Step 2. Table | | c1 | c2 | c3 | c4 | c5 | | --- | --- | --- | --- | --- | --- | | X | 0.6 | 0.8 | 0.3 | 0.5 | 0.7 | | Y | 0.4 | 0.5 | 0.9 | 0.6 | 0.2 | | **X ∪ Y = max** | 0.6 | 0.8 | 0.9 | 0.6 | 0.7 | | **X ∩ Y = min** | 0.4 | 0.5 | 0.3 | 0.5 | 0.2 | | **¬X ∪ ¬Y** | 0.6 | 0.5 | 0.7 | 0.5 | 0.8 | | **¬X ∩ ¬Y** | 0.4 | 0.2 | 0.1 | 0.4 | 0.3 | ##### Step 3. Check with De Morgan's law ¬X ∪ ¬Y must equal 1 − (X ∩ Y) = {0.6, 0.5, 0.7, 0.5, 0.8} ✓, and ¬X ∩ ¬Y must equal 1 − (X ∪ Y) = {0.4, 0.2, 0.1, 0.4, 0.3} ✓. **Final answer.** X ∪ Y = {(c1,0.6),(c2,0.8),(c3,0.9),(c4,0.6),(c5,0.7)}; X ∩ Y = {(c1,0.4),(c2,0.5),(c3,0.3),(c4,0.5),(c5,0.2)}; ¬X ∪ ¬Y = {(c1,0.6),(c2,0.5),(c3,0.7),(c4,0.5),(c5,0.8)}; ¬X ∩ ¬Y = {(c1,0.4),(c2,0.2),(c3,0.1),(c4,0.4),(c5,0.3)}. Official marking: 1 + 1 + 1.5 + 1.5. ## Calculate every element separately ### Element c1 $$ \neg X:1-0.6=0.4\qquad\neg Y:1-0.4=0.6 $$ $$ X\cup Y:\max(0.6,0.4)=0.6 $$ $$ X\cap Y:\min(0.6,0.4)=0.4 $$ $$ \neg X\cup\neg Y:\max(0.4,0.6)=0.6 $$ $$ \neg X\cap\neg Y:\min(0.4,0.6)=0.4 $$ ### Element c2 $$ \neg X:1-0.8=0.2\qquad\neg Y:1-0.5=0.5 $$ $$ X\cup Y:\max(0.8,0.5)=0.8 $$ $$ X\cap Y:\min(0.8,0.5)=0.5 $$ $$ \neg X\cup\neg Y:\max(0.2,0.5)=0.5 $$ $$ \neg X\cap\neg Y:\min(0.2,0.5)=0.2 $$ ### Element c3 $$ \neg X:1-0.3=0.7\qquad\neg Y:1-0.9=0.1 $$ $$ X\cup Y:\max(0.3,0.9)=0.9 $$ $$ X\cap Y:\min(0.3,0.9)=0.3 $$ $$ \neg X\cup\neg Y:\max(0.7,0.1)=0.7 $$ $$ \neg X\cap\neg Y:\min(0.7,0.1)=0.1 $$ ### Element c4 $$ \neg X:1-0.5=0.5\qquad\neg Y:1-0.6=0.4 $$ $$ X\cup Y:\max(0.5,0.6)=0.6 $$ $$ X\cap Y:\min(0.5,0.6)=0.5 $$ $$ \neg X\cup\neg Y:\max(0.5,0.4)=0.5 $$ $$ \neg X\cap\neg Y:\min(0.5,0.4)=0.4 $$ ### Element c5 $$ \neg X:1-0.7=0.3\qquad\neg Y:1-0.2=0.8 $$ $$ X\cup Y:\max(0.7,0.2)=0.7 $$ $$ X\cap Y:\min(0.7,0.2)=0.2 $$ $$ \neg X\cup\neg Y:\max(0.3,0.8)=0.8 $$ $$ \neg X\cap\neg Y:\min(0.3,0.8)=0.3 $$ --- ## M2 · Q2(a) PSO, one iteration [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-final-2025--q2a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-final-2025/q2a Longer past-paper questions on Fuzzy, GA and PSO. Original TEE marks retained; not the M2 paper pattern. **Question. Maximize $$ F(x_1,x_2)=2x_1^2-3x_1x_2+x_2^2+5x_1+2x_2+7 $$ using Particle Swarm Optimization. $$ X=\begin{bmatrix}2.5&-3.0\\3.5&-1.5\end{bmatrix},\;V=\begin{bmatrix}0.2&0.3\\0.1&0.25\end{bmatrix} $$ . Take $$ W=0.9,\;c_1=c_2=1.5,\;r_1=0.5949,\;r_2=0.0855 $$ . Optimize and give the solution after 1 iteration showing the step-by-step process.** [10 marks] ### Answer #### Given Particle 1: $$ x=(2.5,\,-3.0) $$ , $$ v=(0.2,\,0.3) $$ . Particle 2: $$ x=(3.5,\,-1.5) $$ , $$ v=(0.1,\,0.25) $$ . $$ c_1r_1=1.5\times0.5949=0.89235,\qquad c_2r_2=1.5\times0.0855=0.12825 $$ #### To find Fitness, pbest, gbest; new velocities and positions; updated pbest and gbest. #### Formula and why it applies $$ v_{ij}\leftarrow wv_{ij}+c_1r_1(p^b_{ij}-x_{ij})+c_2r_2(p^g_j-x_{ij}),\qquad x_{ij}\leftarrow x_{ij}+v_{ij} $$ Each particle is pulled by its own memory and by the swarm's best; we maximise, so the larger $$ F $$ is better. #### Solution ##### Step 1. Fitness of particle 1 $$ F(2.5,-3)=2(6.25)-3(2.5)(-3)+9+5(2.5)+2(-3)+7=12.5+22.5+9+12.5-6+7=57.5 $$ ##### Step 2. Fitness of particle 2 $$ F(3.5,-1.5)=2(12.25)-3(3.5)(-1.5)+2.25+17.5-3+7=24.5+15.75+2.25+17.5-3+7=64 $$ ##### Step 3. pbest and gbest | Particle | Position | F | pbest | | --- | --- | --- | --- | | 1 | (2.5, −3.0) | 57.5 | (2.5, −3.0) | | 2 | (3.5, −1.5) | 64 | (3.5, −1.5) | No previous iteration, so pbest = current position. Largest fitness is 64, so **gbest = (3.5, −1.5)**. ##### Step 4. Velocity and position of particle 1 $$ v_{11}=0.9(0.2)+0.89235(2.5-2.5)+0.12825(3.5-2.5)=0.18+0+0.12825=0.30825 $$ $$ x_{11}=2.5+0.30825=2.80825 $$ $$ v_{12}=0.9(0.3)+0.89235(0)+0.12825(-1.5-(-3.0))=0.27+0.192375=0.462375 $$ $$ x_{12}=-3.0+0.462375=-2.537625 $$ ##### Step 5. Velocity and position of particle 2 (it is gbest, so both pull terms are 0) $$ v_{21}=0.9(0.1)+0+0=0.09,\qquad x_{21}=3.5+0.09=3.59 $$ $$ v_{22}=0.9(0.25)+0+0=0.225,\qquad x_{22}=-1.5+0.225=-1.275 $$ ##### Step 6. New fitness $$ F(2.80825,-2.537625)=15.772536+21.378856+6.439541+14.04125-5.07525+7=59.5569 $$ $$ F(3.59,-1.275)=25.7762+13.73175+1.625625+17.95-2.55+7=63.5336 $$ ##### Step 7. Update pbest and gbest | Particle | New position | New F | Old pbest F | pbest after iteration | | --- | --- | --- | --- | --- | | 1 | (2.80825, −2.537625) | 59.5569 | 57.5 | (2.80825, −2.537625) — improved | | 2 | (3.59, −1.275) | 63.5336 | 64 | (3.5, −1.5) — kept | The best pbest fitness is still 64, so gbest stays (3.5, −1.5). **Final answer.** $$ V=\begin{bmatrix}0.30825&0.462375\\0.09&0.225\end{bmatrix} $$ , $$ X=\begin{bmatrix}2.80825&-2.537625\\3.59&-1.275\end{bmatrix} $$ . After one iteration particle 1 improved to F = 59.5569; the best solution so far is gbest = (3.5, −1.5) with F = 64. Official key: fitness and gbest (5 marks), velocity and position updates (5 marks). The calculations above use the question data. The key's summary matrix writes 0.462 in place of 2.808 for $$ x_{11} $$ ; that is a copying slip. --- ## M2 · Q3(b) Fuzzy loan-approval system [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-final-2025--q3b/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-final-2025/q3b Longer past-paper questions on Fuzzy, GA and PSO. Original TEE marks retained; not the M2 paper pattern. **A bank uses fuzzy decision support with Credit Score and Income Stability, both on 1–100. Loan Approval is a score on 1–100 with labels Reject, Consider and Approve. Define triangular membership functions, construct nine rules, explain Mamdani inference and defuzzify for Credit Score=55 and Income Stability=35.** [10 marks] ### Answer #### Given | Variable | Crisp value | | --- | --- | | Credit Score | 55 | | Income Stability | 35 | The question asks us to design the membership functions. This solution chooses Low=Tri(1,25,50), Medium=Tri(25,50,75), High=Tri(50,75,100) for both inputs. Output labels use the same triangles in their respective order. These are explicit design choices, not additional printed givens. #### To find A complete stated design and its continuous Mamdani centroid. #### Formula and why it applies Use min for AND and implication, max for aggregation and the true area centroid of the resulting shape. A peak-only weighted average is a different defuzzifier. $$ z^*=\frac{\int z\mu_{out}(z)\,dz}{\int\mu_{out}(z)\,dz} $$ #### Solution ##### Step 1. Specify the membership functions | Input / output position | Triangle | Peak | | --- | --- | --- | | Low / Reject | (1,25,50) | 25 | | Medium / Consider | (25,50,75) | 50 | | High / Approve | (50,75,100) | 75 | Use 0 outside each triangle, the rising fraction before the peak and the falling fraction after it. This design is evaluated at the requested input; a deployed controller would also need coverage/fallback at all boundary values. ##### Step 2. Write all nine rules | Credit Score / Income Stability | Low | Medium | High | | --- | --- | --- | --- | | Low | Reject | Reject | Consider | | Medium | Reject | Consider | Approve | | High | Consider | Approve | Approve | Read each cell as an IF–AND–THEN rule. For example, the centre cell combines Medium with Medium. ##### Step 3. Fuzzify Credit Score = 55 $$ \mu_{Low}(55)=0 $$ $$ \mu_{Medium}(55)=\frac{75-55}{75-50}=0.8 $$ $$ \mu_{High}(55)=\frac{55-50}{75-50}=0.2 $$ ##### Step 4. Fuzzify Income Stability = 35 $$ \mu_{Low}(35)=\frac{50-35}{50-25}=0.6 $$ $$ \mu_{Medium}(35)=\frac{35-25}{50-25}=0.4 $$ $$ \mu_{High}(35)=0 $$ ##### Step 5. Evaluate all rules and aggregate matching consequents | Credit Score | Income Stability | Firing strength | Consequent | | --- | --- | --- | --- | | Low | Low | min(0,0.6) = 0 | Reject | | Low | Medium | min(0,0.4) = 0 | Reject | | Low | High | min(0,0) = 0 | Consider | | Medium | Low | min(0.8,0.6) = 0.6 | Reject | | Medium | Medium | min(0.8,0.4) = 0.4 | Consider | | Medium | High | min(0.8,0) = 0 | Approve | | High | Low | min(0.2,0.6) = 0.2 | Consider | | High | Medium | min(0.2,0.4) = 0.2 | Approve | | High | High | min(0.2,0) = 0 | Approve | Aggregated label strengths: Reject = 0.6, Consider = 0.4, Approve = 0.2. $$ \mu_{out}(z)=\max_r\min(\alpha_r,\mu_{C_r}(z)) $$ ##### Step 6. Draw the aggregated output Swipe across the diagram to read its labels ↔ ![The shaded upper envelope counts any overlap only once.](https://deep-learning-study-room.vercel.app/is2/fig/v2_loan_aggregate.2c5ffa32a7.svg) Diagram: The shaded upper envelope counts any overlap only once.. Labels: Chosen triangular output sets and their max-aggregate | 0 | 0.5 | 1 | 1 | 25 | 50 | 75 | 100 | μ | Output score | Reject | Consider | Approve | centroid ≈ 42.78 The shaded upper envelope counts any overlap only once. ##### Step 7. Calculate area and moment of each straight-line strip For a strip from a to b with endpoint heights u and v, its trapezoid area is A. Its first moment M is the integral of z times the line height. $$ A=(b-a)(u+v)/2 $$ $$ M=\frac{b-a}{6}\big[(2a+b)u+(a+2b)v\big] $$ | a | b | u | v | Area A | Moment M | | --- | --- | --- | --- | --- | --- | | 1 | 15.4 | 0 | 0.6 | 4.32 | 45.792 | | 15.4 | 25 | 0.6 | 0.6 | 5.76 | 116.352 | | 25 | 35 | 0.6 | 0.6 | 6 | 180 | | 35 | 40 | 0.6 | 0.4 | 2.5 | 93.333333333 | | 40 | 50 | 0.4 | 0.4 | 4 | 180 | | 50 | 55 | 0.4 | 0.4 | 2 | 105 | | 55 | 65 | 0.4 | 0.4 | 4 | 240 | | 65 | 70 | 0.4 | 0.2 | 1.5 | 100.833333333 | | 70 | 75 | 0.2 | 0.2 | 1 | 72.5 | | 75 | 95 | 0.2 | 0.2 | 4 | 340 | | 95 | 100 | 0.2 | 0 | 0.5 | 48.333333333 | ##### Step 8. Divide total moment by total area $$ \sum A=35.58 $$ $$ \sum M=1522.144 $$ $$ z^*=\frac{1522.144}{35.58}\approx42.780888139 $$ **Final answer.** For these stated triangles and rules, the centroid is 42.780888139. This is a fuzzy suitability score; the category memberships can be evaluated at this score. The model alone is not a complete loan-decision policy. Because the paper asks for a design, a different explicitly specified membership/rule system can yield a different numerical output. Explain the choices and use the chosen defuzzification method consistently; no particular examiner mark outcome is promised. ## How the centroid table’s breakpoints are obtained For each output triangle, solve “line height = clip height”. Reject is clipped at .6: rising endpoint 1+.6(25−1)=15.4 and falling endpoint 50−.6(50−25)=35. Consider is clipped at .4: endpoints 25+.4(50−25)=35 and 75−.4(75−50)=65. Approve is clipped at .2: endpoints 50+.2(75−50)=55 and 100−.2(100−75)=95. Now take the maximum of the three clipped curves. The declining Reject edge reaches the Consider height .4 at (50−z)/25=.4, hence z=40. The declining Consider edge reaches the Approve height .2 at (75−z)/25=.2, hence z=70. These crossover locations must be included when splitting the aggregate. Original peaks and other clip endpoints may add extra splits without changing the total area. **For the first aggregate strip from a=1 to b=15.4, endpoint heights are u=0 and v=0.6. Find its area and first moment.** [3 marks · Worked example] #### Given Strip endpoints a=1 and b=15.4; left height u=0 and right height v=0.6. This is a straight rising edge of the clipped Reject set. #### To find A and M for one table row. #### Formula and why it applies $$ A=(b-a)(u+v)/2 $$ $$ M=\frac{b-a}{6}[(2a+b)u+(a+2b)v] $$ A is trapezoid area. M weights every small area by its horizontal position; this formula integrates a straight line. #### Solution ##### Step 1. Width and height sum $$ b-a=15.4-1=14.4\qquad u+v=0.6 $$ ##### Step 2. Area $$ A=14.4\times0.6/2=8.64/2=4.32 $$ ##### Step 3. First bracket term $$ (2a+b)u=(2+15.4)\times0=0 $$ ##### Step 4. Second bracket term $$ (a+2b)v=(1+30.8)\times0.6=31.8\times0.6=19.08 $$ ##### Step 5. Moment $$ M=(14.4/6)(0+19.08)=2.4\times19.08=45.792 $$ **Final answer.** Area=4.32; moment=45.792. Repeat the same arithmetic for each row, then sum the areas and moments separately. ### Interpret the loan result using an explicit decision rule Choose the output label having the greatest membership at the centroid; this is an additional stated interpretation rule. At z=42.780888139: $$ \mu_{Reject}=(50-z)/25\approx0.288764474,\quad\mu_{Consider}=(z-25)/25\approx0.711235526,\quad\mu_{Approve}=0 $$ The resulting recommendation is **Consider / manual review**; this design does not recommend automatic approval. If a binary approve/reject answer is required, the question must supply a threshold or you must state the threshold you adopt. A different decision threshold is separate from defuzzification. --- ## M2 · Q4(a) Cartesian product and max–min composition [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-final-2025--q4a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-final-2025/q4a Longer past-paper questions on Fuzzy, GA and PSO. Original TEE marks retained; not the M2 paper pattern. **Question. Consider fuzzy sets A = {0.4/small + 0.3/medium + 0.1/large} and B = {0.2/slow + 0.7/moderate + 0.3/fast}. 1. Find R = A × B. 2. Given C = {0.3/small + 0.1/medium + 0.4/large}, find S = C × B. 3. Calculate the max–min composition S ∘ R.** [10 marks] ### Answer #### Given The fuzzy sets or relation matrices and their listed row/column order. #### To find The relation matrix/matrices and the composition specified in the question. #### Formula and why it applies For Cartesian products take min of row and column membership. For the ordered matrix composition shown here: $$ T_{ik}=\max_j\min(S_{ij},R_{jk}) $$ If max–product is asked instead, replace the inner min by multiplication. #### Solution ##### Step 1. R = A × B (rows small, medium, large; columns slow, moderate, fast) $$ R=\begin{bmatrix}\min(.4,.2)&\min(.4,.7)&\min(.4,.3)\\\min(.3,.2)&\min(.3,.7)&\min(.3,.3)\\\min(.1,.2)&\min(.1,.7)&\min(.1,.3)\end{bmatrix}=\begin{bmatrix}0.2&0.4&0.3\\0.2&0.3&0.3\\0.1&0.1&0.1\end{bmatrix} $$ ##### Step 2. S = C × B $$ S=\begin{bmatrix}0.2&0.3&0.3\\0.1&0.1&0.1\\0.2&0.4&0.3\end{bmatrix} $$ ##### Step 3. S ∘ R by max–min (row of S with column of R) The printed relation domains do not directly compose: both relate size to speed. For the numerical exercise, explicitly assume positional alignment and compute row of S with column of R. Equal dimensions alone are not a justification for semantic relation composition. $$ (1,1)=\max[\min(.2,.2),\min(.3,.2),\min(.3,.1)]=\max(.2,.2,.1)=0.2 $$ $$ (1,2)=\max[\min(.2,.4),\min(.3,.3),\min(.3,.1)]=\max(.2,.3,.1)=0.3 $$ $$ (1,3)=\max[\min(.2,.3),\min(.3,.3),\min(.3,.1)]=0.3 $$ Row 2 of S is all 0.1, so every entry of row 2 is 0.1. $$ (3,1)=\max[\min(.2,.2),\min(.4,.2),\min(.3,.1)]=0.2,\;(3,2)=\max[.2,.3,.1]=0.3,\;(3,3)=\max[.2,.3,.1]=0.3 $$ **Final answer.** $$ S\circ R=\begin{bmatrix}0.2&0.3&0.3\\0.1&0.1&0.1\\0.2&0.3&0.3\end{bmatrix} $$ . **Official key note.** The key first printed a wrong matrix with row 2 = (0.2, 0.3, 0.3), then corrected it by hand to (0.1, 0.1, 0.1). The corrected matrix above is right. Marking: R 3, S 3, composition 4. **A careful remark you can add.** Strictly, composition needs S: X → Y and R: Y → Z, while here both relate size to speed. The numerical matrix calculation above assumes positional identification of the three middle labels; that is a declared repair, not a valid size-to-speed relation composition without an extra mapping. ## Every cell without the row shortcuts Both printed relations run from size to speed; their middle labels do not match for genuine relation composition. The numerical working below follows the question’s intended row-of-S with column-of-R operation using explicitly assumed positional alignment of the three middle entries. ### R: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | R11 | 0.4 | 0.2 | 0.2 | | R12 | 0.4 | 0.7 | 0.4 | | R13 | 0.4 | 0.3 | 0.3 | | R21 | 0.3 | 0.2 | 0.2 | | R22 | 0.3 | 0.7 | 0.3 | | R23 | 0.3 | 0.3 | 0.3 | | R31 | 0.1 | 0.2 | 0.1 | | R32 | 0.1 | 0.7 | 0.1 | | R33 | 0.1 | 0.3 | 0.1 | ### S: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | S11 | 0.3 | 0.2 | 0.2 | | S12 | 0.3 | 0.7 | 0.3 | | S13 | 0.3 | 0.3 | 0.3 | | S21 | 0.1 | 0.2 | 0.1 | | S22 | 0.1 | 0.7 | 0.1 | | S23 | 0.1 | 0.3 | 0.1 | | S31 | 0.4 | 0.2 | 0.2 | | S32 | 0.4 | 0.7 | 0.4 | | S33 | 0.4 | 0.3 | 0.3 | **Result cell (1,1)** $$ \max\{\min(0.2,0.2),\min(0.3,0.2),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (1,2)** $$ \max\{\min(0.2,0.4),\min(0.3,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ **Result cell (1,3)** $$ \max\{\min(0.2,0.3),\min(0.3,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ **Result cell (2,1)** $$ \max\{\min(0.1,0.2),\min(0.1,0.2),\min(0.1,0.1)\} $$ $$ =\max\{0.1,0.1,0.1\}=0.1 $$ **Result cell (2,2)** $$ \max\{\min(0.1,0.4),\min(0.1,0.3),\min(0.1,0.1)\} $$ $$ =\max\{0.1,0.1,0.1\}=0.1 $$ **Result cell (2,3)** $$ \max\{\min(0.1,0.3),\min(0.1,0.3),\min(0.1,0.1)\} $$ $$ =\max\{0.1,0.1,0.1\}=0.1 $$ **Result cell (3,1)** $$ \max\{\min(0.2,0.2),\min(0.4,0.2),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (3,2)** $$ \max\{\min(0.2,0.4),\min(0.4,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ **Result cell (3,3)** $$ \max\{\min(0.2,0.3),\min(0.4,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ --- ## M2 · Q5(b) Genetic algorithm [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-final-2025--q5b/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-final-2025/q5b Longer past-paper questions on Fuzzy, GA and PSO. Original TEE marks retained; not the M2 paper pattern. **Implement a GA to maximise $$ f(x)=2x^3-5x^2+7x $$ for x in [0,15]. Take 4-bit representations of 13, 22, 18, 23. Crossover after the second bit and mutate by flipping any one bit. Produce two offspring and comment on fitness.** [10 marks] ### Answer **The printed data are inconsistent.** 22, 18 and 23 are outside 0–15 and cannot fit into four bits. Increasing the bit count would encode the numbers but would still violate the stated range. There is no single exact solution satisfying all printed conditions. The complete worked variant below explicitly changes the invalid parents to their last four bits (values modulo 16), producing [13,6,2,7]. This is one declared repair, not a silent interpretation of the printed question. If the instructor clarifies different corrected data, follow those data. #### Given Printed parents: [13, 22, 18, 23]. Worked variant: [13, 6, 2, 7] with 4 bits. For the unspecified selection step, select the two highest-fitness candidates. For the unspecified mutation location, flip the last bit of each child. Crossover is after bit 2. #### To find Two valid children under the declared assumptions and their fitness. #### Formula and why it applies Encoding must respect the candidate range. Crossover swaps tails; mutation flips the chosen bit. Fitness is evaluated after mutation. $$ f(x)=2x^3-5x^2+7x $$ #### Solution ##### Step 1. Encode and evaluate the declared parent population | x | Binary | Fitness substitution | | --- | --- | --- | | 13 | 1101 | 2(13³) − 5(13²) + 7(13) = 3640 | | 6 | 0110 | 2(6³) − 5(6²) + 7(6) = 294 | | 2 | 0010 | 2(2³) − 5(2²) + 7(2) = 10 | | 7 | 0111 | 2(7³) − 5(7²) + 7(7) = 490 | ##### Step 2. Select the two fittest under the stated assumption [13, 7] with fitness [3640, 490]. ##### Step 3. Cut after bit 2 and exchange tails | Parent 1 | Parent 2 | Child 1 | Child 2 | | --- | --- | --- | --- | | 11 | 01 | 01 | 11 | 1111 | 0101 | ##### Step 4. Flip the last bit of each child | Before | After | Decoded x | | --- | --- | --- | | 1111 | 1110 | 14 | | 0101 | 0100 | 4 | ##### Step 5. Evaluate both children | x | Fitness calculation | | --- | --- | | 14 | 2(2744) − 5(196) + 7(14) = 4606 | | 4 | 2(64) − 5(16) + 7(4) = 76 | ##### Step 6. Compare with the selected parents Best parent score = 3640; best child score = 4606. Parent average = 2065; child average = 2341. Both the best score and the average score increased in this worked variant. An individual child can still be worse than a parent; GA does not guarantee improvement after every random operator. **Final answer.** Under these declared repairs and operator choices, children are [14, 4] with fitness [4606, 76]. These are not unique answers to the inconsistent and under-specified original question. The printed wording about “two offspring” is not an explicit requirement for two complete generations. The M2 examples separately work two full generations because your class update asks for iteration 2. ## Evaluate the polynomial without skipping the powers Calculate x²=x×x, then x³=x²×x. Multiply each by its coefficient and combine with the written signs. A bit-string repair in the worked answer is an explicitly stated response to inconsistent printed givens; it is not a new instruction in the original paper. ### x=13 $$ x^2=13\times13=169\qquad x^3=169\times13=2197 $$ $$ 2x^3=4394\quad 5x^2=845\quad 7x=91 $$ $$ f(13)=4394-845+91=3640 $$ ### x=6 $$ x^2=6\times6=36\qquad x^3=36\times6=216 $$ $$ 2x^3=432\quad 5x^2=180\quad 7x=42 $$ $$ f(6)=432-180+42=294 $$ ### x=2 $$ x^2=2\times2=4\qquad x^3=4\times2=8 $$ $$ 2x^3=16\quad 5x^2=20\quad 7x=14 $$ $$ f(2)=16-20+14=10 $$ ### x=7 $$ x^2=7\times7=49\qquad x^3=49\times7=343 $$ $$ 2x^3=686\quad 5x^2=245\quad 7x=49 $$ $$ f(7)=686-245+49=490 $$ ### x=15 $$ x^2=15\times15=225\qquad x^3=225\times15=3375 $$ $$ 2x^3=6750\quad 5x^2=1125\quad 7x=105 $$ $$ f(15)=6750-1125+105=5730 $$ ### x=5 $$ x^2=5\times5=25\qquad x^3=25\times5=125 $$ $$ 2x^3=250\quad 5x^2=125\quad 7x=35 $$ $$ f(5)=250-125+35=160 $$ ### x=14 $$ x^2=14\times14=196\qquad x^3=196\times14=2744 $$ $$ 2x^3=5488\quad 5x^2=980\quad 7x=98 $$ $$ f(14)=5488-980+98=4606 $$ ### x=4 $$ x^2=4\times4=16\qquad x^3=16\times4=64 $$ $$ 2x^3=128\quad 5x^2=80\quad 7x=28 $$ $$ f(4)=128-80+28=76 $$ “Modulo 16” means keep the remainder after dividing by 16: 22=1×16+6, so its remainder is 6; 18=1×16+2; 23=1×16+7. This is only the declared repair used here, not an unspoken general encoding rule. --- ## M2 · Q6(a) Applications of fuzzy logic [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-final-2025--q6a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-final-2025/q6a Longer past-paper questions on Fuzzy, GA and PSO. Original TEE marks retained; not the M2 paper pattern. **List the applications of fuzzy logic. Explain any one application in detail.** [10 marks] ### Answer ### Definition Fuzzy logic is a multi-valued logic, introduced by Lotfi Zadeh (1965), in which truth values range over [0, 1]; it lets machines reason with vague, human-like terms such as “slightly dirty” or “very hot”. ### Applications | Area | Examples | | --- | --- | | Home appliances | Washing machines (wash time from dirt and load), air conditioners, refrigerators, rice cookers, vacuum cleaners | | Automotive | Automatic transmission, anti-lock braking (ABS), cruise control, engine control | | Transportation | Traffic-signal control, Sendai subway train control (smooth braking and acceleration) | | Consumer electronics | Camera autofocus and auto-exposure, image stabilisation | | Industrial control | Cement kilns, pH, temperature and distillation control in chemical plants | | Aerospace | Satellite and spacecraft attitude control | | Medicine | Diagnosis support, anaesthesia dosage control | | Finance and decisions | Loan approval, credit and fire-risk scoring, stock trading rules | | AI and robotics | Robot navigation, natural-language interfaces, expert systems | ### One application in detail: fuzzy washing machine 1. **Problem.** Wash time depends on how dirty and how greasy the clothes are. These are vague (“a bit dirty”), and a fixed timer wastes water and power. 2. **Inputs and outputs.** Sensors measure dirt (turbidity of water) and grease; the output is wash time. Each is described by fuzzy sets: dirt Small/Medium/High, grease Small/Medium/High, time Very Short … Very Long. 3. **Fuzzification.** Sensor readings are converted to membership degrees, e.g. dirt 60% → Medium 0.8, High 0.2. 4. **Rule base.** 9 expert rules, e.g. IF dirt is High AND grease is High THEN time is Very Long; IF dirt is Small AND grease is Small THEN time is Very Short. 5. **Inference.** Mamdani min–max: rule strength = min of the two memberships; outputs combined by max. 6. **Defuzzification.** Use the specified defuzzifier. For a separate representative-weighted illustration, strengths [.6,.4,.2] at times [25,40,60] give (.6×25+.4×40+.2×60)/(.6+.4+.2)=(15+16+12)/1.2≈35.833 minutes. These illustrative strengths are supplied here, not derived from unspecified input graphs. A Mamdani area centroid requires the full aggregate shape. 7. **Benefits.** Smooth adjustment to every load, saves water and electricity, easy to design from expert knowledge without an exact mathematical model. ### Advantages and disadvantages (bonus) **Advantages:** handles imprecise or noisy data; mimics human reasoning; easy to design and modify; no exact model needed; low computational cost. **Disadvantages:** rules and membership functions depend on expert judgement; results are hard to verify formally; tuning can take time; not suited where very high precision is required. **Yaad rakho:** Application explain karte waqt 5 cheezein: inputs, fuzzy sets, rules, inference (min/max), defuzzification + ek number ka example. --- ## M2 · Q7(a) Applications of PSO [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-final-2025--q7a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-final-2025/q7a Longer past-paper questions on Fuzzy, GA and PSO. Original TEE marks retained; not the M2 paper pattern. **List the applications of Particle Swarm Optimization and explain any one application in detail.** [10 marks] ### Answer ### Definition (1 line) Particle Swarm Optimization is a population-based metaheuristic, inspired by bird flocking and fish schooling, in which particles move through the search space using their own best position (pbest) and the swarm's best position (gbest) to find optimal or near-optimal solutions. ### Applications 1. **Function optimisation**: maximising or minimising complex non-linear, multimodal functions where derivatives are unavailable. 2. **Neural-network training**: optimising weights and biases (your Lab 8 uses the pyswarms library for this). 3. **Hyper-parameter tuning** of ML models (learning rate, number of hidden units, SVM C and γ). 4. **Feature selection**: choosing the most useful subset of features (binary PSO). 5. **Image processing**: threshold selection for segmentation, edge detection, image enhancement. 6. **Path planning** for robots and drones: shortest collision-free path. 7. **Power systems**: economic load dispatch, optimal placement of capacitors. 8. **Control engineering**: tuning PID controller gains. 9. **Scheduling and wireless sensor networks**: job scheduling, sensor node placement and clustering. ### One application in detail: training a neural network with PSO Training data → Neural network with weights = particle position → Error (fitness) → PSO moves the weights → Best weights = gbest 1. **Encoding.** Flatten all weights and biases of the network into one vector. Each particle is one complete set of weights; a network with 50 weights gives a 50-dimensional search space. 2. **Fitness.** Run the training data through the network with that particle's weights and compute the error (for example mean squared error or cross-entropy). Lower error = better particle (minimisation). 3. **Search.** Apply the velocity and position updates. Each particle is pulled towards the weight set that gave its own lowest error (pbest) and towards the lowest-error weights found by the swarm (gbest). 4. **Stop** after a fixed number of iterations or when the error stops improving; gbest is the trained network. 5. **Why PSO here:** it needs no gradients, so it works with non-differentiable activations and can explore candidate weights without differentiating the objective. It can still stagnate; comparison with gradient methods depends on the problem. **Limitation:** large networks create a high-dimensional search space, and evaluating many complete candidate networks can be computationally expensive. Gradient-based training is often more efficient when useful gradients are available; there is no universal speed ordering for every problem and implementation. ### Advantages and disadvantages (add for 10 marks) | Advantages | Disadvantages | | --- | --- | | Simple to implement; few parameters (w, c₁, c₂) | Can converge prematurely to a local optimum | | No derivatives needed | Sensitive to parameter choice | | Fast convergence in early iterations; memory (pbest, gbest) | No guarantee of the global optimum | | Objective evaluations can be parallel; particles share information through gbest | Less suited to discrete problems without modification | Marking pattern used for ACO in the same paper: listing 3 marks + explanation 2 marks (5-mark version); listing 5 + explanation 5 (10-mark version). **Yaad rakho:** Application likhte waqt teen cheezein: particle kya hai, fitness kya hai, gbest ka matlab kya hai. --- ## M2 · Q1(b) Applications of PSO [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-reexam-2026--q1b/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-reexam-2026/q1b Longer past-paper practice restricted to Fuzzy, GA and PSO. **List the applications of Particle Swarm Optimization and explain any one application in detail.** [5 marks] ### Answer ### Definition (1 line) Particle Swarm Optimization is a population-based metaheuristic, inspired by bird flocking and fish schooling, in which particles move through the search space using their own best position (pbest) and the swarm's best position (gbest) to find optimal or near-optimal solutions. ### Applications 1. **Function optimisation**: maximising or minimising complex non-linear, multimodal functions where derivatives are unavailable. 2. **Neural-network training**: optimising weights and biases (your Lab 8 uses the pyswarms library for this). 3. **Hyper-parameter tuning** of ML models (learning rate, number of hidden units, SVM C and γ). 4. **Feature selection**: choosing the most useful subset of features (binary PSO). 5. **Image processing**: threshold selection for segmentation, edge detection, image enhancement. 6. **Path planning** for robots and drones: shortest collision-free path. 7. **Power systems**: economic load dispatch, optimal placement of capacitors. 8. **Control engineering**: tuning PID controller gains. 9. **Scheduling and wireless sensor networks**: job scheduling, sensor node placement and clustering. ### One application in detail: training a neural network with PSO Training data → Neural network with weights = particle position → Error (fitness) → PSO moves the weights → Best weights = gbest 1. **Encoding.** Flatten all weights and biases of the network into one vector. Each particle is one complete set of weights; a network with 50 weights gives a 50-dimensional search space. 2. **Fitness.** Run the training data through the network with that particle's weights and compute the error (for example mean squared error or cross-entropy). Lower error = better particle (minimisation). 3. **Search.** Apply the velocity and position updates. Each particle is pulled towards the weight set that gave its own lowest error (pbest) and towards the lowest-error weights found by the swarm (gbest). 4. **Stop** after a fixed number of iterations or when the error stops improving; gbest is the trained network. 5. **Why PSO here:** it needs no gradients, so it works with non-differentiable activations and can explore candidate weights without differentiating the objective. It can still stagnate; comparison with gradient methods depends on the problem. **Limitation:** large networks create a high-dimensional search space, and evaluating many complete candidate networks can be computationally expensive. Gradient-based training is often more efficient when useful gradients are available; there is no universal speed ordering for every problem and implementation. ### Advantages and disadvantages (add for 10 marks) | Advantages | Disadvantages | | --- | --- | | Simple to implement; few parameters (w, c₁, c₂) | Can converge prematurely to a local optimum | | No derivatives needed | Sensitive to parameter choice | | Fast convergence in early iterations; memory (pbest, gbest) | No guarantee of the global optimum | | Objective evaluations can be parallel; particles share information through gbest | Less suited to discrete problems without modification | Marking pattern used for ACO in the same paper: listing 3 marks + explanation 2 marks (5-mark version); listing 5 + explanation 5 (10-mark version). **Yaad rakho:** Application likhte waqt teen cheezein: particle kya hai, fitness kya hai, gbest ka matlab kya hai. --- ## M2 · Q1(c) Fuzzy set operations [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-reexam-2026--q1c/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-reexam-2026/q1c Longer past-paper practice restricted to Fuzzy, GA and PSO. **Question. X = {(c1, 0.7), (c2, 0.9), (c3, 0.2), (c4, 0.4), (c5, 0.9)}, Y = {(c1, 0.6), (c2, 0.3), (c3, 0.8), (c4, 0.7), (c5, 0.1)}. Find X ∪ Y, X ∩ Y, ¬X ∪ ¬Y, ¬X ∩ ¬Y.** [5 marks] ### Answer #### Given The aligned X and Y memberships printed in this question. #### To find The four requested membership lists in the original element order. #### Formula and why it applies Use max for OR, min for AND, and 1−μ for NOT. $$ \mu_{X\cup Y}=\max(\mu_X,\mu_Y),\quad\mu_{X\cap Y}=\min(\mu_X,\mu_Y) $$ Complement both operands before applying the operator in ¬X∪¬Y or ¬X∩¬Y. #### Solution | | c1 | c2 | c3 | c4 | c5 | | --- | --- | --- | --- | --- | --- | | ¬X | 0.3 | 0.1 | 0.8 | 0.6 | 0.1 | | ¬Y | 0.4 | 0.7 | 0.2 | 0.3 | 0.9 | | **X ∪ Y** | 0.7 | 0.9 | 0.8 | 0.7 | 0.9 | | **X ∩ Y** | 0.6 | 0.3 | 0.2 | 0.4 | 0.1 | | **¬X ∪ ¬Y** | 0.4 | 0.7 | 0.8 | 0.6 | 0.9 | | **¬X ∩ ¬Y** | 0.3 | 0.1 | 0.2 | 0.3 | 0.1 | **Final answer.** As in the last four rows. De Morgan check: ¬X ∪ ¬Y = 1 − (X ∩ Y) and ¬X ∩ ¬Y = 1 − (X ∪ Y) ✓. ## Calculate every element separately ### Element c1 $$ \neg X:1-0.7=0.3\qquad\neg Y:1-0.6=0.4 $$ $$ X\cup Y:\max(0.7,0.6)=0.7 $$ $$ X\cap Y:\min(0.7,0.6)=0.6 $$ $$ \neg X\cup\neg Y:\max(0.3,0.4)=0.4 $$ $$ \neg X\cap\neg Y:\min(0.3,0.4)=0.3 $$ ### Element c2 $$ \neg X:1-0.9=0.1\qquad\neg Y:1-0.3=0.7 $$ $$ X\cup Y:\max(0.9,0.3)=0.9 $$ $$ X\cap Y:\min(0.9,0.3)=0.3 $$ $$ \neg X\cup\neg Y:\max(0.1,0.7)=0.7 $$ $$ \neg X\cap\neg Y:\min(0.1,0.7)=0.1 $$ ### Element c3 $$ \neg X:1-0.2=0.8\qquad\neg Y:1-0.8=0.2 $$ $$ X\cup Y:\max(0.2,0.8)=0.8 $$ $$ X\cap Y:\min(0.2,0.8)=0.2 $$ $$ \neg X\cup\neg Y:\max(0.8,0.2)=0.8 $$ $$ \neg X\cap\neg Y:\min(0.8,0.2)=0.2 $$ ### Element c4 $$ \neg X:1-0.4=0.6\qquad\neg Y:1-0.7=0.3 $$ $$ X\cup Y:\max(0.4,0.7)=0.7 $$ $$ X\cap Y:\min(0.4,0.7)=0.4 $$ $$ \neg X\cup\neg Y:\max(0.6,0.3)=0.6 $$ $$ \neg X\cap\neg Y:\min(0.6,0.3)=0.3 $$ ### Element c5 $$ \neg X:1-0.9=0.1\qquad\neg Y:1-0.1=0.9 $$ $$ X\cup Y:\max(0.9,0.1)=0.9 $$ $$ X\cap Y:\min(0.9,0.1)=0.1 $$ $$ \neg X\cup\neg Y:\max(0.1,0.9)=0.9 $$ $$ \neg X\cap\neg Y:\min(0.1,0.9)=0.1 $$ --- ## M2 · Q2(a) Fuzzy fire-hazard system [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-reexam-2026--q2a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-reexam-2026/q2a Longer past-paper practice restricted to Fuzzy, GA and PSO. **A building safety department wants a fuzzy logic system to predict fire hazards. Inputs are Temperature (1–100°C) and Smoke Level (1–100). The output Fire Risk (1–100) has labels Low, Moderate and High. Define triangular membership functions, construct at least nine rules, explain Mamdani inference and defuzzify for Temperature=65°C and Smoke=80.** [10 marks] ### Answer #### Given | Variable | Crisp value | | --- | --- | | Temperature | 65 | | Smoke Level | 80 | The question asks us to design the membership functions. This solution chooses Low=Tri(1,25,50), Medium=Tri(25,50,75), High=Tri(50,75,100) for both inputs. Output labels use the same triangles in their respective order. These are explicit design choices, not additional printed givens. #### To find A complete stated design and its continuous Mamdani centroid. #### Formula and why it applies Use min for AND and implication, max for aggregation and the true area centroid of the resulting shape. A peak-only weighted average is a different defuzzifier. $$ z^*=\frac{\int z\mu_{out}(z)\,dz}{\int\mu_{out}(z)\,dz} $$ #### Solution ##### Step 1. Specify the membership functions | Input / output position | Triangle | Peak | | --- | --- | --- | | Low / Low | (1,25,50) | 25 | | Medium / Moderate | (25,50,75) | 50 | | High / High | (50,75,100) | 75 | Use 0 outside each triangle, the rising fraction before the peak and the falling fraction after it. This design is evaluated at the requested input; a deployed controller would also need coverage/fallback at all boundary values. ##### Step 2. Write all nine rules | Temperature / Smoke Level | Low | Medium | High | | --- | --- | --- | --- | | Low | Low | Low | Moderate | | Medium | Low | Moderate | High | | High | Moderate | High | High | Read each cell as an IF–AND–THEN rule. For example, the centre cell combines Medium with Medium. ##### Step 3. Fuzzify Temperature = 65 $$ \mu_{Low}(65)=0 $$ $$ \mu_{Medium}(65)=\frac{75-65}{75-50}=0.4 $$ $$ \mu_{High}(65)=\frac{65-50}{75-50}=0.6 $$ ##### Step 4. Fuzzify Smoke Level = 80 $$ \mu_{Low}(80)=0 $$ $$ \mu_{Medium}(80)=0 $$ $$ \mu_{High}(80)=\frac{100-80}{100-75}=0.8 $$ ##### Step 5. Evaluate all rules and aggregate matching consequents | Temperature | Smoke Level | Firing strength | Consequent | | --- | --- | --- | --- | | Low | Low | min(0,0) = 0 | Low | | Low | Medium | min(0,0) = 0 | Low | | Low | High | min(0,0.8) = 0 | Moderate | | Medium | Low | min(0.4,0) = 0 | Low | | Medium | Medium | min(0.4,0) = 0 | Moderate | | Medium | High | min(0.4,0.8) = 0.4 | High | | High | Low | min(0.6,0) = 0 | Moderate | | High | Medium | min(0.6,0) = 0 | High | | High | High | min(0.6,0.8) = 0.6 | High | Aggregated label strengths: Low = 0, Moderate = 0, High = 0.6. $$ \mu_{out}(z)=\max_r\min(\alpha_r,\mu_{C_r}(z)) $$ ##### Step 6. Draw the aggregated output Swipe across the diagram to read its labels ↔ ![The shaded upper envelope counts any overlap only once.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fire_aggregate.e0cab4ad42.svg) Diagram: The shaded upper envelope counts any overlap only once.. Labels: Chosen triangular output sets and their max-aggregate | 0 | 0.5 | 1 | 1 | 25 | 50 | 75 | 100 | μ | Output score | Low | Moderate | High | centroid ≈ 75 The shaded upper envelope counts any overlap only once. ##### Step 7. Calculate area and moment of each straight-line strip For a strip from a to b with endpoint heights u and v, its trapezoid area is A. Its first moment M is the integral of z times the line height. $$ A=(b-a)(u+v)/2 $$ $$ M=\frac{b-a}{6}\big[(2a+b)u+(a+2b)v\big] $$ | a | b | u | v | Area A | Moment M | | --- | --- | --- | --- | --- | --- | | 50 | 65 | 0 | 0.6 | 4.5 | 270 | | 65 | 85 | 0.6 | 0.6 | 12 | 900 | | 85 | 100 | 0.6 | 0 | 4.5 | 405 | ##### Step 8. Divide total moment by total area $$ \sum A=21 $$ $$ \sum M=1575 $$ $$ z^*=\frac{1575}{21}\approx75 $$ **Final answer.** For these stated triangles and rules, the centroid is 75. This is in the high-risk region of this illustrative fuzzy design. Because the paper asks for a design, a different explicitly specified membership/rule system can yield a different numerical output. Explain the choices and use the chosen defuzzification method consistently; no particular examiner mark outcome is promised. ## Why the clipped High shape is symmetric about 75 Only the High output survives, at height .6. Its triangle is (50,75,100). Solve its rising edge: (z−50)/25=.6, so z=50+15=65. Solve its falling edge: (100−z)/25=.6, so z=100−15=85. | Piece | Area calculation | Centre | Moment | | --- | --- | --- | --- | | Left triangle [50,65] | ½×15×.6=4.5 | (50+65+65)/3=60 | 4.5×60=270 | | Rectangle [65,85] | 20×.6=12 | (65+85)/2=75 | 12×75=900 | | Right triangle [85,100] | ½×15×.6=4.5 | (85+85+100)/3=90 | 4.5×90=405 | The horizontal centroid of a triangle is the average of the three vertex x-coordinates. Add areas: 4.5+12+4.5=21. Add moments:270+900+405=1575. Divide 1575/21=75, agreeing with symmetry. --- ## M2 · Q3(b) Fuzzy relations and composition [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-reexam-2026--q3b/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-reexam-2026/q3b Longer past-paper practice restricted to Fuzzy, GA and PSO. **Question. A = {0.6/small + 0.4/medium + 0.1/large}, B = {0.4/slow + 0.8/moderate + 0.3/fast}, C = {0.5/small + 0.2/medium + 0.4/large}. Find R = A × B, S = C × B and the max–min composition S ∘ R.** [10 marks] ### Answer #### Given The fuzzy sets or relation matrices and their listed row/column order. #### To find The relation matrix/matrices and the composition specified in the question. #### Formula and why it applies For Cartesian products take min of row and column membership. For the ordered matrix composition shown here: $$ T_{ik}=\max_j\min(S_{ij},R_{jk}) $$ If max–product is asked instead, replace the inner min by multiplication. #### Solution ##### Step 1. R and S $$ R=\begin{bmatrix}0.4&0.6&0.3\\0.4&0.4&0.3\\0.1&0.1&0.1\end{bmatrix},\qquad S=\begin{bmatrix}0.4&0.5&0.3\\0.2&0.2&0.2\\0.4&0.4&0.3\end{bmatrix} $$ ##### Step 2. S ∘ R $$ (1,1)=\max[\min(.4,.4),\min(.5,.4),\min(.3,.1)]=\max(.4,.4,.1)=0.4 $$ $$ (1,2)=\max[\min(.4,.6),\min(.5,.4),\min(.3,.1)]=\max(.4,.4,.1)=0.4 $$ $$ (1,3)=\max[\min(.4,.3),\min(.5,.3),\min(.3,.1)]=0.3 $$ $$ \text{row 2 (all 0.2)}=(0.2,0.2,0.2);\quad(3,1)=\max(.4,.4,.1)=0.4,\;(3,2)=\max(.4,.4,.1)=0.4,\;(3,3)=0.3 $$ **Final answer.** $$ R=\begin{bmatrix}.4&.6&.3\\.4&.4&.3\\.1&.1&.1\end{bmatrix} $$ , $$ S=\begin{bmatrix}.4&.5&.3\\.2&.2&.2\\.4&.4&.3\end{bmatrix} $$ , $$ S\circ R=\begin{bmatrix}0.4&0.4&0.3\\0.2&0.2&0.2\\0.4&0.4&0.3\end{bmatrix} $$ . (The key gives R and S only.) ## Every cell without the row shortcuts Both printed relations run from size to speed; their middle labels do not match for genuine relation composition. The numerical working below follows the question’s intended row-of-S with column-of-R operation using explicitly assumed positional alignment of the three middle entries. ### R: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | R11 | 0.6 | 0.4 | 0.4 | | R12 | 0.6 | 0.8 | 0.6 | | R13 | 0.6 | 0.3 | 0.3 | | R21 | 0.4 | 0.4 | 0.4 | | R22 | 0.4 | 0.8 | 0.4 | | R23 | 0.4 | 0.3 | 0.3 | | R31 | 0.1 | 0.4 | 0.1 | | R32 | 0.1 | 0.8 | 0.1 | | R33 | 0.1 | 0.3 | 0.1 | ### S: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | S11 | 0.5 | 0.4 | 0.4 | | S12 | 0.5 | 0.8 | 0.5 | | S13 | 0.5 | 0.3 | 0.3 | | S21 | 0.2 | 0.4 | 0.2 | | S22 | 0.2 | 0.8 | 0.2 | | S23 | 0.2 | 0.3 | 0.2 | | S31 | 0.4 | 0.4 | 0.4 | | S32 | 0.4 | 0.8 | 0.4 | | S33 | 0.4 | 0.3 | 0.3 | **Result cell (1,1)** $$ \max\{\min(0.4,0.4),\min(0.5,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (1,2)** $$ \max\{\min(0.4,0.6),\min(0.5,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (1,3)** $$ \max\{\min(0.4,0.3),\min(0.5,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.3,0.3,0.1\}=0.3 $$ **Result cell (2,1)** $$ \max\{\min(0.2,0.4),\min(0.2,0.4),\min(0.2,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (2,2)** $$ \max\{\min(0.2,0.6),\min(0.2,0.4),\min(0.2,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (2,3)** $$ \max\{\min(0.2,0.3),\min(0.2,0.3),\min(0.2,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (3,1)** $$ \max\{\min(0.4,0.4),\min(0.4,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (3,2)** $$ \max\{\min(0.4,0.6),\min(0.4,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (3,3)** $$ \max\{\min(0.4,0.3),\min(0.4,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.3,0.3,0.1\}=0.3 $$ --- ## M2 · Q4(a) PSO, one iteration [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-reexam-2026--q4a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-reexam-2026/q4a Longer past-paper practice restricted to Fuzzy, GA and PSO. **Question. Maximize $$ F(x)=x_1^2-x_1x_2+x_2^2+3x_1+4x_2+5 $$ using Particle Swarm Optimization. $$ X=\begin{bmatrix}3.1472&-4.0246\\4.0579&-2.2150\end{bmatrix},\;V=\begin{bmatrix}0.2194&0.2449\\0.1908&0.2228\end{bmatrix} $$ . Take $$ W=0.9,\;c_1=c_2=1.5,\;r_1=0.5949,\;r_2=0.0855 $$ . Optimize and give the solution after 1 iteration showing the step-by-step process.** [10 marks] ### Answer #### Given P1: $$ x=(3.1472,\,-4.0246) $$ , $$ v=(0.2194,\,0.2449) $$ . P2: $$ x=(4.0579,\,-2.2150) $$ , $$ v=(0.1908,\,0.2228) $$ . $$ c_1r_1=0.89235 $$ , $$ c_2r_2=0.12825 $$ . #### To find Updated velocities, positions, pbest and gbest after one iteration. #### Formula and why it applies $$ v^{new}=wv+c_1r_1(pbest-x)+c_2r_2(gbest-x),\quad x^{new}=x+v^{new} $$ Initial personal bests equal the starting positions. Use one frozen global best for all particles. This question maximises fitness, so keep a memory only when its score is larger. #### Solution ##### Step 1. Fitness of P1 (term by term) $$ x_1^2=9.904868,\;-x_1x_2=+12.666221,\;x_2^2=16.197405,\;3x_1=9.4416,\;4x_2=-16.0984 $$ $$ F_1=9.904868+12.666221+16.197405+9.4416-16.0984+5=37.1117 $$ ##### Step 2. Fitness of P2 $$ F_2=16.466552+8.988249+4.906225+12.1737-8.86+5=38.6747 $$ ##### Step 3. pbest and gbest pbest = current positions. $$ 38.6747>37.1117 $$ , so **gbest = P2 = (4.0579, −2.2150)**. ##### Step 4. Particle 1 updates $$ v_{11}=0.9(0.2194)+0+0.12825(4.0579-3.1472)=0.19746+0.116797=0.314257 $$ $$ x_{11}=3.1472+0.314257=3.461457 $$ $$ v_{12}=0.9(0.2449)+0+0.12825(-2.2150+4.0246)=0.22041+0.232081=0.452491 $$ $$ x_{12}=-4.0246+0.452491=-3.572109 $$ ##### Step 5. Particle 2 updates (gbest particle: only inertia acts) $$ v_{21}=0.9(0.1908)=0.17172,\quad x_{21}=4.22962;\qquad v_{22}=0.9(0.2228)=0.20052,\quad x_{22}=-2.01448 $$ ##### Step 6. New fitness and bests Use the same objective at each new position. Keep full precision internally; displayed coordinates are rounded. $$ F(3.461457275,-3.5721088)=(3.461457275)^2-(3.461457275)(-3.5721088)+(-3.5721088)^2+3(3.461457275)+4(-3.5721088)+5 $$ $$ =11.981686467+12.364701993+12.759961279+10.384371825+(-14.2884352)+5\approx38.202286364 $$ $$ F(4.22962,-2.01448)=(4.22962)^2-(4.22962)(-2.01448)+(-2.01448)^2+3(4.22962)+4(-2.01448)+5 $$ $$ =17.889685344+8.520484898+4.05812967+12.68886+(-8.05792)+5\approx40.099239912 $$ | Particle | New position | New F | Old F | pbest | | --- | --- | --- | --- | --- | | 1 | (3.461457, −3.572109) | 38.2023 | 37.1117 | updated | | 2 | (4.22962, −2.01448) | 40.0992 | 38.6747 | updated | The best pbest is now P2's with F = 40.0992, so gbest moves to (4.22962, −2.01448). **Final answer.** $$ V=\begin{bmatrix}0.314257&0.452491\\0.17172&0.20052\end{bmatrix} $$ , $$ X=\begin{bmatrix}3.461457&-3.572109\\4.22962&-2.01448\end{bmatrix} $$ . Both particles improved; new gbest = (4.22962, −2.01448) with F = 40.0992. **Official key warning.** The re-exam answer key solves a *different* problem: particles (0.4, 0.3) and (0.3, 0.4) with velocities (0.1, −0.2) and (−0.2, 0.1), giving fitness 7.53 and 7.86. Do not copy it. The numbers above solve the question as printed. This X and V are particles 1 and 2 of your lecture's five-particle example; only the function differs (+3x₁ and +5 here instead of +2x₁ and +3). --- ## M2 · Q5(a) Genetic algorithm [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-reexam-2026--q5a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-reexam-2026/q5a Longer past-paper practice restricted to Fuzzy, GA and PSO. **Implement a GA to maximise $$ f(x)=2x^3-3x^2+8x $$ for x in [0,50]. Take 4-bit representations of 12, 20, 17, 13. Crossover after the second bit and mutate by flipping any one bit. Produce two offspring and comment on fitness.** [10 marks] ### Answer **The printed data are inconsistent.** 20 and 17 do not fit into four bits, and covering 0–50 needs six bits. We use six-bit encoding to preserve the supplied parent values and the stated range. #### Given Printed parents: [12, 20, 17, 13]. Worked variant: [12, 20, 17, 13] with 6 bits. For the unspecified selection step, select the two highest-fitness candidates. For the unspecified mutation location, flip the last bit of each child. Crossover is after bit 2. #### To find Two valid children under the declared assumptions and their fitness. #### Formula and why it applies Encoding must respect the candidate range. Crossover swaps tails; mutation flips the chosen bit. Fitness is evaluated after mutation. $$ f(x)=2x^3-3x^2+8x $$ #### Solution ##### Step 1. Encode and evaluate the declared parent population | x | Binary | Fitness substitution | | --- | --- | --- | | 12 | 001100 | 2(12³) − 3(12²) + 8(12) = 3120 | | 20 | 010100 | 2(20³) − 3(20²) + 8(20) = 14960 | | 17 | 010001 | 2(17³) − 3(17²) + 8(17) = 9095 | | 13 | 001101 | 2(13³) − 3(13²) + 8(13) = 3991 | ##### Step 2. Select the two fittest under the stated assumption [20, 17] with fitness [14960, 9095]. ##### Step 3. Cut after bit 2 and exchange tails | Parent 1 | Parent 2 | Child 1 | Child 2 | | --- | --- | --- | --- | | 01 | 0100 | 01 | 0001 | 010001 | 010100 | ##### Step 4. Flip the last bit of each child | Before | After | Decoded x | | --- | --- | --- | | 010001 | 010000 | 16 | | 010100 | 010101 | 21 | ##### Step 5. Evaluate both children | x | Fitness calculation | | --- | --- | | 16 | 2(4096) − 3(256) + 8(16) = 7552 | | 21 | 2(9261) − 3(441) + 8(21) = 17367 | ##### Step 6. Compare with the selected parents Best parent score = 14960; best child score = 17367. Parent average = 12027.5; child average = 12459.5. Both the best score and the average score increased in this worked variant. An individual child can still be worse than a parent; GA does not guarantee improvement after every random operator. **Final answer.** Under these declared repairs and operator choices, children are [16, 21] with fitness [7552, 17367]. These are not unique answers to the inconsistent and under-specified original question. The printed wording about “two offspring” is not an explicit requirement for two complete generations. The M2 examples separately work two full generations because your class update asks for iteration 2. ## Evaluate the polynomial without skipping the powers Calculate x²=x×x, then x³=x²×x. Multiply each by its coefficient and combine with the written signs. A bit-string repair in the worked answer is an explicitly stated response to inconsistent printed givens; it is not a new instruction in the original paper. ### x=12 $$ x^2=12\times12=144\qquad x^3=144\times12=1728 $$ $$ 2x^3=3456\quad 3x^2=432\quad 8x=96 $$ $$ f(12)=3456-432+96=3120 $$ ### x=20 $$ x^2=20\times20=400\qquad x^3=400\times20=8000 $$ $$ 2x^3=16000\quad 3x^2=1200\quad 8x=160 $$ $$ f(20)=16000-1200+160=14960 $$ ### x=17 $$ x^2=17\times17=289\qquad x^3=289\times17=4913 $$ $$ 2x^3=9826\quad 3x^2=867\quad 8x=136 $$ $$ f(17)=9826-867+136=9095 $$ ### x=13 $$ x^2=13\times13=169\qquad x^3=169\times13=2197 $$ $$ 2x^3=4394\quad 3x^2=507\quad 8x=104 $$ $$ f(13)=4394-507+104=3991 $$ ### x=16 $$ x^2=16\times16=256\qquad x^3=256\times16=4096 $$ $$ 2x^3=8192\quad 3x^2=768\quad 8x=128 $$ $$ f(16)=8192-768+128=7552 $$ ### x=21 $$ x^2=21\times21=441\qquad x^3=441\times21=9261 $$ $$ 2x^3=18522\quad 3x^2=1323\quad 8x=168 $$ $$ f(21)=18522-1323+168=17367 $$ --- ## M2 · Q7(a) Applications of fuzzy logic [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/papers/paper-reexam-2026--q7a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/papers/paper-reexam-2026/q7a Longer past-paper practice restricted to Fuzzy, GA and PSO. **List the applications of fuzzy logic. Explain any one application in detail.** [10 marks] ### Answer ### Definition Fuzzy logic is a multi-valued logic, introduced by Lotfi Zadeh (1965), in which truth values range over [0, 1]; it lets machines reason with vague, human-like terms such as “slightly dirty” or “very hot”. ### Applications | Area | Examples | | --- | --- | | Home appliances | Washing machines (wash time from dirt and load), air conditioners, refrigerators, rice cookers, vacuum cleaners | | Automotive | Automatic transmission, anti-lock braking (ABS), cruise control, engine control | | Transportation | Traffic-signal control, Sendai subway train control (smooth braking and acceleration) | | Consumer electronics | Camera autofocus and auto-exposure, image stabilisation | | Industrial control | Cement kilns, pH, temperature and distillation control in chemical plants | | Aerospace | Satellite and spacecraft attitude control | | Medicine | Diagnosis support, anaesthesia dosage control | | Finance and decisions | Loan approval, credit and fire-risk scoring, stock trading rules | | AI and robotics | Robot navigation, natural-language interfaces, expert systems | ### One application in detail: fuzzy washing machine 1. **Problem.** Wash time depends on how dirty and how greasy the clothes are. These are vague (“a bit dirty”), and a fixed timer wastes water and power. 2. **Inputs and outputs.** Sensors measure dirt (turbidity of water) and grease; the output is wash time. Each is described by fuzzy sets: dirt Small/Medium/High, grease Small/Medium/High, time Very Short … Very Long. 3. **Fuzzification.** Sensor readings are converted to membership degrees, e.g. dirt 60% → Medium 0.8, High 0.2. 4. **Rule base.** 9 expert rules, e.g. IF dirt is High AND grease is High THEN time is Very Long; IF dirt is Small AND grease is Small THEN time is Very Short. 5. **Inference.** Mamdani min–max: rule strength = min of the two memberships; outputs combined by max. 6. **Defuzzification.** Use the specified defuzzifier. For a separate representative-weighted illustration, strengths [.6,.4,.2] at times [25,40,60] give (.6×25+.4×40+.2×60)/(.6+.4+.2)=(15+16+12)/1.2≈35.833 minutes. These illustrative strengths are supplied here, not derived from unspecified input graphs. A Mamdani area centroid requires the full aggregate shape. 7. **Benefits.** Smooth adjustment to every load, saves water and electricity, easy to design from expert knowledge without an exact mathematical model. ### Advantages and disadvantages (bonus) **Advantages:** handles imprecise or noisy data; mimics human reasoning; easy to design and modify; no exact model needed; low computational cost. **Disadvantages:** rules and membership functions depend on expert judgement; results are hard to verify formally; tuning can take time; not suited where very high precision is required. **Yaad rakho:** Application explain karte waqt 5 cheezein: inputs, fuzzy sets, rules, inference (min/max), defuzzification + ek number ka example. --- ## M2 · Q1(a) · Fuzzy operations [2] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-1--q1a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-1/q1a Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q1(a). On U = {u1, u2, u3}, A has memberships [0.2, 0.8, 0.6] and B has memberships [0.7, 0.3, 0.5]. Find A ∪ B, A ∩ B, Bᶜ and A ∖ B using standard max–min fuzzy operations.** [2 marks · Practice] #### Given | Element | u1 | u2 | u3 | | --- | --- | --- | --- | | A | 0.2 | 0.8 | 0.6 | | B | 0.7 | 0.3 | 0.5 | #### To find Four membership lists in exactly the same element order. #### Formula and why it applies Union is OR → max. Intersection is AND → min. Complement is NOT → 1 minus membership. Difference A ∖ B means A AND NOT B, so first complement B, then intersect with A. $$ \mu_{A\cup B}=\max(\mu_A,\mu_B)\qquad\mu_{A\cap B}=\min(\mu_A,\mu_B) $$ $$ \mu_{B^c}=1-\mu_B\qquad\mu_{A\setminus B}=\min(\mu_A,1-\mu_B) $$ #### Solution ##### Step 1. Align the same element All operations compare the two memberships of the same u. Do not compare A at u₁ with B at u₂. ##### Step 2. Calculate one element completely $$ \mu_{B^c}(u_1)=1-0.7=0.3 $$ $$ \mu_{A\setminus B}(u_1)=\min(0.2,0.3)=0.2 $$ ##### Step 3. Repeat for each element | Element | A | B | Union | Intersection | Bᶜ | A ∖ B | | --- | --- | --- | --- | --- | --- | --- | | u1 | 0.2 | 0.7 | max(0.2, 0.7) = 0.7 | min(0.2, 0.7) = 0.2 | 1 − 0.7 = 0.3 | min(0.2, 0.3) = 0.2 | | u2 | 0.8 | 0.3 | max(0.8, 0.3) = 0.8 | min(0.8, 0.3) = 0.3 | 1 − 0.3 = 0.7 | min(0.8, 0.7) = 0.7 | | u3 | 0.6 | 0.5 | max(0.6, 0.5) = 0.6 | min(0.6, 0.5) = 0.5 | 1 − 0.5 = 0.5 | min(0.6, 0.5) = 0.5 | **Final answer.** A ∪ B = [0.7, 0.8, 0.6]; A ∩ B = [0.2, 0.3, 0.5]; Bᶜ = [0.3, 0.7, 0.5]; A ∖ B = [0.2, 0.7, 0.5]. --- ## M2 · Q1(b) · Swarm intelligence [2] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-1--q1b/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-1/q1b Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q1(b). Define swarm intelligence. Briefly explain one nature-inspired algorithm.** [2 marks · Practice] #### Answer Swarm intelligence is collective problem-solving arising from interactions among simple agents. PSO is inspired by social movement in flocks. Each particle represents a candidate solution and has a position and velocity. It moves using its own best visited position and the swarm’s best position. Repeated objective evaluation guides the search. --- ## M2 · Q2 · GA through two generations [3] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-1--q2/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-1/q2 Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q2. Maximise $$ f(x)=x^2 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [3, 5, 10, 12]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 4 of children [1, 2, 3, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Current individual row numbers [3, 4, 4, 3] | | 2 | Current individual row numbers [2, 3, 2, 3] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Use exactly the supplied selection rule; replacing the weakest by the strongest is not a universal GA step. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 3 | 0011 | 3 × 3 = 9 | | 2 | 5 | 0101 | 5 × 5 = 25 | | 3 | 10 | 1010 | 10 × 10 = 100 | | 4 | 12 | 1100 | 12 × 12 = 144 | ##### Step 2. Generation 1: apply the supplied selection order The question directly supplies individual numbers [3, 4, 4, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 3 | 10 | 1010 | | 2 | 4 | 12 | 1100 | | 3 | 4 | 12 | 1100 | | 4 | 3 | 10 | 1010 | ##### Step 3. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 10 | 10 | 11 | 00 | 1000 | 1110 | | 3 & 4 | 11 | 00 | 10 | 10 | 1110 | 1000 | ##### Step 4. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1000 | Flip bit 4 | 1001 | | 2 | 1110 | Flip bit 4 | 1111 | | 3 | 1110 | Flip bit 4 | 1111 | | 4 | 1000 | Flip bit 4 | 1001 | ##### Step 5. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1001 | 8 + 0 + 0 + 1 = 9 | 9² = 81 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1001 | 8 + 0 + 0 + 1 = 9 | 9² = 81 | ##### Step 6. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{9+25+100+144}{4}=\frac{278}{4}=69.5 $$ $$ \overline f_{new}=\frac{81+225+225+81}{4}=\frac{612}{4}=153 $$ Best objective: 144 → 225. Higher is better. The next population is [9, 15, 15, 9]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 7. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 9 | 1001 | 9 × 9 = 81 | | 2 | 15 | 1111 | 15 × 15 = 225 | | 3 | 15 | 1111 | 15 × 15 = 225 | | 4 | 9 | 1001 | 9 × 9 = 81 | ##### Step 8. Generation 2: apply the supplied selection order The question directly supplies individual numbers [2, 3, 2, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 2 | 15 | 1111 | | 2 | 3 | 15 | 1111 | | 3 | 2 | 15 | 1111 | | 4 | 3 | 15 | 1111 | ##### Step 9. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 11 | 11 | 11 | 11 | 1111 | 1111 | | 3 & 4 | 11 | 11 | 11 | 11 | 1111 | 1111 | ##### Step 10. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1111 | Flip bit 4 | 1110 | | 2 | 1111 | Flip bit 4 | 1110 | | 3 | 1111 | Flip bit 4 | 1110 | | 4 | 1111 | Flip bit 4 | 1110 | ##### Step 11. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | ##### Step 12. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{81+225+225+81}{4}=\frac{612}{4}=153 $$ $$ \overline f_{new}=\frac{196+196+196+196}{4}=\frac{784}{4}=196 $$ Best objective: 225 → 196. Higher is better. The next population is [14, 14, 14, 14]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [14, 14, 14, 14]. Best final objective = 196 at x = [14]. --- ## M2 · Q3 · Two-particle PSO through iteration 2 [3] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-1--q3/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-1/q3 Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q3. Minimise $$ f(x)=x^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (-1) | (0) | | B | (3) | (0) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(-1)=(-1)^2=1=1 $$ **Particle B** $$ f(3)=3^2=9=9 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(-1) $$ Its score is 1. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (-1). pbest values are A: (-1), B: (3). ##### Step 4. Iteration 1: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=-1-(-1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-1-(-1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=(-1)+0=-1 $$ ##### Step 5. Iteration 1: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3-(3)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-1-(3)=-4 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-4)=-2 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-2)=-2 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=3+(-2)=1 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(-1)=(-1)^2=1=1 $$ **Particle B** $$ f(1)=1^2=1=1 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 1 | 1 | Keep old: tie | (-1) | | B | 9 | 1 | Replace: new score is lower | (1) | Compare the retained pbest scores. gbest = **(-1)**, with objective **1**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (-1) | (0) | (-1) | | B | (1) | (-2) | (1) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (-1). pbest values are A: (-1), B: (1). ##### Step 10. Iteration 2: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=-1-(-1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-1-(-1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=(-1)+0=-1 $$ ##### Step 11. Iteration 2: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-2)=-1 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-1-(1)=-2 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-2)=-1 $$ Add the three velocity contributions. $$ v^{(2)}=(-1)+0+(-1)=-2 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1+(-2)=-1 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(-1)=(-1)^2=1=1 $$ **Particle B** $$ f(-1)=(-1)^2=1=1 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 1 | 1 | Keep old: tie | (-1) | | B | 1 | 1 | Keep old: tie | (1) | Compare the retained pbest scores. gbest = **(-1)**, with objective **1**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (-1) | (0) | (-1) | | B | (-1) | (-2) | (1) | **Final answer.** After iteration 2, gbest = (-1), objective = 1. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. --- ## M2 · Q4 · Fuzzy architecture [3] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-1--q4/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-1/q4 Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q4. Explain the architecture of a Mamdani fuzzy inference system with a fan-controller example.** [3 marks · Practice] #### Answer The fuzzifier maps crisp temperature and occupancy inputs to membership degrees. The knowledge base contains membership functions and IF–THEN rules. The inference engine combines antecedent degrees, applies each rule’s consequent and aggregates the inferred outputs. The defuzzifier converts this aggregate to a numerical fan speed. For example, IF temperature is High AND occupancy is High THEN fan speed is Very High. --- ## M2 · Q1(a) · Fuzzy operations [2] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-2--q1a/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-2/q1a Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q1(a). On U = {u1, u2, u3}, A has memberships [0.1, 0.5, 1] and B has memberships [0.9, 0.4, 0]. Find A ∪ B, A ∩ B, Bᶜ and A ∖ B using standard max–min fuzzy operations.** [2 marks · Practice] #### Given | Element | u1 | u2 | u3 | | --- | --- | --- | --- | | A | 0.1 | 0.5 | 1 | | B | 0.9 | 0.4 | 0 | #### To find Four membership lists in exactly the same element order. #### Formula and why it applies Union is OR → max. Intersection is AND → min. Complement is NOT → 1 minus membership. Difference A ∖ B means A AND NOT B, so first complement B, then intersect with A. $$ \mu_{A\cup B}=\max(\mu_A,\mu_B)\qquad\mu_{A\cap B}=\min(\mu_A,\mu_B) $$ $$ \mu_{B^c}=1-\mu_B\qquad\mu_{A\setminus B}=\min(\mu_A,1-\mu_B) $$ #### Solution ##### Step 1. Align the same element All operations compare the two memberships of the same u. Do not compare A at u₁ with B at u₂. ##### Step 2. Calculate one element completely $$ \mu_{B^c}(u_1)=1-0.9=0.1 $$ $$ \mu_{A\setminus B}(u_1)=\min(0.1,0.1)=0.1 $$ ##### Step 3. Repeat for each element | Element | A | B | Union | Intersection | Bᶜ | A ∖ B | | --- | --- | --- | --- | --- | --- | --- | | u1 | 0.1 | 0.9 | max(0.1, 0.9) = 0.9 | min(0.1, 0.9) = 0.1 | 1 − 0.9 = 0.1 | min(0.1, 0.1) = 0.1 | | u2 | 0.5 | 0.4 | max(0.5, 0.4) = 0.5 | min(0.5, 0.4) = 0.4 | 1 − 0.4 = 0.6 | min(0.5, 0.6) = 0.5 | | u3 | 1 | 0 | max(1, 0) = 1 | min(1, 0) = 0 | 1 − 0 = 1 | min(1, 1) = 1 | **Final answer.** A ∪ B = [0.9, 0.5, 1.0]; A ∩ B = [0.1, 0.4, 0.0]; Bᶜ = [0.1, 0.6, 1.0]; A ∖ B = [0.1, 0.5, 1.0]. --- ## M2 · Q1(b) · Swarm intelligence [2] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-2--q1b/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-2/q1b Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q1(b). Define swarm intelligence. Briefly explain one nature-inspired algorithm.** [2 marks · Practice] #### Answer Swarm intelligence is collective problem-solving arising from interactions among simple agents. PSO is inspired by social movement in flocks. Each particle represents a candidate solution and has a position and velocity. It moves using its own best visited position and the swarm’s best position. Repeated objective evaluation guides the search. --- ## M2 · Q2 · GA through two generations [3] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-2--q2/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-2/q2 Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q2. Maximise $$ f(x)=x^2 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [1, 6, 9, 14]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 3 of children [1, 2, 3, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Current individual row numbers [3, 4, 4, 3] | | 2 | Current individual row numbers [2, 3, 2, 3] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Use exactly the supplied selection rule; replacing the weakest by the strongest is not a universal GA step. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 1 | 0001 | 1 × 1 = 1 | | 2 | 6 | 0110 | 6 × 6 = 36 | | 3 | 9 | 1001 | 9 × 9 = 81 | | 4 | 14 | 1110 | 14 × 14 = 196 | ##### Step 2. Generation 1: apply the supplied selection order The question directly supplies individual numbers [3, 4, 4, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 3 | 9 | 1001 | | 2 | 4 | 14 | 1110 | | 3 | 4 | 14 | 1110 | | 4 | 3 | 9 | 1001 | ##### Step 3. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 10 | 01 | 11 | 10 | 1010 | 1101 | | 3 & 4 | 11 | 10 | 10 | 01 | 1101 | 1010 | ##### Step 4. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1010 | Flip bit 3 | 1000 | | 2 | 1101 | Flip bit 3 | 1111 | | 3 | 1101 | Flip bit 3 | 1111 | | 4 | 1010 | Flip bit 3 | 1000 | ##### Step 5. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1000 | 8 + 0 + 0 + 0 = 8 | 8² = 64 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1000 | 8 + 0 + 0 + 0 = 8 | 8² = 64 | ##### Step 6. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{1+36+81+196}{4}=\frac{314}{4}=78.5 $$ $$ \overline f_{new}=\frac{64+225+225+64}{4}=\frac{578}{4}=144.5 $$ Best objective: 196 → 225. Higher is better. The next population is [8, 15, 15, 8]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 7. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 8 | 1000 | 8 × 8 = 64 | | 2 | 15 | 1111 | 15 × 15 = 225 | | 3 | 15 | 1111 | 15 × 15 = 225 | | 4 | 8 | 1000 | 8 × 8 = 64 | ##### Step 8. Generation 2: apply the supplied selection order The question directly supplies individual numbers [2, 3, 2, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 2 | 15 | 1111 | | 2 | 3 | 15 | 1111 | | 3 | 2 | 15 | 1111 | | 4 | 3 | 15 | 1111 | ##### Step 9. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 11 | 11 | 11 | 11 | 1111 | 1111 | | 3 & 4 | 11 | 11 | 11 | 11 | 1111 | 1111 | ##### Step 10. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1111 | Flip bit 3 | 1101 | | 2 | 1111 | Flip bit 3 | 1101 | | 3 | 1111 | Flip bit 3 | 1101 | | 4 | 1111 | Flip bit 3 | 1101 | ##### Step 11. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1101 | 8 + 4 + 0 + 1 = 13 | 13² = 169 | | 1101 | 8 + 4 + 0 + 1 = 13 | 13² = 169 | | 1101 | 8 + 4 + 0 + 1 = 13 | 13² = 169 | | 1101 | 8 + 4 + 0 + 1 = 13 | 13² = 169 | ##### Step 12. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{64+225+225+64}{4}=\frac{578}{4}=144.5 $$ $$ \overline f_{new}=\frac{169+169+169+169}{4}=\frac{676}{4}=169 $$ Best objective: 225 → 169. Higher is better. The next population is [13, 13, 13, 13]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [13, 13, 13, 13]. Best final objective = 169 at x = [13]. --- ## M2 · Q3 · Two-particle PSO through iteration 2 [3] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-2--q3/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-2/q3 Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q3. Minimise $$ f(x)=x^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (2) | (0) | | B | (5) | (0) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(2)=2^2=4=4 $$ **Particle B** $$ f(5)=5^2=25=25 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(2) $$ Its score is 4. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (2). pbest values are A: (2), B: (5). ##### Step 4. Iteration 1: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=2-(2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=2-(2)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=2+0=2 $$ ##### Step 5. Iteration 1: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=5-(5)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=2-(5)=-3 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-3)=-1.5 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-1.5)=-1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=5+(-1.5)=3.5 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(2)=2^2=4=4 $$ **Particle B** $$ f(3.5)=3.5^2=12.25=12.25 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 4 | 4 | Keep old: tie | (2) | | B | 25 | 12.25 | Replace: new score is lower | (3.5) | Compare the retained pbest scores. gbest = **(2)**, with objective **4**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (2) | (0) | (2) | | B | (3.5) | (-1.5) | (3.5) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (2). pbest values are A: (2), B: (3.5). ##### Step 10. Iteration 2: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=2-(2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=2-(2)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=2+0=2 $$ ##### Step 11. Iteration 2: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-1.5)=-0.75 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3.5-(3.5)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=2-(3.5)=-1.5 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-1.5)=-0.75 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.75)+0+(-0.75)=-1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=3.5+(-1.5)=2 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(2)=2^2=4=4 $$ **Particle B** $$ f(2)=2^2=4=4 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 4 | 4 | Keep old: tie | (2) | | B | 12.25 | 4 | Replace: new score is lower | (2) | Compare the retained pbest scores. gbest = **(2)**, with objective **4**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (2) | (0) | (2) | | B | (2) | (-1.5) | (2) | **Final answer.** After iteration 2, gbest = (2), objective = 4. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. --- ## M2 · Q4 · Fuzzy membership [3] Canonical: https://deep-learning-study-room.vercel.app/read/is2/m2/mocks/m2-paper-v2-2--q4/ Reader: https://deep-learning-study-room.vercel.app/is2/#m2/mocks/m2-paper-v2-2/q4 Q1(a)+Q1(b) compulsory; any two of Q2–Q4. Answers immediately below each question. **Confirmed M2 pattern:** Q1(a) [2] and Q1(b) [2] are compulsory. Attempt any two of Q2, Q3 and Q4, each worth 3 marks. The paper offers 13 marks; you attempt 10. These are newly written practice questions, not claims about the exact upcoming paper. Solutions are directly below each question; the teaching working is deliberately longer than a concise exam answer. **Q4. A fuzzy set A has triangular membership Tri(15, 25, 35). Find membership at x = 32.** [3 marks · Practice] #### Given Left endpoint a = 15; peak b = 25; right endpoint c = 35; input x = 32. #### To find The membership degree μA(x), a number from 0 to 1. #### Formula and why it applies Locate x relative to a, b, c before choosing the branch. $$ \mu_A(x)=\begin{cases}0,&x\le a\text{ or }x\ge c\\\dfrac{x-a}{b-a},&a0 $$ #### Solution ##### Step 1. Choose numerator and denominator Numerator=140; denominator=160. The quantity after the bar identifies the denominator. ##### Step 2. Divide $$ \frac{140}{160}=0.875 $$ **Final answer.** 0.875. **Given P(A and B)=0.12 and P(B)=0.30, find P(A | B).** [2 marks · Practice] #### Given Joint event probability=0.12; evidence probability=0.30. #### To find The conditional probability. #### Formula and why it applies Keep only the evidence group. The denominator measures that group; the numerator measures the part satisfying both events. $$ P(A\mid B)=\frac{P(A\cap B)}{P(B)},\qquad P(B)>0 $$ #### Solution ##### Step 1. Choose numerator and denominator Numerator=0.12; denominator=0.3. The quantity after the bar identifies the denominator. ##### Step 2. Divide $$ \frac{0.12}{0.3}=0.4 $$ **Final answer.** 0.4. --- ## TEE · Bayes' rule with a tree of 1000 people Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-zero--bayes-tree/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-zero/bayes-tree Start here if probability is new: counting, AND/OR, conditional probability and Bayes with pictures A probability tree branches by successive conditions. Multiply along one complete path because all its events must occur together. Add different complete paths when either path qualifies. Counts such as 0.99 out of 1,000 are **expected counts** in the illustration, not fractions of an actual person. Scaling the same model to 100,000 gives 100 diseased people, 99 positive among them, and 1,998 false positives among 99,900 others. The posterior is 99/(99+1998)=99/2097≈0.0472103. **In one line:** to reverse a “given”, imagine a crowd, split it by the cause, then by the evidence, and count. ## The question A disease affects 1 person in 1000. A test is positive for 99% of sick people, and wrongly positive for 2% of healthy people. Your test is positive. How likely is it that you are sick? Swipe across the diagram to read its labels ↔ ![Imagine 1000 people (disease 1 in 1000, test catches 99%, false alarm 2%)](https://deep-learning-study-room.vercel.app/is2/fig/prob_tree.201d46f630.svg) Diagram: Imagine 1000 people (disease 1 in 1000, test catches 99%, false alarm 2%). Labels: Imagine 1000 people (disease 1 in 1000, test catches 99%, false alarm 2%) | 1000 people | 1 has disease | 0.001 × 1000 | 999 healthy | 0.999 × 1000 | 0.99 test + | 99% of 1 | 0.01 test − | 19.98 test + | 2% of 999 (false alarm) | 979.02 test − | All positives = 0.99 + 19.98 = 20.97 people; only 0.99 of them are ill → 0.99 / 20.97 = 0.047 (4.7%) ## Counting on the tree 1. Of 1000 people, 1 is sick and 999 are healthy. 2. The sick person tests positive with 99%: 0.99 people. 3. Healthy people test positive with 2%: 0.02 × 999 = 19.98 people. 4. Everyone who tests positive: 0.99 + 19.98 = 20.97 people. 5. Of those, sick: 0.99. So P(sick | positive) = 0.99 / 20.97 = 0.047. Only about 4.7%, because the healthy group is so much bigger that its small false-alarm rate still produces most positives. ## The same thing as a formula $$ P(\text{sick}\mid+)=\frac{P(+\mid\text{sick})\,P(\text{sick})}{P(+\mid\text{sick})P(\text{sick})+P(+\mid\text{healthy})P(\text{healthy})}=\frac{0.99\times0.001}{0.99\times0.001+0.02\times0.999}=0.047 $$ Each part of the formula is one branch of the tree. Names: P(sick) is the **prior** (before the test), P(+ | sick) the **likelihood**, the denominator the **evidence** (all positives), and the answer the **posterior** (after the test). **Yaad rakho:** Bayes = tree banao: pehle cause se baanto, phir evidence se. Answer = “jo humein chahiye” ÷ “saare jo evidence dikhate hain”. Next: the formal version and more examples in [Bayes' rule](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-prob--bayes-rule/). --- ## TEE · Why agents need probability (and the basic rules) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-prob--uncertainty/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-prob/uncertainty Unit 1 · Quantifying uncertainty, inference using the full joint distribution, Bayes rule **Action A pays 10 points with probability 0.6 and −2 points with probability 0.4. Action B always pays 4 points. Which has greater expected reward?** [2 marks · Worked example] #### Given Action A: reward 10 with probability 0.6, reward −2 with probability 0.4. Action B: reward 4 with probability 1. The two probabilities for A add to 1. #### To find Expected score for each action. #### Formula and why it applies $$ \mathbb E[R]=\sum_i P_iR_i $$ Expectation is a probability-weighted average. It need not equal any one actual outcome. #### Solution ##### Step 1. A: weight each outcome $$ 0.6\times10=6\qquad0.4\times(-2)=-0.8 $$ ##### Step 2. A: add $$ \mathbb E[R_A]=6-0.8=5.2 $$ ##### Step 3. B: certain outcome $$ \mathbb E[R_B]=1\times4=4 $$ ##### Step 4. Compare 5.2>4, so A has the greater expected reward. A single attempt can still return −2. **Final answer.** A:5.2; B:4. Choose A under an expected-reward criterion. Ω names the sample space (all outcomes). ω names one outcome. The symbol E or 𝔼 means expectation here; it does not mean an observed reward. A rational probabilistic decision uses both how likely outcomes are and how desirable they are. **In one line:** when an agent cannot be certain, it should hold a degree of belief between 0 and 1 for each possibility and update it with evidence; probability theory is the tool for that. ## Acting under uncertainty An agent is uncertain because of **partial observability** (it cannot see everything) and **non-determinism** (actions do not always have the same result). Pure logic fails in such domains. Example: “Toothache ⇒ Cavity” is false (gum disease also causes toothache), and listing every possible cause is impossible. Logic fails for three reasons: 1. **Laziness**: listing every antecedent and consequent is too much work. 2. **Theoretical ignorance**: science has no complete theory of the domain. 3. **Practical ignorance**: we cannot run every test on every patient. So the agent assigns a **degree of belief** (probability), e.g. “80% of toothache patients have cavities”. **Decision theory = probability theory + utility theory**: a rational agent chooses the action with the **maximum expected utility** (e.g. a taxi choosing when to leave for the airport balances the chance of missing the flight against waiting time). Sources of uncertainty in AI: incomplete data, noise, ambiguity, stochastic environments, model and algorithm limits. ## Basic notation | Term | Meaning / rule | | --- | --- | | Sample space Ω | All possible worlds; $$ 0\le P(\omega)\le1 $$ , $$ \sum_\omega P(\omega)=1 $$ | | Prior (unconditional) | $$ P(a) $$ : belief before evidence, e.g. $$ P(cavity)=0.2 $$ | | Posterior (conditional) | $$ P(a\mid b)=\dfrac{P(a\land b)}{P(b)} $$ , $$ P(b)>0 $$ | | Product rule | $$ P(a\land b)=P(a\mid b)P(b)=P(b\mid a)P(a) $$ | | Complement | $$ P(\lnot a)=1-P(a) $$ | | Inclusion–exclusion | $$ P(a\lor b)=P(a)+P(b)-P(a\land b) $$ | | Marginalisation | $$ P(Y)=\sum_z P(Y,z) $$ | | Total probability | $$ P(b)=P(b\mid a)P(a)+P(b\mid\lnot a)P(\lnot a) $$ | | Independence | $$ P(a\land b)=P(a)P(b) $$ , equivalently $$ P(a\mid b)=P(a) $$ | | Distribution | Bold $$ \mathbf P(Weather)=\langle0.6,0.1,0.29,0.01\rangle $$ lists all values | **Yaad rakho:** Logic “haan/na” bolta hai; probability “kitna yakeen” batati hai. Conditional = joint ÷ evidence. --- ## TEE · Inference using the full joint distribution (5-mark PYQs) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-prob--full-joint/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-prob/full-joint Unit 1 · Quantifying uncertainty, inference using the full joint distribution, Bayes rule A joint-probability table is a list of complete combinations. Three Yes/No variables have 2×2×2=8 combinations. “Marginalise” means ignore a variable by adding its possible cases. “Condition” means first restrict attention to the evidence cases, then divide by their total. For unnormalised weights .12 and .08, the total is .20; α=1/.20=5, so α×.12=.6 and α×.08=.4. α is only a rescaling constant. **In one line:** a full joint table lists the probability of every combination; to answer any question, add the cells where it is true, and for “given” questions divide by the cells of the evidence. ## Method 1. **Marginal** $$ P(a) $$ : add every cell in which a is true. 2. **Conditional** $$ P(a\mid e)=\dfrac{P(a\land e)}{P(e)} $$ : add cells where both hold, divide by the sum of cells where e holds. 3. **Normalisation**: $$ \mathbf P(A\mid e)=\alpha\,\mathbf P(A,e) $$ with $$ \alpha=1/P(e) $$ : compute the unnormalised values for every value of A, then scale them to sum to 1. **Question. For the dentist table below find P(cavity ∨ toothache), P(cavity | toothache) and P(¬cavity | toothache).** [class example · 3 marks] #### Given The complete joint table or cell values printed in this question. The rows and columns identify the event combinations. #### To find All requested marginal and conditional probabilities. #### Formula and why it applies Marginal: add cells satisfying the event. Conditional: retain the evidence cells and divide the joint-event sum by the evidence sum. $$ P(A\mid E)=\frac{P(A\cap E)}{P(E)},\quad P(E)>0 $$ #### Solution | | toothache, catch | toothache, ¬catch | ¬toothache, catch | ¬toothache, ¬catch | | --- | --- | --- | --- | --- | | **cavity** | 0.108 | 0.012 | 0.072 | 0.008 | | **¬cavity** | 0.016 | 0.064 | 0.144 | 0.576 | ##### Step 1. Disjunction $$ P(cavity\lor toothache)=0.108+0.012+0.072+0.008+0.016+0.064=0.28 $$ ##### Step 2. Conditionals $$ P(toothache)=0.108+0.012+0.016+0.064=0.2,\qquad P(cavity\mid toothache)=\frac{0.12}{0.2}=0.6,\qquad P(\lnot cavity\mid toothache)=\frac{0.08}{0.2}=0.4 $$ With α: $$ \mathbf P(Cavity\mid toothache)=\alpha\langle0.12,\,0.08\rangle=\langle0.6,\,0.4\rangle $$ , $$ \alpha=1/0.2=5 $$ . **Final answer.** P(cavity OR toothache)=0.28; P(cavity | toothache)=0.6; P(no cavity | toothache)=0.4. **Question. A supermarket recorded the joint distribution of Product type (Y: premium = 1, regular = 0), Payment mode (X: credit = 1, debit = 0) and Gender (Z: female = 1, male = 0). (a) If a customer bought a premium item, find the probability that (1) they are female, (2) they paid by credit card. (b) Find (1) P(premium) and (2) P(credit card). (c) Given a male customer paid by debit card, find the probability he bought a regular item.** [5 marks · Final Nov 2025 Q1a] #### Given The printed eight entries are reproduced below. Their total is 1.002, not 1. Therefore the table is inconsistent as a probability distribution. | | Credit, female | Credit, male | Debit, female | Debit, male | | --- | --- | --- | --- | --- | | **Premium (Y=1)** | 0.08 | 0.10 | 0.072 | 0.05 | | **Regular (Y=0)** | 0.10 | 0.15 | 0.25 | 0.20 | #### To find The five requested probabilities, stating how the inconsistent data are interpreted. #### Formula and why it applies For this worked solution, treat all printed entries as nonnegative weights and normalise by their total 1.002. This preserves the printed ratios. It is a declared interpretation, not a claim about the examiner’s intended correction. $$ P(E)=\frac{\text{sum of weights for }E}{\text{sum of all weights}} $$ $$ P(A\mid E)=\frac{\text{sum of weights for }A\cap E}{\text{sum of weights for }E} $$ #### Solution ##### Step 1. Validate the full table $$ 0.08+0.10+0.072+0.05+0.10+0.15+0.25+0.20=1.002 $$ Use total weight 1.002 consistently for unconditional probabilities. ##### Step 2. Premium row total $$ w(premium)=0.08+0.10+0.072+0.05=0.302 $$ ##### Step 3. Female given premium $$ P(female\mid premium)=\frac{0.08+0.072}{0.302}=\frac{0.152}{0.302}\approx0.503311 $$ ##### Step 4. Credit given premium $$ P(credit\mid premium)=\frac{0.08+0.10}{0.302}=\frac{0.18}{0.302}\approx0.596026 $$ ##### Step 5. Normalised premium marginal $$ P(premium)=\frac{0.302}{1.002}\approx0.301397 $$ ##### Step 6. Normalised credit marginal $$ P(credit)=\frac{0.08+0.10+0.10+0.15}{1.002}=\frac{0.43}{1.002}\approx0.429142 $$ ##### Step 7. Regular given male and debit $$ P(regular\mid male,debit)=\frac{0.20}{0.05+0.20}=0.8 $$ **Final answer.** Under common normalisation: (a) 0.503311 and 0.596026; (b) 0.301397 and 0.429142; (c) 0.8. **Alternative correction:** if 0.072 is corrected to 0.070, the total becomes 1 and the corresponding answers are (a) 0.50 and 0.60; (b) 0.30 and 0.43; (c) 0.8. This agrees with the interpretation used in the supplied key. State the correction; do not call raw totals 0.302 and 0.43 valid marginals of the original inconsistent table. **Question. For a disease Y with fever X as a symptom, a test Z is conducted. Joint probabilities: Disease (Y=1): fever & test+ 0.576, fever & test− 0.144, no fever & test+ 0.072, no fever & test− 0.016; No disease (Y=0): 0.008, 0.064, 0.012, 0.108. (1) If fever occurs and the test is positive, what is the probability of disease? (2) What is the probability that a random person tests positive? (3) Given no fever, what is the probability of still having the disease?** [5 marks · Re-exam Jan 2026 Q1a] #### Given The complete joint table or cell values printed in this question. The rows and columns identify the event combinations. #### To find All requested marginal and conditional probabilities. #### Formula and why it applies Marginal: add cells satisfying the event. Conditional: retain the evidence cells and divide the joint-event sum by the evidence sum. $$ P(A\mid E)=\frac{P(A\cap E)}{P(E)},\quad P(E)>0 $$ #### Solution ##### Step 1. (1) $$ P(Y{=}1\mid X{=}1,Z{=}1)=\frac{0.576}{0.576+0.008}=\frac{0.576}{0.584}=0.986 $$ ##### Step 2. (2) Add every test-positive cell $$ P(Z{=}1)=0.576+0.072+0.008+0.012=0.668 $$ ##### Step 3. (3) $$ P(X{=}0)=0.072+0.016+0.012+0.108=0.208,\qquad P(Y{=}1\mid X{=}0)=\frac{0.072+0.016}{0.208}=\frac{0.088}{0.208}=0.423 $$ **Final answer.** (1) 0.986; (2) 0.668; (3) 0.423. **Key correction.** The key writes 0.088/0.076 = 1.158 for part (3). A probability cannot exceed 1; the denominator must be P(no fever) = 0.208. **Yaad rakho:** “Given” wali cheez denominator mein jaati hai. Pehle evidence ke saare cells jodo, phir upar wale. --- ## TEE · Bayes’ rule: meaning, derivation and three complete examples Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-prob--bayes-rule/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-prob/bayes-rule Unit 1 · Quantifying uncertainty, inference using the full joint distribution, Bayes rule ## What Bayes’ rule reverses A **prior** is the probability of a cause before the new evidence. A **likelihood** is the probability of that evidence if the cause is true. A **posterior** is the probability of the cause after observing the evidence. These are different conditional directions: P(positive | disease) is not P(disease | positive). ## Derivation $$ P(A\cap B)=P(B\mid A)P(A)=P(A\mid B)P(B) $$ Divide both sides by P(B), provided that P(B)>0. $$ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)} $$ If A and its complement are the only two possibilities, evidence B can arise by either path. Add both paths in the denominator. $$ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid\neg A)P(\neg A)} $$ | Symbol | Read it as | | --- | --- | | P(A) | Prior: how common is A? | | P(B | A) | Likelihood: if A is true, how likely is B? | | P(B) | Evidence: how common is B through all possible causes? | | P(A | B) | Posterior: among B cases, what fraction come from A? | **A hypothetical population has a disease prevalence of 1 in 1000. A test is positive in 99% of diseased cases and 2% of healthy cases. Given a positive result, calculate the probability of disease.** [4 marks · Worked probability example] #### Given A = disease; B = positive test. P(A)=0.001; P(B|A)=0.99; P(B|not A)=0.02. #### To find The reversed conditional probability P(A | B). #### Formula and why it applies The question gives evidence given each cause, but asks for cause given evidence. Bayes’ rule combines the prior with that likelihood and divides by all ways of obtaining the evidence. $$ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid\neg A)P(\neg A)} $$ #### Solution ##### Step 1. Find the other prior $$ P(\neg A)=1-0.001=0.999 $$ ##### Step 2. Evidence through the requested cause Multiply the prior for disease by the likelihood within that group. $$ P(A\cap B)=0.001\times0.99=0.00099 $$ ##### Step 3. Evidence through the other cause Now include the path through healthy. $$ P(\neg A\cap B)=0.999\times0.02=0.01998 $$ ##### Step 4. Add both paths for total evidence $$ P(B)=0.00099+0.01998=0.02097 $$ ##### Step 5. Normalise the requested path $$ P(A\mid B)=\frac{0.00099}{0.02097}\approx0.0472103 $$ **Final answer.** 0.0472103 ≈ 4.721030043%. **In a hypothetical population, 49% are female and 51% male. 8% of females and 12% of males are classified high-risk in the supplied data. Given that a person is high-risk, find the probability that the person is male.** [3 marks · Worked probability example] #### Given A = male; B = high-risk category. P(A)=0.51; P(B|A)=0.12; P(B|not A)=0.08. #### To find The reversed conditional probability P(A | B). #### Formula and why it applies The question gives evidence given each cause, but asks for cause given evidence. Bayes’ rule combines the prior with that likelihood and divides by all ways of obtaining the evidence. $$ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid\neg A)P(\neg A)} $$ #### Solution ##### Step 1. Find the other prior $$ P(\neg A)=1-0.51=0.49 $$ ##### Step 2. Evidence through the requested cause Multiply the prior for male by the likelihood within that group. $$ P(A\cap B)=0.51\times0.12=0.0612 $$ ##### Step 3. Evidence through the other cause Now include the path through female. $$ P(\neg A\cap B)=0.49\times0.08=0.0392 $$ ##### Step 4. Add both paths for total evidence $$ P(B)=0.0612+0.0392=0.1004 $$ ##### Step 5. Normalise the requested path $$ P(A\mid B)=\frac{0.0612}{0.1004}\approx0.609561753 $$ **Final answer.** 0.609561753 ≈ 60.956175299%. **Machine A makes 60% of items and Machine B makes 40%. Defect rates are 4% for A and 1% for B. A randomly chosen item is defective. Find the probability that Machine A made it.** [3 marks · Worked probability example] #### Given Cause A = Machine A; evidence B = defective item. P(A)=0.60; P(B|A)=0.04; P(B|not A)=0.01. #### To find The reversed conditional probability P(A | B). #### Formula and why it applies The question gives evidence given each cause, but asks for cause given evidence. Bayes’ rule combines the prior with that likelihood and divides by all ways of obtaining the evidence. $$ P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid\neg A)P(\neg A)} $$ #### Solution ##### Step 1. Find the other prior $$ P(\neg A)=1-0.6=0.4 $$ ##### Step 2. Evidence through the requested cause Multiply the prior for Machine A by the likelihood within that group. $$ P(A\cap B)=0.6\times0.04=0.024 $$ ##### Step 3. Evidence through the other cause Now include the path through Machine B. $$ P(\neg A\cap B)=0.4\times0.01=0.004 $$ ##### Step 4. Add both paths for total evidence $$ P(B)=0.024+0.004=0.028 $$ ##### Step 5. Normalise the requested path $$ P(A\mid B)=\frac{0.024}{0.028}\approx0.857142857 $$ **Final answer.** 0.857142857 ≈ 85.714285714%. A rare prior can make a positive result much less conclusive than its sensitivity suggests. The posterior still depends on the actual prior and both test rates; rarity alone does not determine the answer. **Yaad rakho:** Pehle requested cause ka path multiply. Neeche evidence tak pahunchne wale saare paths add. --- ## TEE · The naive Bayes model (worked on PlayTennis) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-nb--model/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-nb/model Unit 1 · Naive Bayes model · 10-mark PYQ in both papers ## What the data and the question represent A **training row** is one past case whose correct class is known. Its **features** are the information used to predict that class. In the flu exercise, Fever, Cough and Temperature are features; Flu Yes/No is the class. A **query** is the new case whose features are known but whose class must be predicted. A **prior** is the class share before using the query features. A **likelihood** asks how common one query feature is inside that class. A **posterior** is the class probability after accounting for the query. The naive assumption makes the feature likelihoods multiply once the class is fixed; it does not assert that all features are independent in the whole population. | Read this | Meaning | Concrete example | | --- | --- | --- | | Π likelihoods | Multiply every feature factor | With factors .5,.2,.4: .5×.2=.1; .1×.4=.04 | | score(C) | An unnormalised class weight | Prior .3 times .04 gives .012 | | max | The largest numerical score | max(.012,.008)=.012 | | argmax | The label attaining the largest score | The winning label is the class with score .012 | | Laplace denominator N꜀+k | Class count plus number of possible values for this feature | Temperature has three values, so k=3; a Yes/No feature has k=2 | **In one line:** assume the features are independent given the class; then score each class as prior × product of feature likelihoods and pick the largest. ## Definition **Naive Bayes** is a supervised probabilistic classifier based on Bayes' rule with the **naive assumption** that all features are conditionally independent given the class: $$ P(C\mid a_1,\dots,a_n)\;\propto\;P(C)\prod_{i=1}^{n}P(a_i\mid C),\qquad v_{NB}=\arg\max_{c}\,P(c)\prod_iP(a_i\mid c) $$ “Naive” because real features are often correlated; it still works very well for spam filtering, sentiment analysis and medical screening. ## Steps (as taught) 1. **Priors**: $$ P(c)= $$ (rows of class c) / (total rows). 2. **Likelihood tables**: for each attribute value, $$ P(a_i\mid c)= $$ (rows of class c with that value) / (rows of class c). 3. **Score** each class for the new instance and choose the larger. Optionally normalise: divide each score by their sum. **Zero-frequency problem.** If a value never occurs with a class, that factor is 0 and the whole score is 0. The main unsmoothed solutions below leave a zero count as zero. Apply smoothing when it is specified, consistently in every class. Textbooks fix it with **Laplace smoothing**: $$ P(a\mid c)=\dfrac{\text{count}+1}{N_c+k} $$ , k = number of values of the attribute. Mention it in one line; it can change the answer. **Question. Using the 14-day PlayTennis data (9 Yes, 5 No), classify ⟨Outlook = Sunny, Temperature = Cool, Humidity = High, Wind = Strong⟩.** [class example · 6 marks] #### Given Class counts: Yes=9, No=5. Within Yes, counts for Sunny/Cool/High/Strong are 2/3/3/3; within No they are 3/1/4/3. Classify Sunny, Cool, High, Strong. #### To find Both class scores and the predicted class. #### Formula and why it applies Conditional independence given the class allows the product of feature likelihoods. $$ score(C)=P(C)\prod_jP(a_j\mid C) $$ Divide each score by their sum only after computing both. #### Solution ##### Step 1. Priors $$ P(yes)=\tfrac9{14}\approx0.643,\qquad P(no)=\tfrac5{14}\approx0.357 $$ ##### Step 2. Likelihoods needed | | Sunny | Cool | High | Strong | | --- | --- | --- | --- | --- | | Yes | 2/9 | 3/9 | 3/9 | 3/9 | | No | 3/5 | 1/5 | 4/5 | 3/5 | ##### Step 3. Scores $$ v(yes)=\tfrac9{14}\cdot\tfrac29\cdot\tfrac39\cdot\tfrac39\cdot\tfrac39\approx0.005291005,\qquad v(no)=\tfrac5{14}\cdot\tfrac35\cdot\tfrac15\cdot\tfrac45\cdot\tfrac35\approx0.020571429 $$ ##### Step 4. Normalise $$ P(yes\mid x)=\frac{1/189}{1/189+18/875}\approx0.204582651,\qquad P(no\mid x)\approx0.795417349 $$ **Final answer.** PlayTennis = **No** (0.795). **Yaad rakho:** Naive Bayes = prior × (har feature ka fraction) ka guna. Jo bada, wohi class. --- ## TEE · Solved PYQ: naive Bayes for flu (Final 2025, 10 marks) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-nb--flu/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-nb/flu Unit 1 · Naive Bayes model · 10-mark PYQ in both papers **Question. A hospital wants to predict flu from Fever (Yes/No), Cough (Yes/No) and Temperature Level (Low/Normal/High). Training data: (Yes, Yes, High, Yes), (No, No, Normal, No), (Yes, No, Low, No), (Yes, Yes, High, Yes), (No, Yes, Normal, No), (Yes, Yes, High, Yes), (No, No, Low, No), (Yes, No, Normal, No). Using naive Bayes, predict whether a patient with Fever = Yes, Cough = Yes, Temperature = Normal has flu.** [10 marks · Final Nov 2025 Q4b] #### Given The full training data are in Step 1. Query: Fever=Yes, Cough=Yes, Temperature=Normal. The main calculation uses unsmoothed counts. #### To find Both class scores and the predicted class. #### Formula and why it applies $$ score(C)=P(C)\prod_jP(a_j\mid C) $$ C is a class label. Each aⱼ is one query feature value. First count inside a class, then multiply the prior and all three likelihoods. #### Solution ##### Step 1. Write the training rows as a table | Row | Fever | Cough | Temperature | Class | | --- | --- | --- | --- | --- | | 1 | Yes | Yes | High | Yes | | 2 | No | No | Normal | No | | 3 | Yes | No | Low | No | | 4 | Yes | Yes | High | Yes | | 5 | No | Yes | Normal | No | | 6 | Yes | Yes | High | Yes | | 7 | No | No | Low | No | | 8 | Yes | No | Normal | No | ##### Step 2. Separate the two class groups | Class | Rows in this group | Class count | Prior | | --- | --- | --- | --- | | Yes | 1, 4, 6 | 3 | 3/8 = 0.375 | | No | 2, 3, 5, 7, 8 | 5 | 5/8 = 0.625 | ##### Step 3. For class Yes, count Fever=Yes Look only at class Yes rows 1, 4, 6. The matching rows are 1, 4, 6. $$ P(\text{Fever}=\text{Yes}\mid C=\text{Yes})=\frac{3}{3}=1 $$ ##### Step 4. For class Yes, count Cough=Yes Look only at class Yes rows 1, 4, 6. The matching rows are 1, 4, 6. $$ P(\text{Cough}=\text{Yes}\mid C=\text{Yes})=\frac{3}{3}=1 $$ ##### Step 5. For class Yes, count Temperature=Normal Look only at class Yes rows 1, 4, 6. No row in this class has that feature value. $$ P(\text{Temperature}=\text{Normal}\mid C=\text{Yes})=\frac{0}{3}=0 $$ ##### Step 6. Multiply the Yes score one factor at a time $$ P(Yes)=3/8=0.375 $$ $$ 0.375\times1=0.375 $$ $$ 0.375\times1=0.375 $$ $$ 0.375\times0=0 $$ ##### Step 7. For class No, count Fever=Yes Look only at class No rows 2, 3, 5, 7, 8. The matching rows are 3, 8. $$ P(\text{Fever}=\text{Yes}\mid C=\text{No})=\frac{2}{5}=0.4 $$ ##### Step 8. For class No, count Cough=Yes Look only at class No rows 2, 3, 5, 7, 8. The matching rows are 5. $$ P(\text{Cough}=\text{Yes}\mid C=\text{No})=\frac{1}{5}=0.2 $$ ##### Step 9. For class No, count Temperature=Normal Look only at class No rows 2, 3, 5, 7, 8. The matching rows are 2, 5, 8. $$ P(\text{Temperature}=\text{Normal}\mid C=\text{No})=\frac{3}{5}=0.6 $$ ##### Step 10. Multiply the No score one factor at a time $$ P(No)=5/8=0.625 $$ $$ 0.625\times0.4=0.25 $$ $$ 0.25\times0.2=0.05 $$ $$ 0.05\times0.6=0.03 $$ ##### Step 11. Compare the scores $$ score(Yes)=0\qquad score(No)=0.03 $$ The larger score belongs to No. Normalise only if a posterior probability is requested: divide each score by their total 0.03. **Final answer.** Predict No. Yes score=0; No score=0.03. The zero Yes score is caused by an absent training combination. If Laplace smoothing is explicitly required, apply it to all feature likelihoods in both classes. That is a different calculation and may change the prediction; do not mix smoothed and unsmoothed sides. **Yaad rakho:** Class ke rows alag karo. Har feature ka count usi class ke total se divide. Saare factors multiply karo. --- ## TEE · Solved PYQ: naive Bayes for a job offer (Re-exam 2026, 10 marks) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-nb--job-offer/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-nb/job-offer Unit 1 · Naive Bayes model · 10-mark PYQ in both papers **Question. Predict whether a candidate accepts a job offer from Experience (Fresher/Mid/Senior), Expected Salary (Low/Medium/High) and Company Reputation (Good/Average). Data: (Fresher, Low, Good, Yes), (Mid, High, Good, Yes), (Senior, High, Average, No), (Fresher, Medium, Average, Yes), (Senior, Low, Good, No), (Mid, Medium, Good, Yes), (Fresher, High, Average, No), (Senior, Medium, Average, No). New candidate: Mid, Low, Good.** [10 marks · Re-exam Jan 2026 Q4b] #### Given The full training data are in Step 1. Query: Experience=Mid, Salary=Low, Reputation=Good. The main calculation uses unsmoothed counts. #### To find Both class scores and the predicted class. #### Formula and why it applies $$ score(C)=P(C)\prod_jP(a_j\mid C) $$ C is a class label. Each aⱼ is one query feature value. First count inside a class, then multiply the prior and all three likelihoods. #### Solution ##### Step 1. Write the training rows as a table | Row | Experience | Salary | Reputation | Class | | --- | --- | --- | --- | --- | | 1 | Fresher | Low | Good | Yes | | 2 | Mid | High | Good | Yes | | 3 | Senior | High | Average | No | | 4 | Fresher | Medium | Average | Yes | | 5 | Senior | Low | Good | No | | 6 | Mid | Medium | Good | Yes | | 7 | Fresher | High | Average | No | | 8 | Senior | Medium | Average | No | ##### Step 2. Separate the two class groups | Class | Rows in this group | Class count | Prior | | --- | --- | --- | --- | | Yes | 1, 2, 4, 6 | 4 | 4/8 = 0.5 | | No | 3, 5, 7, 8 | 4 | 4/8 = 0.5 | ##### Step 3. For class Yes, count Experience=Mid Look only at class Yes rows 1, 2, 4, 6. The matching rows are 2, 6. $$ P(\text{Experience}=\text{Mid}\mid C=\text{Yes})=\frac{2}{4}=0.5 $$ ##### Step 4. For class Yes, count Salary=Low Look only at class Yes rows 1, 2, 4, 6. The matching rows are 1. $$ P(\text{Salary}=\text{Low}\mid C=\text{Yes})=\frac{1}{4}=0.25 $$ ##### Step 5. For class Yes, count Reputation=Good Look only at class Yes rows 1, 2, 4, 6. The matching rows are 1, 2, 6. $$ P(\text{Reputation}=\text{Good}\mid C=\text{Yes})=\frac{3}{4}=0.75 $$ ##### Step 6. Multiply the Yes score one factor at a time $$ P(Yes)=4/8=0.5 $$ $$ 0.5\times0.5=0.25 $$ $$ 0.25\times0.25=0.0625 $$ $$ 0.0625\times0.75=0.046875 $$ ##### Step 7. For class No, count Experience=Mid Look only at class No rows 3, 5, 7, 8. No row in this class has that feature value. $$ P(\text{Experience}=\text{Mid}\mid C=\text{No})=\frac{0}{4}=0 $$ ##### Step 8. For class No, count Salary=Low Look only at class No rows 3, 5, 7, 8. The matching rows are 5. $$ P(\text{Salary}=\text{Low}\mid C=\text{No})=\frac{1}{4}=0.25 $$ ##### Step 9. For class No, count Reputation=Good Look only at class No rows 3, 5, 7, 8. The matching rows are 5. $$ P(\text{Reputation}=\text{Good}\mid C=\text{No})=\frac{1}{4}=0.25 $$ ##### Step 10. Multiply the No score one factor at a time $$ P(No)=4/8=0.5 $$ $$ 0.5\times0=0 $$ $$ 0\times0.25=0 $$ $$ 0\times0.25=0 $$ ##### Step 11. Compare the scores $$ score(Yes)=0.046875\qquad score(No)=0 $$ The larger score belongs to Yes. Normalise only if a posterior probability is requested: divide each score by their total 0.046875. **Final answer.** Predict Yes. Yes score=0.046875; No score=0. **Official key note.** The key gives No = 0.0068 and P(Yes) = 0.873. That No value comes from Laplace smoothing (1/7 × 2/7 × 2/6 × 0.5), but the key did not smooth the Yes side. If you smooth, smooth both: Yes = 0.5 × 3/7 × 2/7 × 4/6 = 0.0408, No = 0.0068, P(Yes) = 0.857. The prediction is Yes either way. **Yaad rakho:** Class ke rows alag karo. Har feature ka count usi class ke total se divide. Saare factors multiply karo. --- ## TEE · Bayesian networks: structure, CPTs and the chain rule Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-bn--semantics/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-bn/semantics Unit 1 · Semantics of Bayesian networks and exact inference · 10-mark PYQ in both papers ## A Bayesian network starts as a diagram of variables A **node** is a variable, such as Rain. An **edge** is an arrow representing a direct dependency in the model. A **parent** has an arrow into a node. A **root** has no parents. **Directed acyclic graph (DAG)** means arrows have directions and you cannot follow them around a loop back to the starting node. The graph alone does not establish real-world causation. A **CPT** is a conditional probability table: for each parent configuration, it states the probabilities of the child’s values. A **Boolean** variable has two values, true and false. If a CPT gives P(Wet=true|Rain=true)=.9, its complementary entry is P(Wet=false|Rain=true)=1−.9=.1. **In Rain → Wet, P(Rain)=0.2, P(Wet|Rain)=0.9 and P(Wet|no Rain)=0.1. Find P(Rain AND Wet), P(Wet), and P(Rain|Wet).** [2 marks · Worked example] #### Given P(Rain)=0.2; P(Wet|Rain)=0.9; P(Wet|no Rain)=0.1. Each variable has two possible values: true or false. Rain is the root; Wet depends on Rain. #### To find A joint probability, a marginal probability and a posterior. #### Formula and why it applies Multiply down one complete case, add the alternative ways to get Wet, then divide by the evidence probability. #### Solution ##### Step 1. Rain and wet $$ P(R,W)=0.2\times0.9=0.18 $$ ##### Step 2. No rain and wet $$ P(\neg R,W)=(1-0.2)\times0.1=0.8\times0.1=0.08 $$ ##### Step 3. Total wet $$ P(W)=0.18+0.08=0.26 $$ ##### Step 4. Restrict to the wet cases $$ P(R\mid W)=0.18/0.26=9/13\approx0.692307692 $$ **Final answer.** Joint .18; marginal .26; posterior≈.692308. **In one line:** a Bayesian network is a DAG of random variables in which each node stores P(node | its parents); multiplying those tables gives the full joint distribution. ## Definition A **Bayesian network** (belief network) is a directed acyclic graph in which: 1. each **node** is a random variable; 2. a directed **edge** X → Y means the model conditions Y on X (X is a parent of Y); a causal interpretation needs additional assumptions; there are no directed cycles; 3. each node has a **conditional probability table (CPT)** $$ \mathbf P(X_i\mid Parents(X_i)) $$ . A root node stores its prior. Swipe across the diagram to read its labels ↔ ![Burglary network (Russell and Norvig)](https://deep-learning-study-room.vercel.app/is2/fig/bn_burglary.9802d6d53d.svg) Diagram: Burglary network (Russell and Norvig). Labels: Burglary network (Russell and Norvig) | Burglary | P(B) = 0.001 | Earthquake | P(E) = 0.002 | Alarm | P(A|B,E): .95 · .94 · .29 · .001 | (TT · TF · FT · FF) | JohnCalls | P(J|A) = .90, P(J|¬A) = .05 | MaryCalls | P(M|A) = .70, P(M|¬A) = .01 ## Semantics: the chain rule for Bayesian networks $$ P(x_1,\dots,x_n)=\prod_{i=1}^{n}P\big(x_i\mid parents(X_i)\big) $$ So the network *is* a compact description of the full joint distribution. For the burglary network: $$ P(J,M,A,B,E)=P(J\mid A)\,P(M\mid A)\,P(A\mid B,E)\,P(B)\,P(E) $$ **Conditional independence:** each node is conditionally independent of its non-descendants given its parents (e.g. JohnCalls is independent of Burglary once Alarm is known). **Compactness:** a Boolean node with k Boolean parents needs $$ 2^k $$ numbers. The burglary network needs 1 + 1 + 4 + 2 + 2 = 10 numbers instead of $$ 2^5-1=31 $$ for the full joint. ## Building a network 1. Choose the variables. 2. Choose an acyclic variable order; when a causal model is justified, cause-before-effect is often useful. 3. For each variable, choose as parents a suitable set of earlier variables that captures its conditional dependencies. 4. Write its CPT. **Yaad rakho:** Har node sirf apne parents se baat karta hai. Joint = sab nodes ka P(node | parents) ka guna. --- ## TEE · Inference in Bayesian networks: enumeration, α and variable elimination Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-bn--inference/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-bn/inference Unit 1 · Semantics of Bayesian networks and exact inference · 10-mark PYQ in both papers ## Expand the hidden-variable calculation before the compact formula For burglary given both calls, B is the query, J=true and M=true are evidence, and E and A are hidden. “Hidden” only means their values are not supplied in the query. A sum over a Boolean variable adds its true and false cases. | Calculation | Burglary true | Burglary false | | --- | --- | --- | | Average alarm probability over E | .95×.002 + .94×.998 = .94002 | .29×.002 + .001×.998 = .001578 | | No-alarm probability | 1−.94002=.05998 | 1−.001578=.998422 | | Both calls if alarm | .9×.7=.63 | .9×.7=.63 | | Both calls if no alarm | .05×.01=.0005 | .05×.01=.0005 | | Calls and this B value | .001×(.63×.94002 + .0005×.05998)=.00059224259 | .999×(.63×.001578 + .0005×.998422)=.001491857649 | Add the final two weights: .00059224259+.001491857649=.002084100239. Divide the burglary weight by that total: .00059224259/.002084100239≈.284171835. No step equates P(B|calls) with P(calls|B); the prior P(B) is included explicitly. **In one line:** to answer P(query | evidence), sum the chain-rule product over every hidden variable, do it for each value of the query, and normalise with α. ## Three kinds of question | Question type | Method | | --- | --- | | Every variable's value given | Multiply the CPT entries (chain rule) | | One marginal, e.g. P(J) | Sum over hidden variables; no α | | Query given evidence, e.g. P(B | j, m) | Enumerate for each value of B, then $$ \alpha=1/\text{(sum)} $$ | **Question. In the burglary network find (a) P(j, m, a, ¬b, ¬e), (b) P(JohnCalls), (c) P(Burglary | John and Mary call).** [class examples · 10 marks] For the burglary example: B and E are independent roots with P(B)=0.001 and P(E)=0.002. Alarm A has parents B,E. John J and Mary M each have parent A. P(A|B,E)=0.95, P(A|B,¬E)=0.94, P(A|¬B,E)=0.29, P(A|¬B,¬E)=0.001. P(J|A)=0.9, P(J|¬A)=0.05; P(M|A)=0.7, P(M|¬A)=0.01. False probabilities are the complements of the listed true probabilities. #### Given The network, root probabilities and conditional probability tables stated in the question. #### To find The requested joint, marginal or evidence-conditioned probability. #### Formula and why it applies A Bayesian network factorises the joint using each node conditioned on its parents. Sum over hidden values; normalise for evidence. $$ P(x_1,\ldots,x_n)=\prod_iP(x_i\mid parents_i) $$ #### Solution ##### (a) Chain rule $$ P=P(j\mid a)P(m\mid a)P(a\mid\lnot b,\lnot e)P(\lnot b)P(\lnot e)=0.90\times0.70\times0.001\times0.999\times0.998=0.000628 $$ ##### (b) Marginal: first P(a) $$ P(a)=0.95(0.001)(0.002)+0.94(0.001)(0.998)+0.29(0.999)(0.002)+0.001(0.999)(0.998)=0.002516 $$ $$ P(j)=P(j\mid a)P(a)+P(j\mid\lnot a)P(\lnot a)=0.90(0.002516)+0.05(0.997484)=0.0521 $$ ##### (c) Enumeration with α (hidden: E, A) $$ P(b\mid j,m)=\alpha\,P(b)\sum_aP(j\mid a)P(m\mid a)\sum_eP(a\mid b,e)P(e) $$ $$ =\alpha\,(0.001)\big[0.9(0.7)(0.95\times0.002+0.94\times0.998)+0.05(0.01)(0.05\times0.002+0.06\times0.998)\big]=\alpha\times0.000592 $$ $$ P(\lnot b\mid j,m)=\alpha\,(0.999)\big[0.9(0.7)(0.29\times0.002+0.001\times0.998)+0.05(0.01)(0.71\times0.002+0.999\times0.998)\big]=\alpha\times0.001492 $$ $$ \alpha=\frac1{0.000592+0.001492}=479.8\;\Rightarrow\;P(b\mid j,m)=0.284,\;P(\lnot b\mid j,m)=0.716 $$ **Final answer.** (a) 0.000628; (b) 0.0521; (c) 0.284. Even when both neighbours call, a burglary is only about 28% likely because burglaries are rare. **Slide note.** One slide prints .71 for $$ P(\lnot a\mid b,\lnot e) $$ ; it is 1 − 0.94 = 0.06 (corrected by hand on the slide). ## Variable elimination (why it is faster) Enumeration repeats the same products many times (exponential, $$ O(2^n) $$ ). **Variable elimination** pushes each sum as far inside as possible, computes it once and stores the result as a **factor**; hidden variables are “summed out” one at a time: $$ P(J)=\sum_AP(J\mid A)\sum_MP(M\mid A)\sum_BP(B)\underbrace{\sum_EP(A\mid B,E)P(E)}_{f_1(A,B)} $$ Any sum of a CPT over its own variable is 1 (e.g. $$ \sum_MP(M\mid A)=1 $$ ), so irrelevant (barren) nodes drop out. **Question. Heart-disease network: Exercise → HD ← Diet; HD → BP. P(E) = 0.7, P(D = healthy) = 0.25; P(HD | E, D) = 0.25 (E, healthy), 0.45 (E, unhealthy), 0.55 (¬E, healthy), 0.75 (¬E, unhealthy); P(BP high | HD) = 0.85, P(BP high | ¬HD) = 0.2. A patient has high BP. Find P(HD | BP).** [class example · 6 marks] #### Given The network, root probabilities and conditional probability tables stated in the question. #### To find The requested joint, marginal or evidence-conditioned probability. #### Formula and why it applies A Bayesian network factorises the joint using each node conditioned on its parents. Sum over hidden values; normalise for evidence. $$ P(x_1,\ldots,x_n)=\prod_iP(x_i\mid parents_i) $$ #### Solution ##### Step 1. P(HD) by summing out E and D $$ P(hd)=0.7(0.25\times0.25+0.45\times0.75)+0.3(0.55\times0.25+0.75\times0.75)=0.7(0.4)+0.3(0.7)=0.49 $$ ##### Step 2. Unnormalised $$ \alpha\,P(bp\mid hd)P(hd)=\alpha(0.85)(0.49)=\alpha\,0.4165,\qquad\alpha\,P(bp\mid\lnot hd)P(\lnot hd)=\alpha(0.2)(0.51)=\alpha\,0.102 $$ ##### Step 3. Normalise $$ \alpha=\frac1{0.5185},\quad P(hd\mid bp)=0.803,\quad P(\lnot hd\mid bp)=0.197 $$ **Final answer.** 0.803. **Yaad rakho:** Hidden variables par sum, query ki dono values nikaalo, α se 1 banao. Sirf marginal (P(J)) ho to α nahi. --- ## TEE · Solved PYQ: wet grass given no rain (Final 2025, 10 marks) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-bn--sprinkler/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-bn/sprinkler Unit 1 · Semantics of Bayesian networks and exact inference · 10-mark PYQ in both papers The following table enumerates the values of Cloudy and Sprinkler that were not fixed by the question. Two Boolean choices give four rows: true/true, true/false, false/true, false/false. Each row is one mutually exclusive complete case. Read the CPT entry using that row’s parent values, multiply its factors, and add the resulting row weights. The denominator must include all cases matching the evidence, including both possible values of the queried outcome. **Question. Cloudy → Sprinkler, Cloudy → Rain, Sprinkler and Rain → Wet Grass. P(Cloudy) = 0.5; P(Sprinkler | C) = 0.10, P(Sprinkler | ¬C) = 0.50; P(Rain | C) = 0.80, P(Rain | ¬C) = 0.20; P(Wet | S, R) = 0.99, P(Wet | S, ¬R) = 0.90, P(Wet | ¬S, R) = 0.90, P(Wet | ¬S, ¬R) = 0.01. Calculate the probability of the grass being wet given it doesn't rain.** [10 marks · Final Nov 2025 Q3a] #### Given C→S, C→R, (S,R)→W. Root and conditional probabilities are listed in the question. Evidence R=false. #### To find P(W=true | R=false). Hidden variables C and S must be summed out. #### Formula and why it applies Use the network joint and divide by the evidence probability. $$ P(W\mid\neg R)=\frac{\sum_{c,s}P(c)P(s\mid c)P(\neg R\mid c)P(W\mid s,\neg R)}{\sum_cP(c)P(\neg R\mid c)} $$ #### Solution Swipe across the diagram to read its labels ↔ ![Final 2025 Q3a: sprinkler network](https://deep-learning-study-room.vercel.app/is2/fig/bn_sprinkler.a251583272.svg) Diagram: Final 2025 Q3a: sprinkler network. Labels: Final 2025 Q3a: sprinkler network | Cloudy | P(C) = 0.5 | Sprinkler | P(S|C) = 0.1, P(S|¬C) = 0.5 | Rain | P(R|C) = 0.8, P(R|¬C) = 0.2 | Wet grass | P(W|S,R): .99 · .90 · .90 · .01 | (TT · TF · FT · FF) ##### Step 1. Joint by the chain rule $$ P(C,S,R,W)=P(C)\,P(S\mid C)\,P(R\mid C)\,P(W\mid S,R) $$ ##### Step 2. Sum over C and S with R = false, W = true | C | S | P(C) | P(S|C) | P(¬R|C) | P(W|S,¬R) | Product | | --- | --- | --- | --- | --- | --- | --- | | T | T | 0.5 | 0.1 | 0.2 | 0.90 | 0.0090 | | T | F | 0.5 | 0.9 | 0.2 | 0.01 | 0.0009 | | F | T | 0.5 | 0.5 | 0.8 | 0.90 | 0.1800 | | F | F | 0.5 | 0.5 | 0.8 | 0.01 | 0.0020 | | **P(W, ¬R)** | | | | | | **0.1919** | ##### Step 3. Evidence $$ P(\lnot R)=P(C)P(\lnot R\mid C)+P(\lnot C)P(\lnot R\mid\lnot C)=0.5(0.2)+0.5(0.8)=0.5 $$ ##### Step 4. Divide $$ P(W\mid\lnot R)=\frac{0.1919}{0.5}=0.3838 $$ **Final answer.** P(grass wet | no rain) ≈ **0.384**. Without rain, the grass is wet mainly when the sprinkler runs (more likely on non-cloudy days). **Keep the requested conditional probability separate.** The supplied key discusses P(Wet) and P(Rain | Wet), whereas this question asks P(Wet | no rain). The answer to the printed question is 0.3838. For comparison, the exact marginal from the printed CPT is calculated below; it is not 0.6505. #### Optional comparison: calculate P(Wet) from the same network The four no-rain cases already total 0.1919. Now add the four rain cases, keeping the same Cloudy/Sprinkler order. $$ P(W,R)=0.5(0.1)(0.8)(0.99)+0.5(0.9)(0.8)(0.90)+0.5(0.5)(0.2)(0.99)+0.5(0.5)(0.2)(0.90) $$ $$ P(W,R)=0.0396+0.324+0.0495+0.045=0.4581 $$ $$ P(W)=P(W,R)+P(W,\neg R)=0.4581+0.1919=0.6500 $$ $$ P(R\mid W)=\frac{0.4581}{0.6500}\approx0.704769231 $$ **Meaning:** the unconditional chance of Wet is 0.65, the chance of Rain among wet cases is about 0.704769, and the requested chance of Wet among no-rain cases is 0.3838. These are three different questions. **Trap.** Do not plug in the marginal P(Sprinkler) = 0.3; sprinkler and rain both depend on Cloudy, so you must sum over Cloudy (given no rain, P(S) = 0.42). --- ## TEE · Solved PYQ: fever given no flu (Re-exam 2026, 10 marks) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-bn--flu-network/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-bn/flu-network Unit 1 · Semantics of Bayesian networks and exact inference · 10-mark PYQ in both papers The following table enumerates the values of Weather and Cough that were not fixed by the question. Two Boolean choices give four rows: true/true, true/false, false/true, false/false. Each row is one mutually exclusive complete case. Read the CPT entry using that row’s parent values, multiply its factors, and add the resulting row weights. The denominator must include all cases matching the evidence, including both possible values of the queried outcome. **Question. Weather → Flu, Weather → Cough, Flu and Cough → Fever. P(Weather) = 0.75; P(Flu | W) = 0.2, P(Flu | ¬W) = 0.5; P(Cough | W) = 0.70, P(Cough | ¬W) = 0.25; P(Fever | Flu, Cough) = 0.60, P(Fever | Flu, ¬Cough) = 0.02, P(Fever | ¬Flu, Cough) = 0.60, P(Fever | ¬Flu, ¬Cough) = 0.01. Calculate the probability of a person having fever given he doesn't have flu.** [10 marks · Re-exam Jan 2026 Q3a] #### Given W→F, W→C, (F,C)→V. The complete probabilities are in the question. Evidence F=false. #### To find P(V=true | F=false), summing hidden W and C. #### Formula and why it applies Use the network joint and normalise by the no-Flu evidence. $$ P(V\mid\neg F)=\frac{\sum_{w,c}P(w)P(\neg F\mid w)P(c\mid w)P(V\mid\neg F,c)}{\sum_wP(w)P(\neg F\mid w)} $$ #### Solution Swipe across the diagram to read its labels ↔ ![Re-exam 2026 Q3a: weather–flu–cough–fever network](https://deep-learning-study-room.vercel.app/is2/fig/bn_flu.1d732f0fb6.svg) Diagram: Re-exam 2026 Q3a: weather–flu–cough–fever network. Labels: Re-exam 2026 Q3a: weather–flu–cough–fever network | Weather | P(W) = 0.75 | Flu | P(F|W) = 0.2, P(F|¬W) = 0.5 | Cough | P(C|W) = 0.7, P(C|¬W) = 0.25 | Fever | P(V|F,C): .60 · .02 · .60 · .01 | (TT · TF · FT · FF) ##### Step 1. Chain rule $$ P(W,F,C,V)=P(W)\,P(F\mid W)\,P(C\mid W)\,P(V\mid F,C) $$ ##### Step 2. Sum over W and C with F = false, V = true | W | C | P(W) | P(¬F|W) | P(C|W) | P(V|¬F,C) | Product | | --- | --- | --- | --- | --- | --- | --- | | T | T | 0.75 | 0.8 | 0.70 | 0.60 | 0.252 | | T | F | 0.75 | 0.8 | 0.30 | 0.01 | 0.0018 | | F | T | 0.25 | 0.5 | 0.25 | 0.60 | 0.01875 | | F | F | 0.25 | 0.5 | 0.75 | 0.01 | 0.0009375 | | **P(V, ¬F)** | | | | | | **0.2734875** | ##### Step 3. Evidence $$ P(\lnot F)=0.75(0.8)+0.25(0.5)=0.6+0.125=0.725 $$ ##### Step 4. Divide $$ P(V\mid\lnot F)=\frac{0.2734875}{0.725}=0.3772 $$ **Final answer.** P(fever | no flu) ≈ **0.377** (fever without flu comes mostly from cough). **Key correction.** The key uses P(Flu) = 0.05 and P(Fever | ¬Flu) = 0.2, which are not in this network, and answers 0.2. Marking scheme (still applies): write the equation with hidden variables (2), take true/false cases (3), substitute (2), answer (3). --- ## TEE · Before GMM: centre, spread, density and likelihood Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-gmm--centre-spread/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-gmm/centre-spread Unit 1 · Mixture models, GMM, expectation maximization, clustering using GMM · 10-mark PYQ in both papers ## Mean, variance and standard deviation A Gaussian component describes a cluster using its **mean** (centre) and **variance** (spread). The **standard deviation** is the square root of variance. The Greek letters μ and σ name these two quantities; σ² is the variance. **Calculate the mean, population variance and standard deviation of [1, 3, 5].** [2 marks · Worked example] #### Given Data values: [1, 3, 5]. Number of observations n=3. Use population variance, dividing by n. #### To find The mean μ, population variance σ², and standard deviation σ. #### Formula and why it applies $$ \mu=\frac{\sum_i x_i}{n}\quad\sigma^2=\frac{\sum_i(x_i-\mu)^2}{n}\quad\sigma=\sqrt{\sigma^2} $$ This descriptive/maximum-likelihood convention divides by n. The n−1 sample-variance estimator is a different convention. #### Solution ##### Step 1. Add and divide for the mean $$ \mu=\frac{1+3+5}{3}=3 $$ ##### Step 2. Subtract the mean, then square | x | x−mean | Squared deviation | | --- | --- | --- | | 1 | -2 | 4 | | 3 | 0 | 0 | | 5 | 2 | 4 | ##### Step 3. Average the squared deviations $$ \sigma^2=\frac{4+0+4}{3}=2.666666667 $$ ##### Step 4. Take the square root $$ \sigma=\sqrt{2.666666667}\approx1.632993162 $$ **Final answer.** Mean=3; variance=2.666666667; standard deviation≈1.632993162. **Calculate the mean, population variance and standard deviation of [2, 4].** [2 marks · Worked example] #### Given Data values: [2, 4]. Number of observations n=2. Use population variance, dividing by n. #### To find The mean μ, population variance σ², and standard deviation σ. #### Formula and why it applies $$ \mu=\frac{\sum_i x_i}{n}\quad\sigma^2=\frac{\sum_i(x_i-\mu)^2}{n}\quad\sigma=\sqrt{\sigma^2} $$ This descriptive/maximum-likelihood convention divides by n. The n−1 sample-variance estimator is a different convention. #### Solution ##### Step 1. Add and divide for the mean $$ \mu=\frac{2+4}{2}=3 $$ ##### Step 2. Subtract the mean, then square | x | x−mean | Squared deviation | | --- | --- | --- | | 2 | -1 | 1 | | 4 | 1 | 1 | ##### Step 3. Average the squared deviations $$ \sigma^2=\frac{1+1}{2}=1 $$ ##### Step 4. Take the square root $$ \sigma=\sqrt{1}\approx1 $$ **Final answer.** Mean=3; variance=1; standard deviation≈1. **Calculate the mean, population variance and standard deviation of [4, 4, 4].** [2 marks · Worked example] #### Given Data values: [4, 4, 4]. Number of observations n=3. Use population variance, dividing by n. #### To find The mean μ, population variance σ², and standard deviation σ. #### Formula and why it applies $$ \mu=\frac{\sum_i x_i}{n}\quad\sigma^2=\frac{\sum_i(x_i-\mu)^2}{n}\quad\sigma=\sqrt{\sigma^2} $$ This descriptive/maximum-likelihood convention divides by n. The n−1 sample-variance estimator is a different convention. #### Solution ##### Step 1. Add and divide for the mean $$ \mu=\frac{4+4+4}{3}=4 $$ ##### Step 2. Subtract the mean, then square | x | x−mean | Squared deviation | | --- | --- | --- | | 4 | 0 | 0 | | 4 | 0 | 0 | | 4 | 0 | 0 | ##### Step 3. Average the squared deviations $$ \sigma^2=\frac{0+0+0}{3}=0 $$ ##### Step 4. Take the square root $$ \sigma=\sqrt{0}\approx0 $$ **Final answer.** Mean=4; variance=0; standard deviation≈0. The constant data [4,4,4] have variance zero. A standard Gaussian density formula requires positive variance; do not divide by zero when fitting a component. ## Density, likelihood and logarithm have different jobs A **density** is the height of a continuous distribution. Probability over an interval is area, so density height can exceed 1 without violating probability rules. A **likelihood** treats the observed data as fixed and scores candidate model parameters. For independent observations, their densities multiply. **A model assigns density 0.2 to one observation and 0.5 to a second independent observation. Find likelihood and natural log-likelihood.** [2 marks · Worked example] #### Given Two density values .2 and .5. #### To find Product and its natural logarithm. #### Formula and why it applies $$ L=p(x_1)p(x_2)\qquad\ln L=\ln p(x_1)+\ln p(x_2) $$ The logarithm turns a product into a sum. #### Solution ##### Step 1. Multiply the densities $$ L=0.2\times0.5=0.1 $$ ##### Step 2. Take ln of that product $$ \ln L=\ln(0.1)\approx-2.302585093 $$ ##### Step 3. Show the equivalent sum $$ \ln(0.2)+\ln(0.5)\approx-1.609437912-0.693147181=-2.302585093 $$ **Final answer.** Likelihood .1; log-likelihood≈−2.302585093. Do not turn a sum inside a log into a sum of logs: ln(.2+.5)=ln(.7)≈−.356675, whereas ln(.2)+ln(.5)≈−2.302585. A GMM has an inner sum over components, which must stay inside each observation’s logarithm. **Yaad rakho:** Mean = centre. Variance = squared distance ka average. Likelihood = fixed data ko model kitna support karta hai. --- ## TEE · GMM prerequisites: mean, variance, density and mixture Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-gmm--mixture-models/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-gmm/mixture-models Unit 1 · Mixture models, GMM, expectation maximization, clustering using GMM · 10-mark PYQ in both papers | Read this | Meaning | Concrete example | | --- | --- | --- | | Component k | One Gaussian group in the mixture | Component 1 may have mean 2; component 2 mean 4 | | Mixing weight πₖ | The prior share assigned to component k | π₁=.4 and π₂=.6 sum to 1 | | N(x|μ,σ²) | The Gaussian density evaluated at x | The vertical bar supplies parameters; it is not division | | Σ over k | Add contributions of every component | p(x)=.4N₁(x)+.6N₂(x) | ## Mean, spread and a Gaussian component A **mean** is the centre, found by adding observations and dividing by their count. A **variance** measures squared spread around the mean. In one dimension, a Gaussian component is described by mean μ and positive variance σ². Its standard deviation σ is the square root of that variance. A **density** is a height, not the probability of one exact real-valued observation. Areas under a density give probabilities. Density heights can exceed 1; their total area must be 1. $$ \mathcal N(x\mid\mu,\sigma^2)=\frac1{\sqrt{2\pi}\,\sigma}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right) $$ Here π inside 2π is the constant 3.14159…; the separately written πₖ in a mixture denotes a component weight. exp(a)=eᵃ, and the calculator eˣ key evaluates it. The minus sign in the exponent reduces density away from the mean. **For a standard Gaussian with μ=0 and σ²=1, find the density at x=0.** [3 marks · Practice] #### Given x=0; μ=0; σ=1. #### To find Density N(x|0,1). #### Formula and why it applies Use the Gaussian density with the supplied mean and variance. #### Solution ##### Step 1. Squared distance $$ (0-0)^2=0 $$ ##### Step 2. Exponent $$ -0/(2\times1)=0 $$ ##### Step 3. Exponential $$ e^{0}\approx1 $$ ##### Step 4. Divide by the normalising factor $$ \frac{1}{\sqrt{2\pi}}\approx0.39894228 $$ **Final answer.** 0.39894228 (a density, not an exact-point probability). **For a standard Gaussian with μ=0 and σ²=1, find the density at x=1.** [3 marks · Practice] #### Given x=1; μ=0; σ=1. #### To find Density N(x|0,1). #### Formula and why it applies Use the Gaussian density with the supplied mean and variance. #### Solution ##### Step 1. Squared distance $$ (1-0)^2=1 $$ ##### Step 2. Exponent $$ -1/(2\times1)=-0.5 $$ ##### Step 3. Exponential $$ e^{-0.5}\approx0.60653066 $$ ##### Step 4. Divide by the normalising factor $$ \frac{0.60653066}{\sqrt{2\pi}}\approx0.241970725 $$ **Final answer.** 0.241970725 (a density, not an exact-point probability). **For a standard Gaussian with μ=0 and σ²=1, find the density at x=2.** [3 marks · Practice] #### Given x=2; μ=0; σ=1. #### To find Density N(x|0,1). #### Formula and why it applies Use the Gaussian density with the supplied mean and variance. #### Solution ##### Step 1. Squared distance $$ (2-0)^2=4 $$ ##### Step 2. Exponent $$ -4/(2\times1)=-2 $$ ##### Step 3. Exponential $$ e^{-2}\approx0.135335283 $$ ##### Step 4. Divide by the normalising factor $$ \frac{0.135335283}{\sqrt{2\pi}}\approx0.053990967 $$ **Final answer.** 0.053990967 (a density, not an exact-point probability). ## Mixture model and hidden component A **mixture model** combines component densities. A Gaussian mixture model (GMM) uses K Gaussian components. A hidden component label says which component generated a point; it is not observed in ordinary clustering. A weight πₖ is the prior probability of choosing component k. All weights are nonnegative and sum to 1. $$ p(x)=\sum_{k=1}^K\pi_k\mathcal N(x\mid\mu_k,\sigma_k^2) $$ For two components this means π₁×density₁ + π₂×density₂. A GMM can represent several concentrations of data rather than forcing all points around one mean. Swipe across the diagram to read its labels ↔ ![The total density is a weighted sum of component densities.](https://deep-learning-study-room.vercel.app/is2/fig/gmm_mix.84e959698a.svg) Diagram: The total density is a weighted sum of component densities.. Labels: A 1-D Gaussian mixture: three weighted bells add up to p(x) | -8 | -4 | 0 | 4 | 8 | p | x The total density is a weighted sum of component densities. **At x, two component densities are 0.2 and 0.05, with weights 0.6 and 0.4. Find the mixture density.** [2 marks · Practice] #### Given π₁=.6, π₂=.4; densities=.2,.05. #### To find p(x). #### Formula and why it applies Weight each component density, then add. $$ p(x)=\pi_1N_1+\pi_2N_2 $$ #### Solution ##### Step 1. First contribution $$ 0.6(0.2)=0.12 $$ ##### Step 2. Second contribution $$ 0.4(0.05)=0.02 $$ ##### Step 3. Add $$ p(x)=0.12+0.02=0.14 $$ **Final answer.** Mixture density = 0.14. **Yaad rakho:** Har Gaussian ek group ka model; mixture = weights ke saath sab densities ka sum. --- ## TEE · EM for GMM: every responsibility and parameter update Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-gmm--em/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-gmm/em Unit 1 · Mixture models, GMM, expectation maximization, clustering using GMM · 10-mark PYQ in both papers **At one observation, the two weighted component densities are q₁=0.12 and q₂=0.03. Find the responsibilities. Then use responsibilities [0.8,0.2] for data [1,5] to find a component’s new mean.** [2 marks · Worked example] #### Given Part A: weighted component densities q₁=0.12 and q₂=0.03 for one observation. Part B: data x₁=1 and x₂=5, with component responsibilities 0.8 and 0.2 respectively. Part A compares components for one point; Part B combines points for one component. #### To find Two responsibilities and a weighted mean. #### Formula and why it applies $$ \gamma_{ik}=q_{ik}/\sum_jq_{ij}\qquad\mu_k^{new}=\frac{\sum_i\gamma_{ik}x_i}{\sum_i\gamma_{ik}} $$ i identifies an observation; k identifies a component. Notice which index is being added in each formula. #### Solution ##### Step 1. Normalise one observation $$ q_1+q_2=0.12+0.03=0.15 $$ $$ \gamma_1=0.12/0.15=0.8\quad\gamma_2=0.03/0.15=0.2 $$ ##### Step 2. Find the effective component count $$ N_k=0.8+0.2=1 $$ ##### Step 3. Weight each observation $$ 0.8\times1=0.8\qquad0.2\times5=1 $$ ##### Step 4. Add and divide $$ \mu_k^{new}=(0.8+1)/1=1.8 $$ **Final answer.** Responsibilities [.8,.2]; new mean 1.8. A point contributes a fraction of a count instead of being assigned wholly to one component. ## What EM is estimating **Expectation–maximisation (EM)** alternates two calculations when component assignments are hidden. The E-step estimates how responsible each component is for each observation. The M-step uses those responsibilities as weights to update the model parameters. A **responsibility** γᵢₖ is a posterior component probability: for observation i, how likely is component k under the current model? For each observation, the responsibilities sum to 1. Unlike fuzzy memberships, these probabilities are explicitly normalised across mixture components. | Symbol | Meaning | | --- | --- | | i / k | Observation number / component number | | γᵢₖ | Responsibility of component k for observation i | | Nₖ | Sum of responsibilities for component k: an effective count | | μₖ | Component mean | | σ²ₖ | Component variance | | πₖ | Component mixing weight | $$ \gamma_{ik}=\frac{\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)}{\sum_j\pi_j\mathcal N(x_i\mid\mu_j,\sigma_j^2)} $$ $$ N_k=\sum_i\gamma_{ik},\quad\mu_k^{new}=\frac{\sum_i\gamma_{ik}x_i}{N_k} $$ $$ (\sigma_k^2)^{new}=\frac{\sum_i\gamma_{ik}(x_i-\mu_k^{new})^2}{N_k},\quad\pi_k^{new}=N_k/n $$ The E-step uses all old parameters. The M-step mean uses those fixed responsibilities; the variance uses the newly calculated mean. One exact EM update cannot lower the likelihood when its optimisation steps are performed correctly; numerical approximations and regularisation need their own interpretation. EM can still reach a local solution. **For observations {1,3,5}, initialise a two-component GMM with means 2 and 4, both variances 1, and both weights 0.5. Perform one complete EM iteration.** [6 marks · Practice] #### Given n=3, K=2, μ₁=2, μ₂=4; σ₁²=σ₂²=1; π₁=π₂=.5. #### To find Six responsibilities and the new weights, means and variances. #### Formula and why it applies Apply the E-step and M-step equations above. Calculate with full precision; decimals below are displayed to nine places. #### Solution ##### Step 1. E-step: weighted densities for x=1 $$ q_1=0.5\frac{e^{-(1-2)^2/2}}{\sqrt{2\pi}}\approx0.120985362 $$ $$ q_2=0.5\frac{e^{-(1-4)^2/2}}{\sqrt{2\pi}}\approx0.002215924 $$ ##### Step 2. E-step: weighted densities for x=3 $$ q_1=0.5\frac{e^{-(3-2)^2/2}}{\sqrt{2\pi}}\approx0.120985362 $$ $$ q_2=0.5\frac{e^{-(3-4)^2/2}}{\sqrt{2\pi}}\approx0.120985362 $$ ##### Step 3. E-step: weighted densities for x=5 $$ q_1=0.5\frac{e^{-(5-2)^2/2}}{\sqrt{2\pi}}\approx0.002215924 $$ $$ q_2=0.5\frac{e^{-(5-4)^2/2}}{\sqrt{2\pi}}\approx0.120985362 $$ ##### Step 4. E-step: normalise responsibilities for x=1 $$ \gamma_1=\frac{0.120985362}{0.120985362+0.002215924}\approx0.98201379 $$ $$ \gamma_2=1-\gamma_1\approx0.01798621 $$ ##### Step 5. E-step: normalise responsibilities for x=3 $$ \gamma_1=\frac{0.120985362}{0.120985362+0.120985362}\approx0.5 $$ $$ \gamma_2=1-\gamma_1\approx0.5 $$ ##### Step 6. E-step: normalise responsibilities for x=5 $$ \gamma_1=\frac{0.002215924}{0.002215924+0.120985362}\approx0.01798621 $$ $$ \gamma_2=1-\gamma_1\approx0.98201379 $$ ##### Step 7. M-step: effective count for component 1 $$ N_1=0.98201379+0.5+0.01798621=1.5 $$ ##### Step 8. M-step: mean for component 1 $$ \mu_1^{new}=\frac{0.98201379(1)+0.5(3)+0.01798621(5)}{1.5}\approx1.714629893 $$ ##### Step 9. M-step: variance for component 1 | x | Responsibility | Squared deviation from new mean | Weighted squared deviation | | --- | --- | --- | --- | | 1 | 0.98201379 | 0.510695884 | 0.501510401 | | 3 | 0.5 | 1.652176311 | 0.826088156 | | 5 | 0.01798621 | 10.793656738 | 0.194136976 | $$ (\sigma_1^2)^{new}=\frac{1.521735533}{1.5}\approx1.014490355 $$ ##### Step 10. M-step: mixture weight for component 1 $$ \pi_1^{new}=1.5/3=0.5 $$ ##### Step 11. M-step: effective count for component 2 $$ N_2=0.01798621+0.5+0.98201379=1.5 $$ ##### Step 12. M-step: mean for component 2 $$ \mu_2^{new}=\frac{0.01798621(1)+0.5(3)+0.98201379(5)}{1.5}\approx4.285370107 $$ ##### Step 13. M-step: variance for component 2 | x | Responsibility | Squared deviation from new mean | Weighted squared deviation | | --- | --- | --- | --- | | 1 | 0.01798621 | 10.793656738 | 0.194136976 | | 3 | 0.5 | 1.652176311 | 0.826088156 | | 5 | 0.98201379 | 0.510695884 | 0.501510401 | $$ (\sigma_2^2)^{new}=\frac{1.521735533}{1.5}\approx1.014490355 $$ ##### Step 14. M-step: mixture weight for component 2 $$ \pi_2^{new}=1.5/3=0.5 $$ **Final answer.** Means ≈ [1.714629893, 4.285370107]; variances ≈ [1.014490355, 1.014490355]; weights [0.5,0.5]. **Yaad rakho:** E: har point ki responsibility. M: responsibilities se weighted mean, variance, weight. --- ## TEE · 10-mark answer: GMM vs k-means, log-likelihood derivation, parameters for K = 5 Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-gmm--ten-mark-answer/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-gmm/ten-mark-answer Unit 1 · Mixture models, GMM, expectation maximization, clustering using GMM · 10-mark PYQ in both papers ## Read the derivation’s symbols first “i.i.d.” means the observations are modelled as independent draws from the same distribution. The product Π runs over observations; the inner sum Σ runs over components. λki in some notes is the same responsibility that other sections call γik. A superscript i here labels the observation, not an exponent. Jensen’s inequality gives a lower bound on the log of the component sum. EM chooses responsibilities making that bound touch the current likelihood, then updates the parameters to maximise the bound with those responsibilities fixed. This explains the two alternating steps; λ values are not arbitrary new model parameters. | Parameter | Number in a 1D K-component GMM | Why | | --- | --- | --- | | Means | K | One centre for each component | | Variances | K | One spread for each component | | Independent mixing weights | K−1 | The last weight is 1 minus the others | | Total | 3K−1 | K+K+(K−1) | For K=2, choose μ₁,μ₂,σ₁²,σ₂² and π₁: five free numbers. Then π₂=1−π₁. For K=5, 5+5+4=14. In 2D with a full covariance matrix, each component instead has two means and three independent covariance entries (two variances and one shared off-diagonal covariance). Five such components have 5×(2+3)+4=29 free parameters. **Asked twice as:** “With reference to GMM: compare GMM with the k-means algorithm; derive the equation for the log-likelihood function; what will be the number of parameters to be computed if K = 5?” [10 marks · Final 2025 Q2b and Re-exam 2026 Q2b; key: up to the maximum-likelihood equation 5, estimation 5] ### 1. GMM versus k-means | Aspect | k-means | GMM | | --- | --- | --- | | Assignment | Hard: each point in exactly one cluster | Soft: probability $$ \lambda_k^i $$ of belonging to each cluster | | Model | Geometric: nearest centroid | Probabilistic generative model $$ \sum_k\pi_k\mathcal N(\mu_k,\Sigma_k) $$ | | Cluster shape | Spherical, similar size | Elliptical, any size/orientation; overlapping clusters | | Parameters | K centroids | Weights, means, covariances (1-D: 3K − 1) | | Algorithm / objective | Assign–update; minimise squared distances | EM; maximise log-likelihood | | Output | Hard labels and fitted centroids | Labels, membership probabilities and a density (useful for anomaly detection) | | Relation | The hard-assignment, equal spherical variance, small-variance limit of a Gaussian-mixture objective leads to the k-means objective. Ordinary finite-variance EM is not the same algorithm as k-means. Both need K in advance and depend on initialisation. | | ### 2. Derivation of the log-likelihood **Step 1. Model.** Each point is generated by choosing component k with probability $$ \pi_k $$ and drawing $$ x_i\sim\mathcal N(\mu_k,\sigma_k^2) $$ , so its density is $$ f_{mix}(x_i)=\sum_k\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2) $$ . **Step 2. Likelihood.** Points are i.i.d., so the likelihood is a product (θ = all parameters): $$ L(\theta)=\prod_{i=1}^{n}\Big[\sum_{k=1}^{K}\pi_k\frac{1}{\sqrt{2\pi}\,\sigma_k}e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}}\Big] $$ **Step 3. Log-likelihood.** Logs turn the product into a sum: $$ \log L(\theta)=\sum_{i=1}^{n}\log\Big(\sum_{k=1}^{K}\pi_k\frac{1}{\sqrt{2\pi}\,\sigma_k}e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}}\Big) $$ **Step 4. Why it cannot be solved directly.** The sum sits inside the log, so setting derivatives to zero gives no closed-form solution. **Step 5. Jensen's inequality.** log is concave, so for weights $$ \lambda_k\ge0,\;\sum\lambda_k=1 $$ : $$ \log\big(\sum_k\lambda_ka_k\big)\ge\sum_k\lambda_k\log a_k $$ . Introduce $$ \lambda_k^i $$ for every point ( $$ \sum_k\lambda_k^i=1 $$ ), multiply and divide inside the log: $$ \log L(\theta)=\sum_i\log\Big(\sum_k\lambda_k^i\,\frac{\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)}{\lambda_k^i}\Big)\;\ge\;\sum_i\sum_k\lambda_k^i\log\frac{\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)}{\lambda_k^i} $$ **Step 6. Estimation (EM).** Maximise the lower bound alternately. Fixing θ gives the E-step $$ \lambda_k^i=\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)/\sum_l\pi_l\mathcal N(x_i\mid\mu_l,\sigma_l^2) $$ . Fixing λ and differentiating (with a Lagrange multiplier for $$ \sum\pi_k=1 $$ ) gives the M-step: $$ \hat\mu_k=\frac{\sum_i\lambda_k^ix_i}{\sum_i\lambda_k^i},\qquad\hat\sigma_k^2=\frac{\sum_i\lambda_k^i(x_i-\hat\mu_k)^2}{\sum_i\lambda_k^i},\qquad\hat\pi_k=\frac{\sum_i\lambda_k^i}{n} $$ ### 3. Number of parameters for K = 5 The dimension and covariance structure must also be specified. For the one-dimensional, independent-variance model used in the derivation: $$ 2K+(K-1)=3K-1=3(5)-1=\mathbf{14} $$ (5 means + 5 variances + 4 free weights, since the 5 weights must add to 1.) In two dimensions with full covariance it would be 29. **Slide note.** One slide puts the convex inequality under the “concave” heading. Correct: log is *concave*, so $$ f(\sum\lambda a)\ge\sum\lambda f(a) $$ (≥, not >). --- ## TEE · Clustering with GMM: uses, advantages and limits Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-gmm--clustering-applications/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u1-gmm/clustering-applications Unit 1 · Mixture models, GMM, expectation maximization, clustering using GMM · 10-mark PYQ in both papers ## Clustering using GMM 1. Choose K (domain knowledge, or a criterion such as BIC). 2. Fit the GMM with EM. 3. For each point compute $$ \lambda_k^i $$ ; the largest gives its cluster, and the values show how confident the assignment is. 4. Points with very low $$ p(x) $$ under every component are anomalies. **Example (class).** Marks 45, 48, 50, 52, 80, 82, 85, 88, 90 form two groups, “average” near 50 and “high performers” near 85. A two-component GMM gives each student a probability of belonging to each group; students in between (none here) would get split probabilities instead of a forced label. ## Applications - Customer segmentation in marketing - Image segmentation (pixel colours) and background subtraction in video - Speaker identification and speech recognition - Anomaly and fraud detection (low-density points) - Density estimation and generating new samples ## Advantages and limitations | Advantages | Limitations | | --- | --- | | Soft assignments with probabilities | K must be chosen in advance | | Flexible elliptical, overlapping clusters | Sensitive to initialisation; can stop at a local optimum | | Interpretable parameters (weights, means, covariances) | Assumes each cluster is Gaussian | | Gives a full density model | Costly for many dimensions (covariance matrices) | **Probability vs likelihood** (lab question): probability fixes the parameters and asks how likely the data is; likelihood fixes the observed data and asks how plausible each parameter value is. Maximum-likelihood estimation chooses the parameters that make the observed data most probable. ### Why an unrestricted GMM can collapse A Gaussian centred on one training observation can make its variance approach zero, causing its density there and the mixture likelihood to grow without a finite bound. This is a singularity, not evidence of a useful perfect cluster. Practical fitting therefore uses a positive variance/covariance floor or a prior, sensible initialisation, and checks for nearly empty components. Different starts can reach different local solutions. These qualifications do not change the stated one-iteration classroom calculation. Primary reference: [Bishop, Pattern Recognition and Machine Learning, chapter 9](https://www.microsoft.com/en-us/research/wp-content/uploads/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf). --- ## TEE · Fuzzy sets: what membership actually means Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--membership-meaning/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/membership-meaning Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Start with an input and a label Suppose the input is **temperature 32°C** and the fuzzy label is **Hot**. A membership function is the rule that converts that temperature into a degree of matching the label. If μHot(32)=0.7, read it aloud as “the membership of 32 degrees in Hot is 0.7”. The lower word Hot names the set; 32 is the input; 0.7 is the output. The same temperature can have membership 0.3 in Medium and 0.7 in Hot. These are two descriptions of one input. Fuzzy memberships do not generally have to sum to 1. By contrast, probabilities of mutually exclusive, exhaustive outcomes do. | Read this | Meaning | Concrete example | | --- | --- | --- | | Universe U | All input values being considered | Temperatures from 0 to 40°C | | Crisp value | One definite measured number | 32°C | | Linguistic term | A word naming a fuzzy category | Low, Medium, High | | μA(x) | Degree to which input x fits category A | μHot(32)=0.7 | **Yaad rakho:** 32 temperature hai; 0.7 us temperature ka Hot label mein degree hai. Dono ko mix mat karo. ## Why “true or false” is sometimes too rigid Suppose a controller says a room is hot only when its temperature is at least 30°C. At 29.9°C it says “not hot”; at 30°C it says “hot”. That sharp boundary is a **crisp** rule. Fuzzy logic allows a gradual degree of belonging to the category “Hot”. **Definition.** A fuzzy set A on a universe U assigns every element x a membership degree μA(x) between 0 and 1. The symbol μ is the Greek letter “mu”; read μA(x) as “the degree to which x belongs to set A”. $$ \mu_A:U\rightarrow[0,1] $$ | Term | Meaning | Temperature example | | --- | --- | --- | | Universe U | All allowed input values | Temperatures from 0°C to 40°C | | Element x | The particular input being considered | 32°C | | Fuzzy set A | A category described by a membership function | Hot | | Membership μA(x) | How strongly x fits that category | μHot(32) = 0.7 | ## Read membership correctly Membership 0 means no belonging; 1 means full belonging; 0.7 means partial belonging under the chosen definition. **It does not mean a 70% chance that the measured temperature is 32°C.** Nor is it a 70% chance that the sensor is correct. Fuzziness concerns the gradual meaning of “Hot”; probability concerns uncertainty about which event occurs. | Input | Warm membership | Hot membership | Interpretation | | --- | --- | --- | --- | | 28 | 0.8 | 0.1 | Fits Warm strongly and Hot weakly | | 32 | 0.3 | 0.7 | Fits Hot more than Warm | | 35 | 0 | 1 | Fully Hot in this chosen model | These memberships need not add to 1. Overlapping categories are allowed. A crisp set is the special case in which membership can only be 0 or 1. A crisp set can also contain infinitely many elements; “crisp” does not mean “finite”. **On U = {20, 25, 30, 35}, the Hot memberships are [0, 0.2, 0.6, 1]. Write the fuzzy set Hot and identify the fully and partly belonging inputs.** [2 marks · Practice] #### Given Ordered inputs [20, 25, 30, 35]; memberships [0, 0.2, 0.6, 1]. #### To find The set notation and interpretation. #### Formula and why it applies In the common fuzzy-set notation μ/x, the slash separates membership from element; it is not division. #### Solution ##### Step 1. Pair each input with its membership | x | μHot(x) | | --- | --- | | 20 | 0 | | 25 | 0.2 | | 30 | 0.6 | | 35 | 1 | ##### Step 2. Write the set $$ Hot=\{0/20,\;0.2/25,\;0.6/30,\;1/35\} $$ ##### Step 3. Interpret 35 fully belongs; 25 and 30 partly belong; 20 does not belong. **Final answer.** Hot = {0/20, 0.2/25, 0.6/30, 1/35}. **Distinguish crisp and fuzzy sets.** [2 marks · Practice] #### Answer | Crisp set | Fuzzy set | | --- | --- | | Membership is 0 or 1 | Membership is any value in [0,1] | | A sharp boundary separates members | A membership function permits gradual boundaries | | Example: temperature ≥ 30°C | Example: degree of being Hot increases gradually | **Yaad rakho:** μ ka matlab: is category mein kitna fit hota hai. Chance nahi. --- ## TEE · Triangular membership: choose the side, then calculate Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--triangle/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/triangle Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Why the triangle has two different fractions Tri(2,5,8) means membership starts at zero at x=2, rises to 1 at x=5, then falls to zero at x=8. The first three numbers are **input positions**, not three membership degrees. | Where is x? | Meaning | Calculation | | --- | --- | --- | | x≤2 or x≥8 | Outside, or at a zero-height endpoint | 0 | | 20; core: degree=1; boundary: 0 0. **Core:** inputs with μ = 1. **Boundary:** inputs with 0 < μ < 1. **Height:** largest membership (more generally the supremum). A set is normal when its height is 1; it is subnormal when its height is below 1. A crossover point has membership 0.5. An α-cut contains inputs with μ ≥ α. For Tri(2,5,8), support is (2,8), core is {5}, height is 1, and the 0.5-cut is [3.5,6.5]. Solve (x−2)/3=0.5 → x=3.5 and (8−x)/3=0.5 → x=6.5. A one-dimensional fuzzy set is **convex** if every α-cut is an interval (or empty). The membership between two inputs must not dip below the smaller endpoint membership. A single triangle is convex; two separated peaks with a lower valley may not be. **Yaad rakho:** Fuzzy A aur NOT A dono partial ho sakte hain. Isliye crisp waali 1 aur 0 identities blindly nahi lagani. --- ## TEE · Mamdani architecture: what every block does Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--mamdani-architecture/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/mamdani-architecture Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Follow one measurement through the controller A **controller** turns measurements into an action setting. Consider temperature 32°C and occupancy 7 people. The input numbers first become membership degrees. The rules then combine those degrees. Only the final defuzzification returns one fan-speed number. | Stage | What goes in | Operation | What comes out | | --- | --- | --- | --- | | Fuzzifier | 32°C and 7 people | Use the supplied membership graphs | Temperature Medium=.3, High=.7; occupancy High=.5 | | Knowledge base | Chosen labels and their shapes | Store graphs and IF–THEN rules | A reusable definition of how the controller works | | Inference | Medium=.3 AND High=.5 | Rule strength=min(.3,.5)=.3 | “Fan High” fires at .3 | | Aggregation | All clipped output sets | Take the largest output degree at each speed | One combined fuzzy output shape | | Defuzzifier | Combined shape | Apply the specified MoM, centroid or representative method | One numerical speed | The **antecedent** is the IF part (“temperature is Medium AND occupancy is High”). The **consequent** is the THEN part (“fan speed is High”). A rule strength of 0.3 is a membership height; it does not mean the fan runs at 30%. ## Definition: a fuzzy inference system A **fuzzy inference system (FIS)** maps inputs to an output by evaluating fuzzy IF–THEN rules. A **Mamdani FIS** uses fuzzy sets in the rule consequents, combines their inferred output sets, and defuzzifies the result. Crisp inputs32°C; 7 people → Fuzzificationdegrees of Warm/Hot etc. → Rules + inferencerule strengths and clipped outputs → Aggregationone combined output shape → Defuzzificationone fan-speed number ## Architecture: what each block contributes | Block | What it stores or calculates | Concrete meaning | | --- | --- | --- | | Fuzzifier | Membership values of the measured inputs | 32°C may be Medium to degree 0.3 and High to degree 0.7 | | Knowledge base: database | Ranges and membership functions | Medium = Tri(15,25,35) | | Knowledge base: rule base | IF–THEN rules | IF temperature High AND occupancy High THEN fan Very High | | Inference engine | Combines conditions and applies consequents | min for AND; clips the output triangle at the rule strength | | Aggregator | Combines all rule outputs | At every output location, take the maximum clipped membership | | Defuzzifier | Converts the aggregate to one crisp value | Use centroid or mean of maxima, as requested | ## Parts of an IF–THEN rule In “IF temperature is High AND occupancy is High THEN fan speed is Very High”, the IF part is the **antecedent** (condition), and the THEN part is the **consequent** (output category). The input variable temperature, the label High, and the numerical membership μHigh(32) are different things. ## Why a rule strength is not yet the fan speed A strength such as 0.3 says how strongly a rule fires. It is on the membership scale 0–1. Fan speed might be on a 0–100% scale. Defuzzification is the step that produces a number on that output scale. **Explain the architecture of a Mamdani fuzzy inference system.** [3 marks · Practice] #### Answer A Mamdani FIS contains a fuzzifier, knowledge base, inference engine and defuzzifier. The database specifies membership functions and the rule base stores IF–THEN rules. Crisp inputs are fuzzified. The inference engine calculates rule firing strengths and constructs output fuzzy sets. These sets are aggregated, then defuzzified into a crisp action. For a fan controller, temperature and occupancy are inputs and fan speed is the output. **Yaad rakho:** Input ka degree → rule ka strength → output ka shape → final number. --- ## TEE · Rule firing, clipping and aggregation: four solved cases Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--rules-clipping/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/rules-clipping Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Clipping means limiting a height A rule firing at strength 0.3 cannot give its consequent a membership above 0.3. At every candidate speed z, keep the smaller of 0.3 and the original output-set membership. The speed axis is unchanged. **At speeds 40, 60 and 80, output set A has memberships [0.1,0.8,0.2] and set B has [0,0.4,1]. Rule A fires at 0.3 and rule B at 0.5. Clip and max-aggregate the sets.** [2 marks · Worked example] #### Given | Speed | Original A membership | Original B membership | | --- | --- | --- | | 40 | 0.1 | 0 | | 60 | 0.8 | 0.4 | | 80 | 0.2 | 1 | Rule A strength=0.3; rule B strength=0.5. Compare memberships at the same speed. #### To find The combined membership at each speed. #### Formula and why it applies $$ \mu_{out}(z)=\max(\min(0.3,\mu_A(z)),\min(0.5,\mu_B(z))) $$ Inner min limits a rule’s height; outer max keeps the stronger support at that speed. #### Solution ##### Step 1. Clip A At 40: min(.3,.1)=.1. At 60: min(.3,.8)=.3. At 80: min(.3,.2)=.2. ##### Step 2. Clip B At 40: min(.5,0)=0. At 60: min(.5,.4)=.4. At 80: min(.5,1)=.5. ##### Step 3. Aggregate at each speed | Speed | Clipped A | Clipped B | max(A,B) | | --- | --- | --- | --- | | 40 | 0.1 | 0 | 0.1 | | 60 | 0.3 | 0.4 | 0.4 | | 80 | 0.2 | 0.5 | 0.5 | **Final answer.** Combined degrees [0.1,0.4,0.5] at speeds [40,60,80]. A separate defuzzification step is still required. **Yaad rakho:** Clip = height ki ceiling. Aggregate = har same x par sabse upar wala height. ## Four different calculations happen in inference **AND/OR** combine conditions within a rule. **Implication** uses the resulting strength to modify that rule’s output membership function. **Aggregation** merges different rule outputs. Here we use min-AND, max-OR, min-implication and max-aggregation. $$ \alpha_{AND}=\min(a,b)\qquad\alpha_{OR}=\max(a,b) $$ $$ \mu_{\mathrm{rule}}(z)=\min(\alpha,\mu_{\mathrm{consequent}}(z)) $$ $$ \mu_{\mathrm{aggregate}}(z)=\max_r\mu_{\mathrm{rule}\ r}(z) $$ α (“alpha”) is the rule firing strength. z names a possible output value. The subscript r labels a rule, not a multiplication. **A rule has two condition degrees 0.7 and 0.5, joined by AND. Find its firing strength.** [2 marks · Practice] #### Given a=0.7; b=0.5; connective=AND. #### To find The strength α. #### Formula and why it applies Use the smaller degree because both conditions must hold. #### Solution ##### Step 1. Select the operator $$ \alpha=\min(0.7,0.5) $$ ##### Step 2. Evaluate $$ \alpha=0.5 $$ **Final answer.** 0.5. **A rule has two condition degrees 0.3 and 0, joined by AND. Find its firing strength.** [2 marks · Practice] #### Given a=0.3; b=0; connective=AND. #### To find The strength α. #### Formula and why it applies Use the smaller degree because both conditions must hold. #### Solution ##### Step 1. Select the operator $$ \alpha=\min(0.3,0) $$ ##### Step 2. Evaluate $$ \alpha=0 $$ **Final answer.** 0. **A rule has two condition degrees 0.2 and 0.8, joined by OR. Find its firing strength.** [2 marks · Practice] #### Given a=0.2; b=0.8; connective=OR. #### To find The strength α. #### Formula and why it applies Use the larger degree because either condition is sufficient. #### Solution ##### Step 1. Select the operator $$ \alpha=\max(0.2,0.8) $$ ##### Step 2. Evaluate $$ \alpha=0.8 $$ **Final answer.** 0.8. **Rule 1 fires at 0.3 and its original output membership at z=60 is 0.8. Rule 2 fires at 0.6 and its original membership at z=60 is 0.2. Find the aggregate membership at z=60 using min-implication and max-aggregation.** [3 marks · Practice] #### Given Rule 1: α₁=0.3, μ₁(60)=0.8. Rule 2: α₂=0.6, μ₂(60)=0.2. #### To find Membership at the single output location z=60 after implication and aggregation. #### Formula and why it applies Clip each original degree at its rule strength; then take the larger clipped degree. #### Solution ##### Step 1. Clip rule 1 $$ \mu_{R1}(60)=\min(0.3,0.8)=0.3 $$ ##### Step 2. Clip rule 2 $$ \mu_{R2}(60)=\min(0.6,0.2)=0.2 $$ ##### Step 3. Aggregate at this output location $$ \mu_{out}(60)=\max(0.3,0.2)=0.3 $$ **Final answer.** The aggregate membership at z=60 is 0.3. This is not a crisp fan-speed answer. Swipe across the diagram to read its labels ↔ ![Above the 0.3 level, the original triangle is cut off. Below 0.3, the original slope remains.](https://deep-learning-study-room.vercel.app/is2/fig/v2_clip.f101b5cfa9.svg) Diagram: Above the 0.3 level, the original triangle is cut off. Below 0.3, the original slope remains.. Labels: Min-implication clips the triangle at strength 0.3 | 0 | 0.3 | 1 | 50 | 54.5 | 65 | 75.5 | 80 | μ | Output z | Original High | clip 0.3 Above the 0.3 level, the original triangle is cut off. Below 0.3, the original slope remains. **Yaad rakho:** AND ke liye min; output clip karne ke liye bhi min; rules ko jodne ke liye max. Har min ka role alag hai. --- ## TEE · Mean of maxima: discrete peaks and clipped plateaus Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--mom/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/mom Definitions → membership graphs → operations → rules → defuzzification → complete controller. **Maximum membership** is a height. **Mean of maxima** averages the horizontal locations attaining that height. If heights at speeds [20,50,80] are [.2,.6,.6], the maximum height is .6 and the qualifying speeds are 50 and 80; MoM=(50+80)/2=65. Do not average [.6,.6], which would return a degree rather than a speed. A continuous flat maximum from 50 to 80 also has midpoint 65. ## Mean of maxima (MoM) **Defuzzification** converts the aggregate membership function into one crisp output. For MoM, first find the largest membership height, then average the output locations attaining that height. Do not average the membership values themselves. $$ z^*_{MoM}=\frac{z_1+\cdots+z_m}{m} $$ For a continuous flat maximum from a to b, the mean is its midpoint: $$ z^*_{MoM}=\frac{a+b}{2} $$ **For the following discrete aggregate, find the mean-of-maxima output.** [2 marks · Practice] #### Given | Output z | Membership μ | | --- | --- | | 10 | 0.2 | | 20 | 0.8 | | 30 | 0.8 | | 40 | 0.3 | #### To find MoM crisp output. #### Formula and why it applies Find all locations at the largest height, then average only those locations. #### Solution ##### Step 1. Find the maximum height $$ \mu_{max}=0.8 $$ ##### Step 2. List the locations attaining it 20, 30 ##### Step 3. Average these locations $$ z^*=\frac{20+30}{2}=25 $$ **Final answer.** 25. **For the following discrete aggregate, find the mean-of-maxima output.** [2 marks · Practice] #### Given | Output z | Membership μ | | --- | --- | | 1 | 0.1 | | 2 | 0.4 | | 3 | 0.9 | | 4 | 0.2 | #### To find MoM crisp output. #### Formula and why it applies Find all locations at the largest height, then average only those locations. #### Solution ##### Step 1. Find the maximum height $$ \mu_{max}=0.9 $$ ##### Step 2. List the locations attaining it 3 ##### Step 3. Average these locations $$ z^*=\frac{3}{1}=3 $$ **Final answer.** 3. **A single triangular output Tri(50,65,80) is clipped at α=0.3. Find its crisp value by mean of maxima.** [3 marks · Practice] #### Given a=50, b=65, c=80; clip height α=0.3. #### To find Both plateau endpoints and their mean. #### Formula and why it applies Set the rising and falling membership fractions equal to 0.3. Every point between those intersections has the maximum height. #### Solution ##### Step 1. Find the left endpoint $$ \frac{z_L-50}{65-50}=0.3 $$ $$ z_L-50=0.3\times15=4.5 $$ $$ z_L=54.5 $$ ##### Step 2. Find the right endpoint $$ \frac{80-z_R}{80-65}=0.3 $$ $$ 80-z_R=4.5 $$ $$ z_R=75.5 $$ ##### Step 3. Average endpoints $$ z^*=(54.5+75.5)/2=65 $$ **Final answer.** MoM = 65. Swipe across the diagram to read its labels ↔ ![The maximum extends over [54.5,75.5], whose midpoint is 65.](https://deep-learning-study-room.vercel.app/is2/fig/v2_clip.f101b5cfa9.svg) Diagram: The maximum extends over [54.5,75.5], whose midpoint is 65.. Labels: Min-implication clips the triangle at strength 0.3 | 0 | 0.3 | 1 | 50 | 54.5 | 65 | 75.5 | 80 | μ | Output z | Original High | clip 0.3 The maximum extends over [54.5,75.5], whose midpoint is 65. For several disjoint continuous maximum plateaus, a continuous MoM uses a length-weighted mean over all maximum intervals; simply averaging two outside endpoints is not generally valid. In tabulated questions, average the listed locations at the maximum. **Yaad rakho:** Sabse upar ka level dekho; us level par x-values ka mean lo. --- ## TEE · Centroid: three table sums and one area calculation Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--centroid/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/centroid Definitions → membership graphs → operations → rules → defuzzification → complete controller. ## Centroid means the balance point of the whole shape Imagine the area under the output graph as a thin sheet of uniform material. The centroid is its horizontal balance point. More area on one side pulls the answer toward that side. In a discrete problem, each membership is a weight. In a continuous problem, the entire area supplies the weights. The integral sign ∫ means adding very thin strips across a continuous interval. The denominator ∫μ(z)dz is the total area. The numerator ∫zμ(z)dz is the **first moment**: position multiplied by area, accumulated across the shape. Their ratio has units of the output, such as fan-speed percent. **A non-overlapping aggregate consists of a rectangle of width 4 and height 0.5 centred at z=2, plus a triangle of base 4 and height 0.5 centred horizontally at z=6. Find the area centroid.** [2 marks · Worked example] #### Given Rectangle: width 4, height 0.5, horizontal centre 2. Triangle: base 4, height 0.5, horizontal centroid 6. Their areas do not overlap. #### To find The combined horizontal centroid. #### Formula and why it applies $$ z^*=\frac{A_1c_1+A_2c_2}{A_1+A_2} $$ Each piece contributes area times its own centre. Use this only for a valid decomposition of the aggregate. #### Solution ##### Step 1. Rectangle area $$ A_1=4\times0.5=2 $$ ##### Step 2. Triangle area $$ A_2=\tfrac12\times4\times0.5=1 $$ ##### Step 3. First moments $$ A_1c_1=2\times2=4 $$ $$ A_2c_2=1\times6=6 $$ ##### Step 4. Totals $$ A=2+1=3\qquad M=4+6=10 $$ ##### Step 5. Divide $$ z^*=10/3\approx3.333333 $$ **Final answer.** Centroid≈3.333333, closer to the larger-area piece. For overlapping clipped sets, first form their max-aggregate. Adding the areas of the original overlapping sets would count the overlap twice. ## Centroid: the balance point of the output The centroid uses the whole aggregate membership distribution. For a discrete output table, multiply each output location by its membership, add the products, then divide by total membership. $$ z^*=\frac{\sum_i z_i\mu(z_i)}{\sum_i\mu(z_i)} $$ The index i simply numbers table entries. Σ (“sigma”) means add all listed terms. The denominator must be nonzero. **Find the discrete centroid of the aggregate membership table.** [3 marks · Practice] #### Given | z | μ(z) | | --- | --- | | 10 | 0.2 | | 20 | 0.6 | | 30 | 0.4 | #### To find One centroid value on the output scale. #### Formula and why it applies This question supplies memberships at discrete output locations; use the discrete centroid, not a peak-only approximation. $$ z^*=\frac{\sum z_i\mu_i}{\sum\mu_i} $$ #### Solution ##### Step 1. Multiply every location by its membership | z | μ | z × μ | | --- | --- | --- | | 10 | 0.2 | 2 | | 20 | 0.6 | 12 | | 30 | 0.4 | 12 | ##### Step 2. Add the numerator $$ 2+12+12=26 $$ ##### Step 3. Add the denominator $$ 0.2+0.6+0.4=1.2 $$ ##### Step 4. Divide $$ z^*=\frac{26}{1.2}=21.666666667 $$ **Final answer.** 21.666666667. **Find the discrete centroid of the aggregate membership table.** [3 marks · Practice] #### Given | z | μ(z) | | --- | --- | | 0 | 0.5 | | 5 | 1 | | 10 | 0.5 | #### To find One centroid value on the output scale. #### Formula and why it applies This question supplies memberships at discrete output locations; use the discrete centroid, not a peak-only approximation. $$ z^*=\frac{\sum z_i\mu_i}{\sum\mu_i} $$ #### Solution ##### Step 1. Multiply every location by its membership | z | μ | z × μ | | --- | --- | --- | | 0 | 0.5 | 0 | | 5 | 1 | 5 | | 10 | 0.5 | 5 | ##### Step 2. Add the numerator $$ 0+5+5=10 $$ ##### Step 3. Add the denominator $$ 0.5+1+0.5=2 $$ ##### Step 4. Divide $$ z^*=\frac{10}{2}=5 $$ **Final answer.** 5. **Find the discrete centroid of the aggregate membership table.** [3 marks · Practice] #### Given | z | μ(z) | | --- | --- | | 20 | 0 | | 40 | 0.3 | | 60 | 0.6 | | 80 | 0.1 | #### To find One centroid value on the output scale. #### Formula and why it applies This question supplies memberships at discrete output locations; use the discrete centroid, not a peak-only approximation. $$ z^*=\frac{\sum z_i\mu_i}{\sum\mu_i} $$ #### Solution ##### Step 1. Multiply every location by its membership | z | μ | z × μ | | --- | --- | --- | | 20 | 0 | 0 | | 40 | 0.3 | 12 | | 60 | 0.6 | 36 | | 80 | 0.1 | 8 | ##### Step 2. Add the numerator $$ 0+12+36+8=56 $$ ##### Step 3. Add the denominator $$ 0+0.3+0.6+0.1=1 $$ ##### Step 4. Divide $$ z^*=\frac{56}{1}=56 $$ **Final answer.** 56. ## For a continuous shape, integrate the area $$ z^*=\frac{\int z\mu(z)\,dz}{\int\mu(z)\,dz} $$ The denominator is the area under the aggregate. The numerator is its first moment about zero: area weighted by horizontal location. For a shape made of non-overlapping rectangles and triangles, compute each part’s area A and its own centroid c, then use $$ z^*=(\sum A_jc_j)/(\sum A_j) $$ . Overlapping regions of a max-aggregate must not be counted twice. **An aggregate is a rectangle of height 0.5 from z=0 to z=4, followed by a right triangle decreasing from height 0.5 at z=4 to 0 at z=6. Find its area centroid.** [3 marks · Practice] #### Given Rectangle [0,4], height 0.5; triangle with vertices (4,0),(4,0.5),(6,0). #### To find Centroid of the combined area. #### Formula and why it applies The parts do not overlap except at their boundary. Weight each part’s horizontal centroid by its area. #### Solution ##### Step 1. Rectangle area and centre $$ A_1=4\times0.5=2\qquad c_1=(0+4)/2=2 $$ ##### Step 2. Triangle area and centre $$ A_2=\tfrac12\times2\times0.5=0.5 $$ $$ c_2=(4+4+6)/3=14/3 $$ ##### Step 3. Add first moments $$ A_1c_1+A_2c_2=2(2)+0.5(14/3)=19/3 $$ ##### Step 4. Divide by total area $$ z^*=(19/3)/(2+0.5)=38/15\approx2.533333 $$ **Final answer.** Centroid ≈ 2.533333. The plateau MoM would be 2, so different methods can legitimately give different results. **Yaad rakho:** Centroid = har location × uska weight, total karke total weight se divide. --- ## TEE · Weighted average and centroid are not automatically the same Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--weighted-average/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/weighted-average Definitions → membership graphs → operations → rules → defuzzification → complete controller. This is the same weighted-average calculation taught in Essential maths. Here a representative output value c is weighted by a rule strength α. Keep the pair together: strength .3 belonging to speed 65 contributes .3×65. The denominator is the total strength. It is not automatically the number of rules, and the representative is not automatically the area-centre of a clipped set. ## Weighted average of output representatives Some classroom solutions use one representative (often the peak) per output label and weight it by the firing strength. That is a **weighted-average or height-based calculation**. It is exact for a specified weighted combination of singleton outputs, but is not generally the centroid of a max-aggregated, clipped Mamdani shape. Always name the method. A table of memberships at actual output locations supports the discrete centroid. A list of category peaks and strengths supports the specified representative-weighted calculation. Full continuous output shapes support area centroid. **Using the weighted-average method, combine output representatives [25, 75] mph with respective strengths [0.1, 0.7].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 25 | 0.1 | | 75 | 0.7 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0.1 × 25 | 2.5 | | 0.7 × 75 | 52.5 | ##### Step 2. Add the products $$ 2.5+52.5=55 $$ ##### Step 3. Add the strengths $$ 0.1+0.7=0.8 $$ ##### Step 4. Divide $$ z^*=\frac{55}{0.8}=68.75 $$ **Final answer.** 68.75 mph. **Using the weighted-average method, combine output representatives [40, 65, 85] % fan speed with respective strengths [0, 0.3, 0.5].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 40 | 0 | | 65 | 0.3 | | 85 | 0.5 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0 × 40 | 0 | | 0.3 × 65 | 19.5 | | 0.5 × 85 | 42.5 | ##### Step 2. Add the products $$ 0+19.5+42.5=62 $$ ##### Step 3. Add the strengths $$ 0+0.3+0.5=0.8 $$ ##### Step 4. Divide $$ z^*=\frac{62}{0.8}=77.5 $$ **Final answer.** 77.5 % fan speed. **Using the weighted-average method, combine output representatives [25, 40, 60] minutes with respective strengths [0.6, 0.4, 0.2].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 25 | 0.6 | | 40 | 0.4 | | 60 | 0.2 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0.6 × 25 | 15 | | 0.4 × 40 | 16 | | 0.2 × 60 | 12 | ##### Step 2. Add the products $$ 15+16+12=43 $$ ##### Step 3. Add the strengths $$ 0.6+0.4+0.2=1.2 $$ ##### Step 4. Divide $$ z^*=\frac{43}{1.2}=35.833333333 $$ **Final answer.** 35.833333333 minutes. | Method | What it uses | Limitation | | --- | --- | --- | | MoM | Locations at maximum membership | Ignores lower portions of the shape; can change abruptly | | Area centroid | Entire aggregated output area | Needs an area/moment calculation or numerical integration | | Representative weighted average | Supplied representative values and strengths | Not interchangeable with a clipped Mamdani area centroid | No method is universally “most accurate”. The correct exam calculation is the method specified by the question. If all memberships or all weights are zero, division by zero is undefined; a controller needs an explicit fallback rule. **Yaad rakho:** Formula similar dikhega, par weight kya hai? Membership, area, ya rule strength—pehle woh identify karo. ## Faculty slide example: one temperature input This is a different example from the two-input fan design. With Warm=Tri(20,30,40) and Hot=Tri(30,40,50), temperature 32 gives Warm=(40−32)/10=0.8 and Hot=(32−30)/10=0.2. The slide uses output representatives 50 and 80. **Using the weighted-average method, combine output representatives [50, 80] % fan speed with respective strengths [0.8, 0.2].** [3 marks · Practice] #### Given | Output representative | Strength | | --- | --- | | 50 | 0.8 | | 80 | 0.2 | #### To find One crisp output by the specified weighted-average method. #### Formula and why it applies Each representative is multiplied by its strength; add these products and divide by the total strength. Here α is a strength and z is a representative value. These are not automatically the full area of a clipped Mamdani set. $$ z^*=\frac{\sum_i\alpha_i z_i}{\sum_i\alpha_i} $$ #### Solution ##### Step 1. Multiply each value by its strength | Strength × value | Product | | --- | --- | | 0.8 × 50 | 40 | | 0.2 × 80 | 16 | ##### Step 2. Add the products $$ 40+16=56 $$ ##### Step 3. Add the strengths $$ 0.8+0.2=1 $$ ##### Step 4. Divide $$ z^*=\frac{56}{1}=56 $$ **Final answer.** 56 % fan speed. --- ## TEE · Class fan question: 32°C, 7 people, every rule and step Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--class-fan/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/class-fan Definitions → membership graphs → operations → rules → defuzzification → complete controller. There are 3 temperature labels and 3 occupancy labels. Pair each temperature label with each occupancy label: 3×3=9 possible AND rules. At the specified input, any rule containing a zero input membership has strength min(0,other)=0 and contributes no output. The rules with nonzero strength remain; their output labels come from the supplied rule table, not from the magnitude of the strength. This is the fan-controller design question shared in class: temperature 32°C and occupancy 7. The photos contain inconsistent graph breakpoints and arithmetic. A design question permits a stated design; below, every membership function is specified explicitly. These choices make the solution reproducible and should not be read as a uniquely recoverable answer from the photos. **Design a Mamdani fan controller for temperature 0–40°C and occupancy 0–10, with three input labels each and five fan-speed labels on 0–100%. For T=32°C and occupancy=7, use the membership functions and nine rules below. Calculate all memberships, firing strengths, and active output sets. Find fan speed by (a) mean of maxima and (b) the specified representative-weighted average.** [10 marks · Class-question adaptation; full design] #### Given | Variable | Low / Very Low | Medium / Low | High / Medium | High output | Very High output | | --- | --- | --- | --- | --- | --- | | Temperature | Tri(0,10,20) | Tri(15,25,35) | Tri(25,35,40) | — | — | | Occupancy | Tri(0,2,4) | Tri(3,5,7) | Tri(6,8,10) | — | — | | Fan speed | Tri(0,10,20) | Tri(10,25,40) | Tri(20,40,60) | Tri(50,65,80) | Tri(70,85,100) | Input T=32°C; occupancy=7. Use min-AND, min-clipping and max-aggregation. | Temperature | Occupancy | Fan output | | --- | --- | --- | | Low | Low | Very Low | | Low | Medium | Low | | Low | High | Medium | | Medium | Low | Low | | Medium | Medium | Medium | | Medium | High | High | | High | Low | Medium | | High | Medium | High | | High | High | Very High | #### To find Input memberships → nine rule strengths → clipped output → two named crisp outputs. #### Formula and why it applies Use the appropriate triangle branch for membership. AND takes the smaller degree. Each output is clipped at that strength; rules with the same consequent combine by max. MoM uses only the highest plateau. The requested representative weighted average uses category peaks and strengths. $$ \alpha_r=\min(\mu_{T,r},\mu_{O,r})\quad\mu_{out}(z)=\max_r\min(\alpha_r,\mu_{C_r}(z)) $$ #### Solution ##### Step 1. Read the temperature graph Swipe across the diagram to read its labels ↔ ![At 32, Medium is on its falling side; High is on its rising side.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fan_temp.434c9b6635.svg) Diagram: At 32, Medium is on its falling side; High is on its rising side.. Labels: Chosen temperature membership functions | 0 | 0.5 | 1 | 0 | 10 | 15 | 20 | 25 | 32 | 35 | 40 | μ | Temperature in °C | Low | Medium | High | 32°C At 32, Medium is on its falling side; High is on its rising side. $$ \mu_L(32)=0 $$ $$ \mu_M(32)=\frac{35-32}{35-25}=\frac3{10}=0.3 $$ $$ \mu_H(32)=\frac{32-25}{35-25}=\frac7{10}=0.7 $$ ##### Step 2. Read the occupancy graph Swipe across the diagram to read its labels ↔ ![At 7, Medium reaches its right zero and High is halfway up its rising edge.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fan_people.1861531dd1.svg) Diagram: At 7, Medium reaches its right zero and High is halfway up its rising edge.. Labels: Chosen occupancy membership functions | 0 | 0.5 | 1 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | μ | Number of people | Low | Medium | High | 7 people At 7, Medium reaches its right zero and High is halfway up its rising edge. $$ \mu_L(7)=0\qquad\mu_M(7)=0 $$ $$ \mu_H(7)=\frac{7-6}{8-6}=\frac12=0.5 $$ ##### Step 3. Evaluate all nine rules | T label / degree | O label / degree | min = strength | Output | | --- | --- | --- | --- | | Low / 0 | Low / 0 | 0 | Very Low | | Low / 0 | Medium / 0 | 0 | Low | | Low / 0 | High / 0.5 | 0 | Medium | | Medium / 0.3 | Low / 0 | 0 | Low | | Medium / 0.3 | Medium / 0 | 0 | Medium | | Medium / 0.3 | High / 0.5 | 0.3 | High | | High / 0.7 | Low / 0 | 0 | Medium | | High / 0.7 | Medium / 0 | 0 | High | | High / 0.7 | High / 0.5 | 0.5 | Very High | ##### Step 4. Clip and aggregate $$ \mu_{out}(z)=\max\{\min(0.3,\mu_H(z)),\min(0.5,\mu_{VH}(z))\} $$ Swipe across the diagram to read its labels ↔ ![High is clipped at 0.3; Very High is clipped at 0.5. Use the upper envelope where the two shapes overlap.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fan_output.8708d0ea77.svg) Diagram: High is clipped at 0.3; Very High is clipped at 0.5. Use the upper envelope where the two shapes overlap.. Labels: Two active output sets | 0 | 0.5 | 1 | 50 | 54.5 | 65 | 70 | 75 | 77.5 | 85 | 92.5 | 100 | μ | Fan speed (%) | High | Very High | clip 0.3 | clip 0.5 High is clipped at 0.3; Very High is clipped at 0.5. Use the upper envelope where the two shapes overlap. ##### Step 5. Find the highest plateau for MoM The maximum is 0.5, belonging to Very High. Its triangle is (70,85,100). $$ \frac{z_L-70}{15}=0.5\Rightarrow z_L=77.5 $$ $$ \frac{100-z_R}{15}=0.5\Rightarrow z_R=92.5 $$ $$ z^*_{MoM}=\frac{77.5+92.5}{2}=85 $$ ##### Step 6. Multiply the representatives by strengths $$ 0.3\times65=19.5 $$ $$ 0.5\times85=42.5 $$ ##### Step 7. Add, then divide for the weighted average $$ z^*_{WA}=\frac{19.5+42.5}{0.3+0.5}=\frac{62}{0.8}=77.5 $$ **Final answer.** For this explicitly stated design: MoM = 85%; representative-weighted output = 77.5%. These are different methods. 77.5% is not claimed to be the true continuous Mamdani area centroid. **How to adapt it in the exam:** if breakpoints or output representatives differ, recompute the memberships from those data. The sequence of operations stays unchanged. If the question explicitly asks for the area centroid, integrate the max-aggregated shape; do not silently substitute the weighted average above. **Yaad rakho:** Temperature aur people ka degree nikalo → 9 min → active outputs clip → jo method poocha hai woh lagao. ## Why the Very High plateau has midpoint 85 For the stated Very High triangle (70,85,100) clipped at height 0.5, solve the rising edge: (z−70)/(85−70)=0.5, so z−70=0.5×15=7.5 and z=77.5. On the falling edge, (100−z)/(100−85)=0.5, so 100−z=7.5 and z=92.5. Thus the highest flat interval is [77.5,92.5]. Its midpoint is (77.5+92.5)/2=170/2=85. Continue with the [complete area-centroid calculation for this same fan design](https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--centroid-comparison/): 77.086956522%, with all overlap breakpoints and area/moment arithmetic shown. --- ## TEE · The same fan controller: true area centroid versus MoM and weighted average Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--centroid-comparison/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/centroid-comparison Definitions → membership graphs → operations → rules → defuzzification → complete controller. The preceding class-fan lesson makes the input shapes and nine rules explicit because the photographs do not specify one internally consistent design. Keep that exact stated design here: temperature 32°C, occupancy 7, High output Tri(50,65,80) firing at 0.3 and Very High output Tri(70,85,100) firing at 0.5. This continuation calculates the actual area centroid of their max-aggregate. It does not reinterpret 77.5 as a centroid. **For the stated two-input class fan controller at T=32°C and occupancy=7, calculate the true continuous Mamdani area centroid of High=Tri(50,65,80) clipped at 0.3 and Very High=Tri(70,85,100) clipped at 0.5. Compare it with mean of maxima and the specified representative-weighted average.** [6 marks] Worked teaching continuation of the class-design adaptation; the mark allocation here is for practice. #### Given All other output rules have strength zero. Use min for implication, max for aggregation and speed z on 0–100%. The two active rule strengths were calculated from input memberships: min(0.3,0.5)=0.3 and min(0.7,0.5)=0.5. #### To find Every visible segment of the aggregate, its area and first moment, the centroid, and the differences between the three crisp answers. #### Formula and why it applies $$ \mu_{out}(z)=\max\{\min(0.3,\mu_H(z)),\min(0.5,\mu_{VH}(z))\} $$ $$ z^*_{COA}=\frac{\int z\mu_{out}(z)\,dz}{\int\mu_{out}(z)\,dz} $$ Centroid balances all the area under the upper envelope. A straight strip with endpoints (a,u) and (b,v) has the following area A and first moment M; adding only disjoint strips counts overlap once. $$ A=\frac{(b-a)(u+v)}2,\qquad M=\frac{b-a}{6}[(2a+b)u+(a+2b)v] $$ #### Solution ##### Step 1. Locate the two clipped plateaus $$ H:\quad z_L=50+0.3(65-50)=54.5,\quad z_R=80-0.3(80-65)=75.5 $$ $$ VH:\quad z_L=70+0.5(85-70)=77.5,\quad z_R=100-0.5(100-85)=92.5 $$ High has a 0.3 plateau from 54.5 to 75.5. Very High has a 0.5 plateau from 77.5 to 92.5. ##### Step 2. Find where the upper envelope changes owner Very High rises through the High plateau at height 0.3. Solve its rising equation: $$ (z-70)/15=0.3\quad\Rightarrow\quad z=74.5 $$ Before 74.5, clipped High supplies the upper envelope. After 74.5, Very High is above it. The descending High edge from 75.5 to 80 lies underneath and contributes no extra aggregate area. ##### Step 3. Trace the aggregate graph Swipe across the diagram to read its labels ↔ ![Fan output: trace the upper envelope before finding its centroid](https://deep-learning-study-room.vercel.app/is2/fig/deep_fan_centroid.98bf1c217d.svg) Diagram: Fan output: trace the upper envelope before finding its centroid. Labels: Fan output: trace the upper envelope before finding its centroid | High clipped at 0.3; Very High clipped at 0.5. Hidden overlap is counted once. | 0 | 0.3 | 0.5 | 50 | 54.5 | 74.5 | 77.5 | 92.5 | 100 | Centroid ≈77.087% | Fan speed z (%) | Area = 17.25; first moment = 1329.75; centroid = moment ÷ area. Read left to right: (50,0) → (54.5,0.3) → (74.5,0.3) → (77.5,0.5) → (92.5,0.5) → (100,0). Membership is zero outside this support. Each adjacent pair is one straight strip in the table. ##### Step 4. Work the rising first strip explicitly $$ A_1=(54.5-50)(0+0.3)/2=0.675 $$ $$ M_1=\frac{4.5}{6}[(100+54.5)0+(50+109)0.3]=35.775 $$ The first moment has units of speed multiplied by area. Dividing total moment by total area returns speed. ##### Step 5. Calculate all disjoint strips | a | b | u | v | Area A | First moment M | | --- | --- | --- | --- | --- | --- | | 50 | 54.5 | 0 | 0.3 | 0.675 | 35.775 | | 54.5 | 74.5 | 0.3 | 0.3 | 6 | 387 | | 74.5 | 77.5 | 0.3 | 0.5 | 1.2 | 91.35 | | 77.5 | 92.5 | 0.5 | 0.5 | 7.5 | 637.5 | | 92.5 | 100 | 0.5 | 0 | 1.875 | 178.125 | $$ \sum A=0.675+6+1.2+7.5+1.875=17.25 $$ $$ \sum M=35.775+387+91.35+637.5+178.125=1329.75 $$ ##### Step 6. Divide for the actual centroid $$ z^*_{COA}=1329.75/17.25=1773/23\approx77.086956522\% $$ ##### Step 7. Compare methods on the same aggregate | Method | Calculation | Answer | | --- | --- | --- | | Area centroid | 1329.75 ÷ 17.25 | 77.086956522% | | Mean of maxima | (77.5+92.5) ÷ 2 | 85% | | Specified representative weighted average | (0.3×65+0.5×85) ÷ (0.3+0.5) | 77.5% | MoM uses only the highest plateau. The representative method uses two chosen peaks and two rule strengths. The area centroid uses the full max-aggregate, so these are three legitimate but different calculations. **Final answer.** Area centroid ≈77.086956522%; MoM=85%; representative-weighted output=77.5%. Report the method with the value. The centroid is inside the support [50,100], as a balance point must be. **Boundary coverage of this teaching design.** With the ordinary triangles stated in the class adaptation, every temperature degree is zero at T=0 or T=40, and every occupancy degree is zero at occupancy=0 or 10. Then no rule fires and an area centroid is undefined. To make a controller cover the entire declared input range, explicitly redesign outer sets as shoulders or specify an out-of-coverage rule; do not silently divide by zero. The worked T=32, occupancy=7 case has positive area and is unaffected. **Yaad rakho:** Centroid mein poore shaded area ka balance chahiye. Peak × strength wala shortcut same calculation nahi hai. --- ## TEE · Fuzzy theory: definitions, architecture, advantages and applications Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--exam-theory/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/exam-theory Definitions → membership graphs → operations → rules → defuzzification → complete controller. **Explain fuzzy logic with one application.** [2 marks · Practice] #### Answer Fuzzy logic represents gradual truth or category membership using values between 0 and 1. A temperature controller can use Low, Medium and High temperature sets. It evaluates rules such as IF temperature is High THEN fan speed is High, combines the rule outputs and defuzzifies them into a numerical fan speed. This permits gradual control around category boundaries. **Give the advantages and limitations of a Mamdani fuzzy controller.** [3 marks · Practice] #### Answer | Advantages | Limitations | | --- | --- | | Human-readable rules express expert knowledge | Membership functions and rules require careful design | | Can model gradual boundaries without an exact analytic plant model | Rule count can grow rapidly with more inputs and categories | | Handles overlapping linguistic descriptions | A poorly chosen rule base can produce poor or uncovered outputs | | Suitable for many control applications | Does not automatically learn or guarantee optimal control | **Describe the design of a fuzzy fan controller.** [3 marks · Practice] #### Answer Choose temperature and occupancy as inputs and fan speed as output. Define their ranges and Low/Medium/High membership functions. Construct a rule table linking input labels to output labels. Fuzzify measured inputs, use min to obtain AND-rule strengths, clip the consequent sets and max-aggregate them. Apply the specified defuzzification method to obtain a fan speed. Test the design at normal inputs and range boundaries. ## Other applications you can explain | Application | Inputs | Example rule | Output | | --- | --- | --- | --- | | Washing machine | Dirtiness, load | IF dirt High AND load High THEN wash time Long | Wash duration | | Loan decision support | Income, credit score | IF income High AND credit Good THEN approval suitability High | Suitability score (not an automatic legal lending decision) | | Fire warning | Temperature, smoke | IF temperature High OR smoke High THEN risk High | Alarm-risk score | | Vehicle control | Distance, relative speed | IF distance Small AND closing speed High THEN braking Strong | Braking command | **Yaad rakho:** Theory answer mein definition → blocks/steps → ek concrete example. Sirf headings se marks nahi milte. --- ## TEE · Gaussian, singleton and bell memberships Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-core--smooth-memberships/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-core/smooth-memberships Definitions → membership graphs → operations → rules → defuzzification → complete controller. For a Gaussian membership, c is the input at the centre and σ controls the width. Compute x−c, square it, divide by 2σ², put a minus sign in front, and apply eˣ. The maximum membership is 1 at x=c because the exponent becomes 0. Increasing σ makes the same distance from the centre count as a smaller relative distance, so the curve becomes wider. A Gaussian membership is not the same formula as a normal probability density: there is no 1/(√(2π)σ) multiplier here. ## A smooth membership function A Gaussian membership function reaches 1 at its centre c and decreases smoothly on both sides. Its width σ must be positive. This is a fuzzy membership curve, not a normalised probability density: there is no 1/(√(2π)σ) factor. $$ \mu(x)=\exp\left(-\frac{(x-c)^2}{2\sigma^2}\right) $$ exp(u) means eᵘ. The calculator evaluates it using eˣ; calculate the entire negative exponent first. A squared distance treats equal left and right offsets equally. **For Gaussian membership with c=10 and σ=2, find μ(10).** [2 marks · Practice] #### Given x=10; centre c=10; width σ=2. #### To find Membership degree. #### Formula and why it applies Use squared distance from the centre in the Gaussian membership formula. #### Solution ##### Step 1. Distance from the centre $$ 10-10=0 $$ ##### Step 2. Square the distance and width $$ (0)^2=0\qquad2\sigma^2=2(2^2)=8 $$ ##### Step 3. Exponent $$ -0/8=0 $$ ##### Step 4. Evaluate e to this power $$ \mu(10)=e^{0}\approx1 $$ **Final answer.** 1. **For Gaussian membership with c=10 and σ=2, find μ(12).** [2 marks · Practice] #### Given x=12; centre c=10; width σ=2. #### To find Membership degree. #### Formula and why it applies Use squared distance from the centre in the Gaussian membership formula. #### Solution ##### Step 1. Distance from the centre $$ 12-10=2 $$ ##### Step 2. Square the distance and width $$ (2)^2=4\qquad2\sigma^2=2(2^2)=8 $$ ##### Step 3. Exponent $$ -4/8=-0.5 $$ ##### Step 4. Evaluate e to this power $$ \mu(12)=e^{-0.5}\approx0.60653066 $$ **Final answer.** 0.60653066. **For Gaussian membership with c=10 and σ=2, find μ(14).** [2 marks · Practice] #### Given x=14; centre c=10; width σ=2. #### To find Membership degree. #### Formula and why it applies Use squared distance from the centre in the Gaussian membership formula. #### Solution ##### Step 1. Distance from the centre $$ 14-10=4 $$ ##### Step 2. Square the distance and width $$ (4)^2=16\qquad2\sigma^2=2(2^2)=8 $$ ##### Step 3. Exponent $$ -16/8=-2 $$ ##### Step 4. Evaluate e to this power $$ \mu(14)=e^{-2}\approx0.135335283 $$ **Final answer.** 0.135335283. ## Singleton A singleton fuzzy set has membership 1 at exactly one specified point and 0 elsewhere. For a singleton at c=50: μ(50)=1, μ(49)=0 and μ(51)=0. A singleton consequent is represented by one numerical output; this is why a specified weighted average of singleton outputs is a legitimate calculation. ## Generalised bell (when its formula is supplied) $$ \mu(x)=\frac1{1+\left|\frac{x-c}{a}\right|^{2b}} $$ c is the centre, a>0 controls width and b>0 controls the slope. Absolute value |u| is the nonnegative magnitude: |−2|=2. At x=c membership is 1; at x=c±a membership is 1/2. **For a=2, b=1 and c=10, find generalised-bell membership at x=14.** [2 marks · Practice] #### Given a=2, b=1, c=10; x=14. #### To find μ(14). #### Formula and why it applies Subtract the centre, divide by a, take the absolute value, raise to 2b, then use the reciprocal. #### Solution ##### Step 1. Scaled distance $$ |(14-10)/2|=|2|=2 $$ ##### Step 2. Power $$ 2^{2(1)}=4 $$ ##### Step 3. Final reciprocal $$ \mu(14)=1/(1+4)=0.2 $$ **Final answer.** 0.2. **Yaad rakho:** Gaussian MF ka centre 1 hota hai. GMM density ke normalising factor ko yahan mat lagao. --- ## TEE · Cartesian product: every cell explained Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-relations--cartesian/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-relations/cartesian Longer Unit 2 numerical practice, also useful for TEE. ## How a row membership and a column membership meet Let A₁=.6 and A₂=.2 label two rows. Let B₁=.3, B₂=.8 and B₃=.5 label three columns. Cell R₂₃ belongs to row 2 and column 3, so use A₂=.2 and B₃=.5. Their min is .2. You do not multiply 2×3; those subscripts only locate the cell. $$ R_{23}=\min(A_2,B_3)=\min(0.2,0.5)=0.2 $$ There are two choices of row and three of column, so fill six cells in a 2×3 rectangle. Keep the row and column labels when reporting a relation. ## What is a fuzzy relation? A fuzzy set gives a membership to one element. A **fuzzy relation** gives a degree to an ordered pair. A matrix is a convenient table: rows name the first element and columns name the second. For the standard min-based Cartesian product R=A×B, put the A memberships down the rows and the B memberships across the columns. Each cell is the smaller of its row and column memberships. $$ R_{ij}=\min(\mu_A(a_i),\mu_B(b_j)) $$ **Find the fuzzy Cartesian product A×B for memberships A=[0.6, 0.2] and B=[0.3, 0.8, 0.5].** [3 marks · Practice] #### Given Ordered A memberships [0.6, 0.2]; B memberships [0.3, 0.8, 0.5]. #### To find A 2×3 relation matrix. #### Formula and why it applies Every row corresponds to an A element and every column to a B element; apply min in every cell. $$ R_{ij}=\min(A_i,B_j) $$ #### Solution ##### Step 1. Determine dimensions 2 row elements × 3 column elements. ##### Step 2. Compute each cell | A ↓ / B → | 0.3 | 0.8 | 0.5 | | --- | --- | --- | --- | | 0.6 | min(0.6,0.3) = 0.3 | min(0.6,0.8) = 0.6 | min(0.6,0.5) = 0.5 | | 0.2 | min(0.2,0.3) = 0.2 | min(0.2,0.8) = 0.2 | min(0.2,0.5) = 0.2 | ##### Step 3. Write the matrix $$ R=\begin{bmatrix}0.3&0.6&0.5\\0.2&0.2&0.2\end{bmatrix} $$ **Final answer.** R is the matrix in Step 3. **Find the fuzzy Cartesian product A×B for memberships A=[1, 0.4] and B=[0.7, 0].** [3 marks · Practice] #### Given Ordered A memberships [1, 0.4]; B memberships [0.7, 0]. #### To find A 2×2 relation matrix. #### Formula and why it applies Every row corresponds to an A element and every column to a B element; apply min in every cell. $$ R_{ij}=\min(A_i,B_j) $$ #### Solution ##### Step 1. Determine dimensions 2 row elements × 2 column elements. ##### Step 2. Compute each cell | A ↓ / B → | 0.7 | 0 | | --- | --- | --- | | 1 | min(1,0.7) = 0.7 | min(1,0) = 0 | | 0.4 | min(0.4,0.7) = 0.4 | min(0.4,0) = 0 | ##### Step 3. Write the matrix $$ R=\begin{bmatrix}0.7&0\\0.4&0\end{bmatrix} $$ **Final answer.** R is the matrix in Step 3. **Find the fuzzy Cartesian product A×B for memberships A=[0.2, 0.9, 0.5] and B=[0.6, 0.1].** [3 marks · Practice] #### Given Ordered A memberships [0.2, 0.9, 0.5]; B memberships [0.6, 0.1]. #### To find A 3×2 relation matrix. #### Formula and why it applies Every row corresponds to an A element and every column to a B element; apply min in every cell. $$ R_{ij}=\min(A_i,B_j) $$ #### Solution ##### Step 1. Determine dimensions 3 row elements × 2 column elements. ##### Step 2. Compute each cell | A ↓ / B → | 0.6 | 0.1 | | --- | --- | --- | | 0.2 | min(0.2,0.6) = 0.2 | min(0.2,0.1) = 0.1 | | 0.9 | min(0.9,0.6) = 0.6 | min(0.9,0.1) = 0.1 | | 0.5 | min(0.5,0.6) = 0.5 | min(0.5,0.1) = 0.1 | ##### Step 3. Write the matrix $$ R=\begin{bmatrix}0.2&0.1\\0.6&0.1\\0.5&0.1\end{bmatrix} $$ **Final answer.** R is the matrix in Step 3. **Yaad rakho:** Row ka degree aur column ka degree—dono ka min us cell mein. --- ## TEE · Max–min composition: three complete matrix examples Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/fuzzy-relations--composition/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/fuzzy-relations/composition Longer Unit 2 numerical practice, also useful for TEE. ## One result cell combines alternative two-link paths For T₁₁, start at u₁ and end at w₁. If the middle domain has v₁ and v₂, there are two candidate paths. Their strengths come from matching row 1 of R with column 1 of S. | Path | First link from R | Second link from S | Path strength | | --- | --- | --- | --- | | u₁ → v₁ → w₁ | .2 | .5 | min(.2,.5)=.2 | | u₁ → v₂ → w₁ | .8 | .7 | min(.8,.7)=.7 | Choose the stronger path: max(.2,.7)=.7, so T₁₁=.7. The middle label j runs through v₁ and v₂. The labels i and k stay fixed at 1 while this one cell is calculated. Then move to the next result cell. **Yaad rakho:** Ek path mein dono links chahiye, isliye min. Alag paths mein best chalega, isliye max. ## Composition connects through a middle element Suppose R relates U to V and S relates V to W. To find how strongly u connects to w through each middle v, take min of the two links. Then choose the strongest such path using max. $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ We call this matrix operation R composed with S here. Some books write the symbolic order in reverse. State the formula and row/column labels, especially when a question says S∘R. Composition requires the intermediate domains and their ordering to match; having two 3×3 matrices alone does not prove their labels match. **Calculate the max–min composition using $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ .** [3 marks · Practice] #### Given $$ R=\begin{bmatrix}0.2&0.8\\0.6&0.3\end{bmatrix} $$ $$ S=\begin{bmatrix}0.5&0.9\\0.7&0.1\end{bmatrix} $$ #### To find Every entry of T. #### Formula and why it applies Pair each row entry with the corresponding column entry; min for a two-link path, max across the possible middle elements. This is not ordinary matrix multiplication. #### Solution ##### Step 1. Cell (1,1): row 1 of R with column 1 of S $$ T_{11}=\max\{\min(0.2,0.5),\min(0.8,0.7)\} $$ $$ =\max\{0.2,0.7\}=0.7 $$ ##### Step 2. Cell (1,2): row 1 of R with column 2 of S $$ T_{12}=\max\{\min(0.2,0.9),\min(0.8,0.1)\} $$ $$ =\max\{0.2,0.1\}=0.2 $$ ##### Step 3. Cell (2,1): row 2 of R with column 1 of S $$ T_{21}=\max\{\min(0.6,0.5),\min(0.3,0.7)\} $$ $$ =\max\{0.5,0.3\}=0.5 $$ ##### Step 4. Cell (2,2): row 2 of R with column 2 of S $$ T_{22}=\max\{\min(0.6,0.9),\min(0.3,0.1)\} $$ $$ =\max\{0.6,0.1\}=0.6 $$ **Final answer.** $$ T=\begin{bmatrix}0.7&0.2\\0.5&0.6\end{bmatrix} $$ . **Calculate the max–min composition using $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ .** [3 marks · Practice] #### Given $$ R=\begin{bmatrix}1&0\\0.4&0.7\end{bmatrix} $$ $$ S=\begin{bmatrix}0.2&0.6\\0.8&0.3\end{bmatrix} $$ #### To find Every entry of T. #### Formula and why it applies Pair each row entry with the corresponding column entry; min for a two-link path, max across the possible middle elements. This is not ordinary matrix multiplication. #### Solution ##### Step 1. Cell (1,1): row 1 of R with column 1 of S $$ T_{11}=\max\{\min(1,0.2),\min(0,0.8)\} $$ $$ =\max\{0.2,0\}=0.2 $$ ##### Step 2. Cell (1,2): row 1 of R with column 2 of S $$ T_{12}=\max\{\min(1,0.6),\min(0,0.3)\} $$ $$ =\max\{0.6,0\}=0.6 $$ ##### Step 3. Cell (2,1): row 2 of R with column 1 of S $$ T_{21}=\max\{\min(0.4,0.2),\min(0.7,0.8)\} $$ $$ =\max\{0.2,0.7\}=0.7 $$ ##### Step 4. Cell (2,2): row 2 of R with column 2 of S $$ T_{22}=\max\{\min(0.4,0.6),\min(0.7,0.3)\} $$ $$ =\max\{0.4,0.3\}=0.4 $$ **Final answer.** $$ T=\begin{bmatrix}0.2&0.6\\0.7&0.4\end{bmatrix} $$ . **Calculate the max–min composition using $$ T_{ik}=\max_j\min(R_{ij},S_{jk}) $$ .** [3 marks · Practice] #### Given $$ R=\begin{bmatrix}0.3&0.6&0.9\end{bmatrix} $$ $$ S=\begin{bmatrix}0.8&0.1\\0.2&0.7\\0.5&0.4\end{bmatrix} $$ #### To find Every entry of T. #### Formula and why it applies Pair each row entry with the corresponding column entry; min for a two-link path, max across the possible middle elements. This is not ordinary matrix multiplication. #### Solution ##### Step 1. Cell (1,1): row 1 of R with column 1 of S $$ T_{11}=\max\{\min(0.3,0.8),\min(0.6,0.2),\min(0.9,0.5)\} $$ $$ =\max\{0.3,0.2,0.5\}=0.5 $$ ##### Step 2. Cell (1,2): row 1 of R with column 2 of S $$ T_{12}=\max\{\min(0.3,0.1),\min(0.6,0.7),\min(0.9,0.4)\} $$ $$ =\max\{0.1,0.6,0.4\}=0.6 $$ **Final answer.** $$ T=\begin{bmatrix}0.5&0.6\end{bmatrix} $$ . **Yaad rakho:** Har path mein min, phir saare paths mein max. Multiply-and-add nahi. --- ## TEE · GA meaning, biological idea and every basic term Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--meaning/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/meaning Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. | Object | Concrete value | Do not confuse it with | | --- | --- | --- | | Individual number | Row 2 | The candidate’s value x | | Candidate value x | 11 | Its chromosome 1011 | | Chromosome | 1011 | Its fitness | | Fitness when maximising x² | 11×11=121 | Its row number or bit string | In this guide, “gene 3” counts the third bit from the **left** starting at 1. Binary place values were built from the **right** starting with weight 1. These are two labelling conventions, not contradictory place-value rules. Read the end specified in an exam. ## What a genetic algorithm does A **genetic algorithm (GA)** is a population-based optimisation method inspired by biological evolution. It maintains candidate solutions, evaluates their fitness, selects parents, generates children through crossover and mutation, and repeats the process. In biology, genes carry inherited information. In a GA, an encoding plays the role of inherited information. The analogy motivates the operators; it does not mean that a computer chromosome is literally DNA. | Term | Meaning | Four-bit example | | --- | --- | --- | | Individual | One candidate solution | x=11 | | Chromosome | The complete encoded candidate | 1011 | | Gene | One component of an encoding | The third bit | | Allele | The value at a gene | 0 or 1 for binary encoding | | Population | The collection of current individuals | [3,5,10,12] | | Fitness | A score used to favour useful candidates | For max x², fitness of 11 is 121 | | Generation | One cycle producing a new population | Select → cross → mutate → replace | | Search space | All candidate solutions permitted | 4-bit integers: 0–15 | | Elitism | Preserving best individuals unchanged | Copy the best into the next population | An objective describes what the problem wants. Fitness describes what the selection mechanism favours. For maximising a nonnegative objective, they may be the same. For minimising cost, directly selecting in proportion to cost would favour bad candidates; use a specified transformation or a comparison-based selection method. **For a maximisation GA with objective f(x)=x², find the fitness of x=3.** [2 marks · Practice] #### Given x=3; fitness equals the nonnegative objective x². #### To find Fitness, not the encoded chromosome. #### Formula and why it applies Square x to evaluate its objective; binary encoding is a separate representation step. #### Solution ##### Step 1. Substitute x $$ f(3)=3^2 $$ ##### Step 2. Multiply $$ 3\times3=9 $$ **Final answer.** Fitness = 9. **For a maximisation GA with objective f(x)=x², find the fitness of x=7.** [2 marks · Practice] #### Given x=7; fitness equals the nonnegative objective x². #### To find Fitness, not the encoded chromosome. #### Formula and why it applies Square x to evaluate its objective; binary encoding is a separate representation step. #### Solution ##### Step 1. Substitute x $$ f(7)=7^2 $$ ##### Step 2. Multiply $$ 7\times7=49 $$ **Final answer.** Fitness = 49. **For a maximisation GA with objective f(x)=x², find the fitness of x=12.** [2 marks · Practice] #### Given x=12; fitness equals the nonnegative objective x². #### To find Fitness, not the encoded chromosome. #### Formula and why it applies Square x to evaluate its objective; binary encoding is a separate representation step. #### Solution ##### Step 1. Substitute x $$ f(12)=12^2 $$ ##### Step 2. Multiply $$ 12\times12=144 $$ **Final answer.** Fitness = 144. **Yaad rakho:** Chromosome answer ka code; fitness us code wale answer ka score. --- ## TEE · Encoding: binary, real values and permutations Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--encoding/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/encoding Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. ## Expand the compact decoding formula before using it $$ x=b_1 2^3+b_2 2^2+b_3 2^1+b_4 2^0 $$ For four bits, b₁ is the leftmost bit and receives weight 8; b₄ is the rightmost and receives weight 1. In general n−i gives the power belonging to bit i. For 1011, the four terms are 1×8, 0×4, 1×2, 1×1. Add 8+0+2+1=11. For real-range mapping, d/(2ⁿ−1) is the fraction of the way through the possible codes. Multiplying by U−L travels that fraction of the physical range. Adding L shifts the result to start at the lower limit. | 3-bit code | Decoded d | Fraction d/7 | Mapped to [−2,5] | | --- | --- | --- | --- | | 000 | 0 | 0/7=0 | −2+0×7=−2 | | 011 | 3 | 3/7 | −2+(3/7)×7=1 | | 111 | 7 | 7/7=1 | −2+1×7=5 | ## Choose an encoding that can express legal solutions **Binary encoding** uses bit strings; **real-valued encoding** stores real numbers directly; **permutation encoding** stores an order, such as a TSP city tour. An operator valid for bits may create invalid city tours, so encoding and operators must fit the problem. | Representation | Candidate | Appropriate use | | --- | --- | --- | | Binary | 1011 | An unsigned integer or a coded real value | | Real-valued | [0.2, 1.7] | Continuous parameters | | Permutation | [A,C,B,D] | Visit order in TSP | ## Decode an n-bit integer $$ x=b_1 2^{n-1}+b_2 2^{n-2}+\cdots+b_n2^0 $$ b₁ is the leftmost bit, not a coefficient you have to learn. In four bits the weights are 8,4,2,1. The maximum is $$ 2^n-1 $$ . **Decode the 4-bit chromosome 0011.** [2 marks · Practice] #### Given Bits 0011; place values 8,4,2,1. #### To find Integer x. #### Formula and why it applies Multiply each bit by its matching place value, then add. #### Solution ##### Step 1. List contributions | Bit | Place value | Contribution | | --- | --- | --- | | 0 | 8 | 0 | | 0 | 4 | 0 | | 1 | 2 | 2 | | 1 | 1 | 1 | ##### Step 2. Add $$ 0+0+2+1=3 $$ **Final answer.** x=3. **Decode the 4-bit chromosome 1010.** [2 marks · Practice] #### Given Bits 1010; place values 8,4,2,1. #### To find Integer x. #### Formula and why it applies Multiply each bit by its matching place value, then add. #### Solution ##### Step 1. List contributions | Bit | Place value | Contribution | | --- | --- | --- | | 1 | 8 | 8 | | 0 | 4 | 0 | | 1 | 2 | 2 | | 0 | 1 | 0 | ##### Step 2. Add $$ 8+0+2+0=10 $$ **Final answer.** x=10. **Decode the 4-bit chromosome 1111.** [2 marks · Practice] #### Given Bits 1111; place values 8,4,2,1. #### To find Integer x. #### Formula and why it applies Multiply each bit by its matching place value, then add. #### Solution ##### Step 1. List contributions | Bit | Place value | Contribution | | --- | --- | --- | | 1 | 8 | 8 | | 1 | 4 | 4 | | 1 | 2 | 2 | | 1 | 1 | 1 | ##### Step 2. Add $$ 8+4+2+1=15 $$ **Final answer.** x=15. ## Encoding a real interval If the question maps an n-bit integer d to a real interval [L,U], first decode d, then use: $$ x=L+\frac{d}{2^n-1}(U-L) $$ **Map the 3-bit string 101 to the real interval [−2,5] using linear binary decoding.** [3 marks · Practice] #### Given n=3, L=−2, U=5; string 101. #### To find Real x. #### Formula and why it applies The smallest bit code maps to L and the largest to U; the remaining codes are equally spaced. $$ x=L+\frac d{2^n-1}(U-L) $$ #### Solution ##### Step 1. Decode integer d $$ d=1(4)+0(2)+1(1)=5 $$ ##### Step 2. Calculate interval width and maximum code $$ U-L=5-(-2)=7\qquad2^3-1=7 $$ ##### Step 3. Map to the interval $$ x=-2+(5/7)(7)=3 $$ **Final answer.** x=3. **Yaad rakho:** Pehle bits se integer. Real range mapping explicitly diya ho tabhi next formula lagao. --- ## TEE · Roulette selection, cumulative intervals and alternatives Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--selection/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/selection Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. Before the three complete sums below, read the running-total example in [Percentages, probabilities and cumulative intervals](https://deep-learning-study-room.vercel.app/read/is2/m2/read/core-maths--probability/). For weights [1,2,3,4], candidate C’s own probability is 0.3, but its interval is [0.3,0.6). A draw of 0.55 selects C because it lies inside that interval; it is not required to be less than C’s individual probability. ## Selection chooses parents, usually with replacement **Roulette-wheel selection** gives each candidate a probability proportional to a nonnegative fitness weight. A high weight occupies a larger interval on [0,1]. A supplied random number selects the interval containing it. $$ P_i=F_i/\sum_jF_j $$ Fᵢ is the fitness weight of individual i. The sum must be positive. Probabilities add to 1. Cumulative probability is the running sum and supplies the interval boundaries. **Fitness weights of successive individuals are [1, 2, 3, 4]. Select parents using random numbers [0.05, 0.25, 0.55, 0.9].** [3 marks · Practice] #### Given Fitness weights [1, 2, 3, 4]; draws [0.05, 0.25, 0.55, 0.9]. #### To find Parent row number for each draw. #### Formula and why it applies Divide by the fitness sum, form cumulative intervals, then locate each random draw. Use half-open intervals [lower,upper); the final upper endpoint 1 is included. #### Solution ##### Step 1. Sum fitness $$ \sum F=1+2+3+4=10 $$ ##### Step 2. Find probabilities and cumulative bounds | Individual | Fitness | Probability | Cumulative | | --- | --- | --- | --- | | 1 | 1 | 0.1 | 0.1 | | 2 | 2 | 0.2 | 0.3 | | 3 | 3 | 0.3 | 0.6 | | 4 | 4 | 0.4 | 1 | ##### Step 3. Locate each draw | Draw | Selected individual | | --- | --- | | 0.05 | 1 | | 0.25 | 2 | | 0.55 | 3 | | 0.9 | 4 | **Final answer.** Selected row numbers: [1, 2, 3, 4]. Repeated selection is permitted. **Fitness weights of successive individuals are [4, 4, 2]. Select parents using random numbers [0.2, 0.6, 0.85].** [3 marks · Practice] #### Given Fitness weights [4, 4, 2]; draws [0.2, 0.6, 0.85]. #### To find Parent row number for each draw. #### Formula and why it applies Divide by the fitness sum, form cumulative intervals, then locate each random draw. Use half-open intervals [lower,upper); the final upper endpoint 1 is included. #### Solution ##### Step 1. Sum fitness $$ \sum F=4+4+2=10 $$ ##### Step 2. Find probabilities and cumulative bounds | Individual | Fitness | Probability | Cumulative | | --- | --- | --- | --- | | 1 | 4 | 0.4 | 0.4 | | 2 | 4 | 0.4 | 0.8 | | 3 | 2 | 0.2 | 1 | ##### Step 3. Locate each draw | Draw | Selected individual | | --- | --- | | 0.2 | 1 | | 0.6 | 2 | | 0.85 | 3 | **Final answer.** Selected row numbers: [1, 2, 3]. Repeated selection is permitted. **Fitness weights of successive individuals are [0, 3, 1]. Select parents using random numbers [0.2, 0.8].** [3 marks · Practice] #### Given Fitness weights [0, 3, 1]; draws [0.2, 0.8]. #### To find Parent row number for each draw. #### Formula and why it applies Divide by the fitness sum, form cumulative intervals, then locate each random draw. Use half-open intervals [lower,upper); the final upper endpoint 1 is included. #### Solution ##### Step 1. Sum fitness $$ \sum F=0+3+1=4 $$ ##### Step 2. Find probabilities and cumulative bounds | Individual | Fitness | Probability | Cumulative | | --- | --- | --- | --- | | 1 | 0 | 0 | 0 | | 2 | 3 | 0.75 | 0.75 | | 3 | 1 | 0.25 | 1 | ##### Step 3. Locate each draw | Draw | Selected individual | | --- | --- | | 0.2 | 2 | | 0.8 | 3 | **Final answer.** Selected row numbers: [2, 3]. Repeated selection is permitted. ## Other selection methods | Method | Procedure | Why it is used | | --- | --- | --- | | Tournament | Sample a group and take its best candidate | Only objective comparisons are needed | | Rank | Sort candidates; assign weights based on rank | Reduces sensitivity to a single huge raw fitness | | Elitist survival | Preserve top individuals unchanged into the next population | Protects good solutions from destructive variation | Selection is not automatically “replace the weakest with the fittest”. That is one possible classroom reproduction rule. If a problem gives selected parents, a rank rule, tournament contestants or roulette draws, follow that exact instruction. **Yaad rakho:** Probability nikalo → cumulative banao → random number ka interval dekho. --- ## TEE · Crossover and mutation: four complete worked variants Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--operators/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/operators Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. ## Cutting means splitting the written string The **prefix** is the part before the cut; the **suffix** is the part after it. “10 joined with 00 gives 1000” means writing strings next to each other. It is not the arithmetic addition 10+00. | Operation | Parents / instruction | Construction | Children | | --- | --- | --- | --- | | One-point crossover | 10|11 and 01|00 | First prefix + second tail; second prefix + first tail | 1000 and 0111 | | Two-point crossover | 1|01|10 and 0|10|01 | Exchange only the middle blocks | 11010 and 00101 | | Uniform crossover | A=1010, B=0101, mask=1100 | Child 1 takes A where mask=1 and B where mask=0; child 2 reverses this | 1001 and 0110 | | Mutation | 1000; flip position 3 from left | The third bit is 0; replace it by 1; keep every other bit | 1010 | These crossover operators are alternatives. Apply the operator stated in the question; do not apply all three in one generation. Mutation is a separate step after crossover in the worked exam workflow. ## Crossover combines; mutation changes In **single-point crossover**, cut both parents after the specified bit. Child 1 keeps parent 1’s prefix and takes parent 2’s suffix; child 2 does the reverse. A cut after bit 2 in a four-bit string looks like 10|11. **Cross parents 1011 and 0100 after bit 2. Then flip bit 4 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 1011, 0100; cut after bit 2; mutation bit 4. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 10 | 11 and 01 | 00 ##### Step 2. Exchange tails 10 + 00 → 1000 01 + 11 → 0111 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 1000 | 0 | 1 | 1001 | | 0111 | 1 | 0 | 0110 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 1001 | 8 + 0 + 0 + 1 | 9 | | 0110 | 0 + 4 + 2 + 0 | 6 | **Final answer.** Final children 1001 (9) and 0110 (6). **Cross parents 0011 and 1100 after bit 1. Then flip bit 2 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 0011, 1100; cut after bit 1; mutation bit 2. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 0 | 011 and 1 | 100 ##### Step 2. Exchange tails 0 + 100 → 0100 1 + 011 → 1011 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 0100 | 1 | 0 | 0000 | | 1011 | 0 | 1 | 1111 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 0000 | 0 + 0 + 0 + 0 | 0 | | 1111 | 8 + 4 + 2 + 1 | 15 | **Final answer.** Final children 0000 (0) and 1111 (15). **Cross parents 1110 and 0001 after bit 3. Then flip bit 1 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 1110, 0001; cut after bit 3; mutation bit 1. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 111 | 0 and 000 | 1 ##### Step 2. Exchange tails 111 + 1 → 1111 000 + 0 → 0000 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 1111 | 1 | 0 | 0111 | | 0000 | 0 | 1 | 1000 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 0111 | 0 + 4 + 2 + 1 | 7 | | 1000 | 8 + 0 + 0 + 0 | 8 | **Final answer.** Final children 0111 (7) and 1000 (8). **Cross parents 1010 and 1010 after bit 2. Then flip bit 4 of both children, counting from the left. Decode the final children.** [3 marks · Practice] #### Given Parents 1010, 1010; cut after bit 2; mutation bit 4. #### To find Children after crossover, after mutation and their decimal values. #### Formula and why it applies Exchange only suffixes. Mutation means 0→1 or 1→0; it does not mean set the bit to 1. #### Solution ##### Step 1. Mark the cut 10 | 10 and 10 | 10 ##### Step 2. Exchange tails 10 + 10 → 1010 10 + 10 → 1010 ##### Step 3. Flip the specified bit | Crossover child | Bit before | Bit after | Mutated child | | --- | --- | --- | --- | | 1010 | 0 | 1 | 1011 | | 1010 | 0 | 1 | 1011 | ##### Step 4. Decode | String | Place-value sum | x | | --- | --- | --- | | 1011 | 8 + 0 + 2 + 1 | 11 | | 1011 | 8 + 0 + 2 + 1 | 11 | **Final answer.** Final children 1011 (11) and 1011 (11). **Two-point crossover** exchanges the segment between two cuts. **Uniform crossover** uses a given mask to choose, at each position, which parent contributes a gene. Mutation may flip a bit, perturb a real value, or swap two positions in a permutation. These are different operators; apply the one named in the question. **Yaad rakho:** Crossover = tail exchange. Mutation = specified bit flip. Flip ka matlab 1 bhi 0 banega. --- ## TEE · GA architecture, two-generation flow, constraints and theory Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--algorithm/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/algorithm Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. | Read this | Meaning | Concrete example | | --- | --- | --- | | Feasible | Obeys every stated constraint | x≤10: x=9 qualifies; x=12 does not | | Penalty | A deduction for violating a constraint | For maximisation, subtract 100×excess | | Elitism | Explicitly preserve a best old individual | Only apply if the question specifies it | | Replacement | Choose what population the next generation uses | Our full-replacement examples use the new children | | Stopping condition | A rule ending the repeated process | Stop after two complete generations | ## A complete GA generation Encode + evaluate → Select parents → Crossover → Mutation → Evaluate childrenreplace and repeat A **simple generational GA** replaces the population in generations using a chosen encoding, selection rule and variation operators. A more general GA may use steady-state replacement, different encodings, adaptive operators or explicit elitism. Always state the choices used in your worked answer. ## What “iteration 2” means here Initial population → first generation’s selection/crossover/mutation → new population → second generation’s selection/crossover/mutation → final population. Stopping after the first children and labelling them iteration 2 is not two full generations. ## Stopping conditions Stop after a fixed generation budget, after reaching a specified acceptable objective, or after little improvement for a stated number of generations. A finite run does not prove the global optimum. ## Constraints A **constraint** restricts allowed solutions, such as 0≤x≤15 or “visit every city once”. Common approaches are valid-by-design encoding, repairing an invalid child, rejecting it, or using a penalty. A penalty must reduce attractiveness: subtract it in maximisation or add it to a minimised cost. Do not silently remove the original constraint because you used more bits. **A maximisation problem has objective f(x)=x² and constraint x≤10. The given penalised fitness is F(x)=x²−100 max(0,x−10). Compare x=9 and x=12.** [3 marks · Practice] #### Given Penalty coefficient 100; bound 10; candidates 9 and 12. #### To find Penalised fitness of both candidates. #### Formula and why it applies Only excess above the bound receives a penalty. #### Solution ##### Step 1. Feasible candidate 9 $$ \max(0,9-10)=0 $$ $$ F(9)=81-100(0)=81 $$ ##### Step 2. Infeasible candidate 12 $$ \max(0,12-10)=2 $$ $$ F(12)=144-100(2)=-56 $$ ##### Step 3. Compare For this maximisation fitness, 81 > −56, so x=9 is preferred. Raw objective alone would misleadingly favour 12. **Final answer.** F(9)=81; F(12)=−56. | Advantages | Limitations | | --- | --- | | Derivative-free; supports many encodings | Many objective evaluations may be costly | | Population explores several candidates | Premature convergence is possible | | Crossover and mutation can create new candidates | Parameter and encoding choices affect results | | Applicable to discontinuous or combinatorial searches | No guaranteed global optimum; variation can worsen fitness | **Yaad rakho:** Do generation = do baar poora cycle. Sirf children banana aur decode karna do alag iterations nahi. --- ## TEE · Two generations with given selection, cut and mutation Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--two-generations/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/two-generations Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. Swipe across the diagram to read its labels ↔ ![This diagram traces the first pair in generation 1. The full solution also processes the second pair and the entire second generation.](https://deep-learning-study-room.vercel.app/is2/fig/v2_ga_generation.fa7bf86c05.svg) Diagram: This diagram traces the first pair in generation 1. The full solution also processes the second pair and the entire second generation.. Labels: One complete GA generation: read from top to bottom | Selected parents | 1010 × 1100 | Cut after bit 2 | Mark the cut | 10 | 10 11 | 00 | Keep prefix, exchange suffix | After crossover | 1000 1110 | Children are 8 and 14 | Flip the last bit | 1001 1111 | 0 becomes 1 in each child | Decode and evaluate | 9² = 81 15² = 225 | These are the children’s fitness values This diagram traces the first pair in generation 1. The full solution also processes the second pair and the entire second generation. **Maximise $$ f(x)=x^2 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [3, 5, 10, 12]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 4 of children [1, 2, 3, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Current individual row numbers [3, 4, 4, 3] | | 2 | Current individual row numbers [2, 3, 2, 3] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Use exactly the supplied selection rule; replacing the weakest by the strongest is not a universal GA step. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 3 | 0011 | 3 × 3 = 9 | | 2 | 5 | 0101 | 5 × 5 = 25 | | 3 | 10 | 1010 | 10 × 10 = 100 | | 4 | 12 | 1100 | 12 × 12 = 144 | ##### Step 2. Generation 1: apply the supplied selection order The question directly supplies individual numbers [3, 4, 4, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 3 | 10 | 1010 | | 2 | 4 | 12 | 1100 | | 3 | 4 | 12 | 1100 | | 4 | 3 | 10 | 1010 | ##### Step 3. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 10 | 10 | 11 | 00 | 1000 | 1110 | | 3 & 4 | 11 | 00 | 10 | 10 | 1110 | 1000 | ##### Step 4. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1000 | Flip bit 4 | 1001 | | 2 | 1110 | Flip bit 4 | 1111 | | 3 | 1110 | Flip bit 4 | 1111 | | 4 | 1000 | Flip bit 4 | 1001 | ##### Step 5. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1001 | 8 + 0 + 0 + 1 = 9 | 9² = 81 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1111 | 8 + 4 + 2 + 1 = 15 | 15² = 225 | | 1001 | 8 + 0 + 0 + 1 = 9 | 9² = 81 | ##### Step 6. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{9+25+100+144}{4}=\frac{278}{4}=69.5 $$ $$ \overline f_{new}=\frac{81+225+225+81}{4}=\frac{612}{4}=153 $$ Best objective: 144 → 225. Higher is better. The next population is [9, 15, 15, 9]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 7. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 9 | 1001 | 9 × 9 = 81 | | 2 | 15 | 1111 | 15 × 15 = 225 | | 3 | 15 | 1111 | 15 × 15 = 225 | | 4 | 9 | 1001 | 9 × 9 = 81 | ##### Step 8. Generation 2: apply the supplied selection order The question directly supplies individual numbers [2, 3, 2, 3]. These refer to the current generation’s row numbers, not their x values. | Mating-pool slot | Chosen individual | x | Chromosome | | --- | --- | --- | --- | | 1 | 2 | 15 | 1111 | | 2 | 3 | 15 | 1111 | | 3 | 2 | 15 | 1111 | | 4 | 3 | 15 | 1111 | ##### Step 9. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 11 | 11 | 11 | 11 | 1111 | 1111 | | 3 & 4 | 11 | 11 | 11 | 11 | 1111 | 1111 | ##### Step 10. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1111 | Flip bit 4 | 1110 | | 2 | 1111 | Flip bit 4 | 1110 | | 3 | 1111 | Flip bit 4 | 1110 | | 4 | 1111 | Flip bit 4 | 1110 | ##### Step 11. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | | 1110 | 8 + 4 + 2 + 0 = 14 | 14² = 196 | ##### Step 12. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{81+225+225+81}{4}=\frac{612}{4}=153 $$ $$ \overline f_{new}=\frac{196+196+196+196}{4}=\frac{784}{4}=196 $$ Best objective: 225 → 196. Higher is better. The next population is [14, 14, 14, 14]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [14, 14, 14, 14]. Best final objective = 196 at x = [14]. --- ## TEE · Two generations including roulette selection Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--roulette-generations/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/roulette-generations Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. **Maximise $$ f(x)=x(15-x)+1 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [2, 5, 9, 12]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 4 of children [2, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Random draws [0.1, 0.4, 0.65, 0.9] | | 2 | Random draws [0.2, 0.4, 0.6, 0.95] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Use exactly the supplied selection rule; replacing the weakest by the strongest is not a universal GA step. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 2 | 0010 | 2 × (15 − 2) + 1 = 27 | | 2 | 5 | 0101 | 5 × (15 − 5) + 1 = 51 | | 3 | 9 | 1001 | 9 × (15 − 9) + 1 = 55 | | 4 | 12 | 1100 | 12 × (15 − 12) + 1 = 37 | ##### Step 2. Generation 1: calculate roulette intervals Fitness weights are nonnegative. The nonnegative objective itself is the fitness in this maximisation example. $$ \sum F=27+51+55+37\approx170 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 27 ÷ 170 ≈ 0.158823529 | 0 + 0.158823529 ≈ 0.158823529 | | 2 | 51 ÷ 170 ≈ 0.3 | 0.158823529 + 0.3 ≈ 0.458823529 | | 3 | 55 ÷ 170 ≈ 0.323529412 | 0.458823529 + 0.323529412 ≈ 0.782352941 | | 4 | 37 ÷ 170 ≈ 0.217647059 | 0.782352941 + 0.217647059 ≈ 1 | ##### Step 3. Generation 1: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.1 | 0 ≤ 0.1 < 0.158823529 | 1 | 2 | | 0.4 | 0.158823529 ≤ 0.4 < 0.458823529 | 2 | 5 | | 0.65 | 0.458823529 ≤ 0.65 < 0.782352941 | 3 | 9 | | 0.9 | 0.782352941 ≤ 0.9 < 1 | 4 | 12 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 4. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 00 | 10 | 01 | 01 | 0001 | 0110 | | 3 & 4 | 10 | 01 | 11 | 00 | 1000 | 1101 | ##### Step 5. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 0001 | No mutation | 0001 | | 2 | 0110 | Flip bit 4 | 0111 | | 3 | 1000 | No mutation | 1000 | | 4 | 1101 | Flip bit 4 | 1100 | ##### Step 6. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 0001 | 0 + 0 + 0 + 1 = 1 | 1(15−1)+1 = 15 | | 0111 | 0 + 4 + 2 + 1 = 7 | 7(15−7)+1 = 57 | | 1000 | 8 + 0 + 0 + 0 = 8 | 8(15−8)+1 = 57 | | 1100 | 8 + 4 + 0 + 0 = 12 | 12(15−12)+1 = 37 | ##### Step 7. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{27+51+55+37}{4}=\frac{170}{4}=42.5 $$ $$ \overline f_{new}=\frac{15+57+57+37}{4}=\frac{166}{4}=41.5 $$ Best objective: 55 → 57. Higher is better. The next population is [1, 7, 8, 12]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 8. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 1 | 0001 | 1 × (15 − 1) + 1 = 15 | | 2 | 7 | 0111 | 7 × (15 − 7) + 1 = 57 | | 3 | 8 | 1000 | 8 × (15 − 8) + 1 = 57 | | 4 | 12 | 1100 | 12 × (15 − 12) + 1 = 37 | ##### Step 9. Generation 2: calculate roulette intervals Fitness weights are nonnegative. The nonnegative objective itself is the fitness in this maximisation example. $$ \sum F=15+57+57+37\approx166 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 15 ÷ 166 ≈ 0.090361446 | 0 + 0.090361446 ≈ 0.090361446 | | 2 | 57 ÷ 166 ≈ 0.343373494 | 0.090361446 + 0.343373494 ≈ 0.43373494 | | 3 | 57 ÷ 166 ≈ 0.343373494 | 0.43373494 + 0.343373494 ≈ 0.777108434 | | 4 | 37 ÷ 166 ≈ 0.222891566 | 0.777108434 + 0.222891566 ≈ 1 | ##### Step 10. Generation 2: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.2 | 0.090361446 ≤ 0.2 < 0.43373494 | 2 | 7 | | 0.4 | 0.090361446 ≤ 0.4 < 0.43373494 | 2 | 7 | | 0.6 | 0.43373494 ≤ 0.6 < 0.777108434 | 3 | 8 | | 0.95 | 0.777108434 ≤ 0.95 < 1 | 4 | 12 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 11. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 01 | 11 | 01 | 11 | 0111 | 0111 | | 3 & 4 | 10 | 00 | 11 | 00 | 1000 | 1100 | ##### Step 12. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 0111 | No mutation | 0111 | | 2 | 0111 | Flip bit 4 | 0110 | | 3 | 1000 | No mutation | 1000 | | 4 | 1100 | Flip bit 4 | 1101 | ##### Step 13. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 0111 | 0 + 4 + 2 + 1 = 7 | 7(15−7)+1 = 57 | | 0110 | 0 + 4 + 2 + 0 = 6 | 6(15−6)+1 = 55 | | 1000 | 8 + 0 + 0 + 0 = 8 | 8(15−8)+1 = 57 | | 1101 | 8 + 4 + 0 + 1 = 13 | 13(15−13)+1 = 27 | ##### Step 14. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{15+57+57+37}{4}=\frac{166}{4}=41.5 $$ $$ \overline f_{new}=\frac{57+55+57+27}{4}=\frac{196}{4}=49 $$ Best objective: 57 → 57. Higher is better. The next population is [7, 6, 8, 13]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [7, 6, 8, 13]. Best final objective = 57 at x = [7, 8]. --- ## TEE · GA minimisation: transform cost before roulette selection Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/ga-core--minimisation/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/ga-core/minimisation Candidate → fitness → encoding → selection → crossover → mutation → two complete generations. Roulette favours a larger weight. A minimisation objective instead favours a smaller cost. The stated transformation F=1/(1+cost) reverses their order for nonnegative costs: cost 0 → weight 1/1=1; cost 3 → 1/4=.25; cost 24 → 1/25=.04. The +1 prevents division by zero. This is one specified transformation, not a universal rule for every minimisation problem. **Minimise $$ f(x)=(x-7)^2 $$ over unsigned 4-bit integers using the supplied GA settings. Perform two complete generations and compare the population fitness.** [3 marks · Practice; complete two-generation solution] #### Given Initial ordered population: [2, 4, 8, 12]. Use 4 bits, crossover after bit 2, pair mating-pool slots 1–2 and 3–4. Flip bit 4 of children [2, 4] in every generation. Replace the whole population with the four children; no elitism. | Generation | Selection instruction | | --- | --- | | 1 | Random draws [0.1, 0.4, 0.65, 0.9] | | 2 | Random draws [0.2, 0.4, 0.6, 0.95] | #### To find Both generations: evaluation, selection, crossover, mutation, decoding and comparison. #### Formula and why it applies Encoding represents x; it does not change the objective. Crossover exchanges tails. Mutation flips specified bits. Fitness must be recomputed after mutation. Roulette uses Fᵢ/ΣF. For minimisation here, use Fᵢ=1/(1+f(xᵢ)); do not directly favour large costs. #### Solution ##### Step 1. Generation 1: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 2 | 0010 | (2 − 7)² = 25 | | 2 | 4 | 0100 | (4 − 7)² = 9 | | 3 | 8 | 1000 | (8 − 7)² = 1 | | 4 | 12 | 1100 | (12 − 7)² = 25 | ##### Step 2. Generation 1: calculate roulette intervals Fitness weights are nonnegative. For this minimisation example use F=1/(1+objective), so a smaller cost receives a larger weight. | Individual | Cost | Fitness calculation | | --- | --- | --- | | 1 | 25 | 1 / (1 + 25) = 1 / 26 ≈ 0.038461538 | | 2 | 9 | 1 / (1 + 9) = 1 / 10 ≈ 0.1 | | 3 | 1 | 1 / (1 + 1) = 1 / 2 ≈ 0.5 | | 4 | 25 | 1 / (1 + 25) = 1 / 26 ≈ 0.038461538 | $$ \sum F=0.038461538+0.1+0.5+0.038461538\approx0.676923077 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 0.038461538 ÷ 0.676923077 ≈ 0.056818182 | 0 + 0.056818182 ≈ 0.056818182 | | 2 | 0.1 ÷ 0.676923077 ≈ 0.147727273 | 0.056818182 + 0.147727273 ≈ 0.204545455 | | 3 | 0.5 ÷ 0.676923077 ≈ 0.738636364 | 0.204545455 + 0.738636364 ≈ 0.943181818 | | 4 | 0.038461538 ÷ 0.676923077 ≈ 0.056818182 | 0.943181818 + 0.056818182 ≈ 1 | ##### Step 3. Generation 1: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.1 | 0.056818182 ≤ 0.1 < 0.204545455 | 2 | 4 | | 0.4 | 0.204545455 ≤ 0.4 < 0.943181818 | 3 | 8 | | 0.65 | 0.204545455 ≤ 0.65 < 0.943181818 | 3 | 8 | | 0.9 | 0.204545455 ≤ 0.9 < 0.943181818 | 3 | 8 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 4. Generation 1: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 01 | 00 | 10 | 00 | 0100 | 1000 | | 3 & 4 | 10 | 00 | 10 | 00 | 1000 | 1000 | ##### Step 5. Generation 1: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 0100 | No mutation | 0100 | | 2 | 1000 | Flip bit 4 | 1001 | | 3 | 1000 | No mutation | 1000 | | 4 | 1000 | Flip bit 4 | 1001 | ##### Step 6. Generation 1: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 0100 | 0 + 4 + 0 + 0 = 4 | (4−7)² = 9 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | | 1000 | 8 + 0 + 0 + 0 = 8 | (8−7)² = 1 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | ##### Step 7. Generation 1: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{25+9+1+25}{4}=\frac{60}{4}=15 $$ $$ \overline f_{new}=\frac{9+4+1+4}{4}=\frac{18}{4}=4.5 $$ Best objective: 1 → 1. Lower is better. The next population is [4, 9, 8, 9]. No elitism is imposed, so the best previous individual is not automatically protected. ##### Step 8. Generation 2: evaluate and encode the current population | Individual | x | Binary | Objective calculation | | --- | --- | --- | --- | | 1 | 4 | 0100 | (4 − 7)² = 9 | | 2 | 9 | 1001 | (9 − 7)² = 4 | | 3 | 8 | 1000 | (8 − 7)² = 1 | | 4 | 9 | 1001 | (9 − 7)² = 4 | ##### Step 9. Generation 2: calculate roulette intervals Fitness weights are nonnegative. For this minimisation example use F=1/(1+objective), so a smaller cost receives a larger weight. | Individual | Cost | Fitness calculation | | --- | --- | --- | | 1 | 9 | 1 / (1 + 9) = 1 / 10 ≈ 0.1 | | 2 | 4 | 1 / (1 + 4) = 1 / 5 ≈ 0.2 | | 3 | 1 | 1 / (1 + 1) = 1 / 2 ≈ 0.5 | | 4 | 4 | 1 / (1 + 4) = 1 / 5 ≈ 0.2 | $$ \sum F=0.1+0.2+0.5+0.2\approx1 $$ | Individual | Probability: weight ÷ total | Cumulative: old boundary + probability | | --- | --- | --- | | 1 | 0.1 ÷ 1 ≈ 0.1 | 0 + 0.1 ≈ 0.1 | | 2 | 0.2 ÷ 1 ≈ 0.2 | 0.1 + 0.2 ≈ 0.3 | | 3 | 0.5 ÷ 1 ≈ 0.5 | 0.3 + 0.5 ≈ 0.8 | | 4 | 0.2 ÷ 1 ≈ 0.2 | 0.8 + 0.2 ≈ 1 | ##### Step 10. Generation 2: use the supplied random draws | Random draw | Interval containing it | Selected individual | Selected x | | --- | --- | --- | --- | | 0.2 | 0.1 ≤ 0.2 < 0.3 | 2 | 9 | | 0.4 | 0.3 ≤ 0.4 < 0.8 | 3 | 8 | | 0.6 | 0.3 ≤ 0.6 < 0.8 | 3 | 8 | | 0.95 | 0.8 ≤ 0.95 < 1 | 4 | 9 | Intervals are left-closed and right-open, except that a draw of exactly 1 selects the last interval. No draw here is on a boundary. Draws select parents with replacement: selecting a parent once does not remove it. Calculations use unrounded weights; the table displays rounded numbers. ##### Step 11. Generation 2: crossover after bit 2 Keep each prefix and exchange the suffixes. Bit positions are counted from the left starting at 1. | Pool slots paired | Parent A | Parent B | Child A | Child B | | --- | --- | --- | --- | --- | | 1 & 2 | 10 | 01 | 10 | 00 | 1000 | 1001 | | 3 & 4 | 10 | 00 | 10 | 01 | 1001 | 1000 | ##### Step 12. Generation 2: mutate only the specified bits | Child | Before mutation | Instruction | After mutation | | --- | --- | --- | --- | | 1 | 1000 | No mutation | 1000 | | 2 | 1001 | Flip bit 4 | 1000 | | 3 | 1001 | No mutation | 1001 | | 4 | 1000 | Flip bit 4 | 1001 | ##### Step 13. Generation 2: decode and evaluate every child | Chromosome | Decode using place values | Objective | | --- | --- | --- | | 1000 | 8 + 0 + 0 + 0 = 8 | (8−7)² = 1 | | 1000 | 8 + 0 + 0 + 0 = 8 | (8−7)² = 1 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | | 1001 | 8 + 0 + 0 + 1 = 9 | (9−7)² = 4 | ##### Step 14. Generation 2: compare and form the next population Add the objective scores in each population, then divide by its number of individuals. $$ \overline f_{old}=\frac{9+4+1+4}{4}=\frac{18}{4}=4.5 $$ $$ \overline f_{new}=\frac{1+1+4+4}{4}=\frac{10}{4}=2.5 $$ Best objective: 1 → 1. Lower is better. The next population is [8, 8, 9, 9]. No elitism is imposed, so the best previous individual is not automatically protected. **Final answer.** After two generations the ordered population is [8, 8, 9, 9]. Best final objective = 1 at x = [8]. --- ## TEE · GA approach to the travelling salesman problem Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u3-ga--ga-tsp/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u3-ga/ga-tsp Unit 3 · Evolutionary intelligence and genetic algorithms · 10-mark GA numerical in both 2025–26 papers ## Read the route before encoding it A **city** is a location. An **edge** connects two cities and has a travel distance. A **tour** visits each city exactly once and returns to its start. A **permutation** is an ordered arrangement without omissions or repetitions: [A,B,D,C] is valid for four cities; [A,B,B,C] is not. A distance matrix uses row=departure city and column=arrival city. Entry dAB=5 means the A→B leg costs 5. In a symmetric problem dBA=5 too. A diagonal 0 means no distance from a city to itself, not a free trip to another city. For A→B→D→C→A, add dAB+dBD+dDC+dCA. Factorial counts arrangements: 3!=3×2×1=6. With one of four cities fixed as the start, the other three can be ordered in six ways. In a symmetric TSP reverse-direction tours have equal length, although they are different written orders. **In one line:** encode a tour as a permutation of cities, use fitness = 1/(tour length), and use crossover and mutation operators that always produce valid tours. ## The problem **Travelling salesman problem (TSP):** given n cities and the distance between every pair, find the shortest route that visits each city exactly once and returns to the start. For a symmetric TSP with a fixed starting city, reversal-equivalent tours counted once, and n ≥ 3, the number of distinct tours is $$ (n-1)!/2 $$ (12 for 5 cities, about $$ 6\times10^{16} $$ for 20 cities), so exhaustive search quickly becomes impossible. ## How the GA is adapted | GA part | For TSP | | --- | --- | | Encoding | **Permutation**: C A E B D means C → A → E → B → D → back to C. | | Fitness | $$ F=1/L $$ , where $$ L=\sum d(c_i,c_{i+1})+d(c_n,c_1) $$ includes the return edge. Shorter tour = higher fitness. | | Selection | Roulette wheel on F, or tournament. | | Crossover | Order crossover (OX), partially mapped crossover (PMX) or cycle crossover (CX). | | Mutation | Swap two cities, or invert a segment (inversion), with small probability. | **Why ordinary crossover fails.** Single-point crossover of A B C | D E and C E A | B D gives A B C B D: city B twice and E missing, which is not a tour. Permutation operators avoid this. ## Order-preserving crossover: the stated left-to-right variant 1. Copy a randomly chosen segment from Parent 1 into the child, in the same positions. 2. Fill the remaining positions from left to right with the cities of Parent 2 in their order, skipping cities already copied. This variant fills empty positions from left to right; standard cyclic OX instead starts scanning and filling after the second cut. Do not mix the two conventions. Example: P1 = 2 4 [1 8 3] 5 7 6, P2 = 7 5 6 8 3 4 2 1. Copy 1 8 3; P2 order without them is 7 5 6 4 2 → child = **7 5 1 8 3 6 4 2**. (One GA slide prints 4 2 1 8 3 7 5 6, which uses a different fill order; state your rule in the exam.) **Question. Five cities with distances AB = 2, AC = 9, AD = 10, AE = 7, BC = 6, BD = 4, BE = 3, CD = 8, CE = 5, DE = 1. Parents are ABECD and ADBEC. (a) Find their lengths and fitness. (b) Apply the left-to-right order-preserving crossover variant described above with the segment at positions 2–3. (c) Apply swap mutation to positions 3 and 4 of the child. Comment.** [practice · 10 marks] #### Given All ten symmetric distances in the question; parents ABECD and ADBEC. Use the explicitly stated left-to-right order-preserving crossover at positions 2–3, then swap positions 3 and 4. #### To find Parent lengths/fitness, the crossover child, and the mutated child. #### Formula and why it applies Add all edges including the return to the start. A shorter positive length gives a larger reciprocal fitness. $$ L=\sum_{i=1}^{n-1}d(c_i,c_{i+1})+d(c_n,c_1),\quad F=1/L $$ #### Solution ##### Step 1. Parent lengths (include the return edge) $$ L(ABECD)=2+3+5+8+10=28,\quad F=1/28=0.0357 $$ $$ L(ADBEC)=10+4+3+5+9=31,\quad F=1/31=0.0323 $$ ##### Step 2. Order crossover Copy positions 2–3 from P1: \_ B E \_ \_. P2's order without B, E: A, D, C → fill positions 1, 4, 5. Swipe across the diagram to read its labels ↔ ![Order crossover keeps a valid tour: copy B E from P1, fill A, D, C in P2's order](https://deep-learning-study-room.vercel.app/is2/fig/ga_ox.8fd21cbc51.svg) Diagram: Order crossover keeps a valid tour: copy B E from P1, fill A, D, C in P2's order. Labels: Order crossover keeps a valid tour: copy B E from P1, fill A, D, C in P2's order | Parent 1 | A | B | E | C | D | length 28 | Parent 2 | A | D | B | E | C | length 31 | Child (OX) | A | B | E | D | C | length 23 | After swap | A | B | D | E | C | length 21 (shortest) $$ \text{Child}=ABEDC,\quad L=2+3+1+8+9=23,\quad F=0.0435 $$ ##### Step 3. Swap mutation (positions 3 and 4) $$ ABEDC\to ABDEC,\quad L=2+4+1+5+9=21,\quad F=0.0476 $$ **Final answer.** Child ABEDC (23) is shorter than both parents (28, 31) because it kept P1's short edge B–E and gained D–E (1). The mutated child ABDEC (21) is the shortest possible tour for this graph (checked against all 12 tours). ## Two more permutation crossovers (textbook, not in your slides) **PMX** (partially mapped crossover): swap a middle segment and use the position mapping to repair duplicates. With the slide parents and segment 3–5 the mapping is 1↔6, giving children 7 5 1 8 3 4 2 6 and 2 4 6 8 3 5 7 1. **Cycle crossover (CX)**: find cycles of positions between the parents and take alternate cycles from each parent; every city keeps a position it had in one of the parents. ## Algorithm (from your slides) 1. Initialise random tours (permutations). 2. Evaluate fitness = 1/L. 3. Select parents (roulette or tournament). 4. Apply OX to each pair. 5. Apply swap mutation with small probability. 6. Replace the population. 7. Repeat until the stopping condition; return the shortest tour. **Yaad rakho:** TSP mein chromosome = shehron ka order. Fitness = 1/length. Normal crossover se shehar repeat honge, isliye OX. --- ## TEE · How pheromone feedback guides ant-colony search Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u3-aco--how-ants-work/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u3-aco/how-ants-work Unit 3 · Ant colony optimization for TSP · asked as a 10-mark numerical in both 2025–26 papers **In one line:** ants leave pheromone as they walk; shorter paths get refreshed more often, so their scent stays stronger, so more ants choose them: positive feedback can favour shorter paths. ## Definition **Ant colony optimization (ACO)** is a population-based, probabilistic metaheuristic, introduced by **Marco Dorigo in 1992**, that finds good paths through a graph by imitating how ants communicate with **pheromone** trails. It is especially suited to routing problems such as the **travelling salesman problem (TSP)**. Swipe across the diagram to read its labels ↔ ![Why ants find the shorter path](https://deep-learning-study-room.vercel.app/is2/fig/aco_bridge.3fc4d22f6d.svg) Diagram: Why ants find the shorter path. Labels: Why ants find the shorter path | Nest | Food | short path: ants return sooner, pheromone builds up | long path: fewer trips per minute, pheromone evaporates ## How shorter paths can gain an advantage 1. **Random exploration.** At first, ants take all paths roughly equally and drop pheromone as they walk. 2. **Shorter path → faster round trip.** Ants on the short path return sooner, so that path receives pheromone more often per minute. 3. **Evaporation.** Pheromone evaporates. On the long path it has more time to fade between visits. 4. **Positive feedback.** Later ants prefer stronger scent, add even more pheromone, which can reinforce a useful route. A finite run is not guaranteed to discover the global shortest path. This indirect communication through changes in the environment is called **stigmergy**: no ant tells another where to go; the trail does. ## From real ants to artificial ants | Real ants | Artificial ants in ACO | | --- | --- | | Walk from nest to food | Build a solution by moving from node to node in a graph | | Drop pheromone on the path used | Increase the pheromone value $$ \tau_{ij} $$ on every edge used | | Pheromone evaporates | All $$ \tau_{ij} $$ shrink by the evaporation rate $$ \rho $$ each iteration | | Prefer stronger scent | Choose the next node with a probability based on pheromone and distance | | Colony reinforces frequently used paths | The algorithm searches for low-cost tours; it may stagnate on a suboptimal tour | ## TSP as a graph | TSP | ACO | | --- | --- | | City | Node | | Distance between two cities | Edge weight $$ d_{ij} $$ | | Round trip visiting every city once | One ant's path | | Tour length | Sum of the edges used, **including the edge back to the start** | **Yaad rakho:** Chhota raasta = jaldi wapas = zyada pheromone = aur cheentiyan. Lamba raasta = pheromone ud jaata hai. --- ## TEE · ACO formulas: transition probability, city choice and pheromone update Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u3-aco--formulas/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u3-aco/formulas Unit 3 · Ant colony optimization for TSP · asked as a 10-mark numerical in both 2025–26 papers **From city A an ant may choose B or C. Distances are AB=2 and AC=4. Pheromones are both 1 and α=β=1. Find the next-city probabilities.** [2 marks · Worked example] #### Given Current city A; allowed next cities B and C. Distances dAB=2 and dAC=4. Pheromones τAB=τAC=1, and exponents α=β=1. Visited cities are excluded. #### To find P(A→B) and P(A→C). #### Formula and why it applies $$ \eta_{ij}=1/d_{ij}\qquad P_{ij}=\tau_{ij}\eta_{ij}/\sum_{l\in allowed}\tau_{il}\eta_{il} $$ Shorter distance gives a larger reciprocal. Normalise the desirability weights to probabilities. #### Solution ##### Step 1. Take reciprocal distances $$ \eta_{AB}=1/2=0.5\qquad\eta_{AC}=1/4=0.25 $$ ##### Step 2. Multiply by pheromone $$ w_B=1\times0.5=0.5\qquad w_C=1\times0.25=0.25 $$ ##### Step 3. Add allowed weights $$ 0.5+0.25=0.75 $$ ##### Step 4. Divide each weight by their total $$ P_{AB}=0.5/0.75=2/3 $$ $$ P_{AC}=0.25/0.75=1/3 $$ **Final answer.** B has probability 2/3; C has probability 1/3. These are probabilities, not a rule forcing the nearest city every time. Evaporation ρ=.5 means retain 1−.5=.5 of the old pheromone. A completed tour of length 20 deposits Q/20 on each edge it used; with Q=1 this is .05. An unused edge receives no deposit from that ant. Use complete tour length, not the length of the individual edge, in this denominator. **In one line:** an ant picks the next city with probability ∝ pheromone × (1/distance); after all ants finish, pheromone evaporates and each ant adds Q/(its tour length) to every edge it used. ## 1. Transition probability An ant $$ k $$ at city $$ i $$ moves to an unvisited city $$ j $$ with probability $$ P^{k}_{ij}=\frac{\tau_{ij}^{\,\alpha}\,\eta_{ij}^{\,\beta}}{\displaystyle\sum_{l\in\text{allowed}}\tau_{il}^{\,\alpha}\,\eta_{il}^{\,\beta}},\qquad \eta_{ij}=\frac1{d_{ij}} $$ - $$ \tau_{ij} $$ : pheromone on edge $$ i $$ – $$ j $$ ; $$ \eta_{ij} $$ : visibility (heuristic), the inverse of distance. - $$ \alpha $$ : weight of pheromone; $$ \beta $$ : weight of distance. In every class example $$ \alpha=\beta=1 $$ . - “allowed” = cities this ant has not visited yet. With all $$ \tau=1 $$ and $$ \alpha=\beta=1 $$ , the probabilities are simply proportional to $$ 1/d $$ . ## 2. Choosing the city with a random number (cumulative rule) 1. List the allowed cities **in alphabetical (city) order**, not sorted by probability. 2. Write their cumulative probabilities. 3. Pick the **first** city whose cumulative value is ≥ the random number $$ r $$ . 4. When only one city is left, the ant goes there without using a random number, then returns to the start. **Exam trap.** In Final 2025 Q7b, from city C the candidates are B (0.4) and D (0.6). In city order the ranges are B: 0–0.4, D: 0.4–1, so r = 0.8 selects D. If you sort by probability (D first) you would pick B and get the wrong tour. The official key uses city order. ## 3. Pheromone update (after all ants finish) $$ \tau_{ij}\leftarrow(1-\rho)\,\tau_{ij}+\sum_{k=1}^{m}\Delta\tau^{k}_{ij},\qquad \Delta\tau^{k}_{ij}=\begin{cases}Q/L_k&\text{if ant }k\text{ used edge }(i,j)\\0&\text{otherwise}\end{cases} $$ - $$ \rho $$ : evaporation rate (fraction lost). Class examples use $$ \rho=0.5 $$ . $$ Q $$ : deposit constant, usually 1. - $$ L_k $$ : length of ant $$ k $$ 's complete tour **including the return edge**. Shorter tours deposit more. - Edges are undirected: update $$ \tau_{ij} $$ and $$ \tau_{ji} $$ together (the matrix stays symmetric). - The update happens once per iteration, after every ant has finished; within an iteration all ants use the same pheromone values. **When the question gives no ρ or Q** (both 2025–26 papers): write “Assume ρ = 0.5 and Q = 1, as in the class example” and proceed. That is exactly what the Final 2025 key does. ## 4. Algorithm (Ant System) 1. Initialise parameters: number of ants, α, β, ρ, Q, initial pheromone $$ \tau_0 $$ , maximum iterations. 2. Place each ant on its start city. 3. Each ant repeatedly chooses the next city with the transition probability until it has visited all cities, then returns to the start. 4. Compute each tour length $$ L_k $$ and record the best tour. 5. Update pheromone: evaporation, then deposit $$ Q/L_k $$ . 6. Repeat steps 2–5 until the iteration limit; output the best tour. | Parameter | Meaning | Typical value | | --- | --- | --- | | m | Number of ants | number of cities (or 20) | | α | Pheromone importance | 1 | | β | Distance (visibility) importance | 1 (often 2–5 in practice) | | ρ | Evaporation rate | 0.5 in class numericals (0.05–0.5 in practice) | | Q | Deposit constant | 1 (100 in one slide example) | | $$ \tau_0 $$ | Initial pheromone | 1 in exam questions | **Yaad rakho:** Probability = τ × (1/d), phir normalize. City order mein cumulative, pehla ≥ r. Update = ½ purana + Q/L har us edge par jo use hua (wapas wala edge bhi). --- ## TEE · Solved PYQ: ACO for a 4-city TSP (Final 2025, 10 marks) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u3-aco--final-2025/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u3-aco/final-2025 Unit 3 · Ant colony optimization for TSP · asked as a 10-mark numerical in both 2025–26 papers **Question. For the four-city travelling salesman problem, the distance and pheromone matrices (A, B, C, D) are given. Apply Ant Colony Optimization. Calculate the path travelled by 2 ants to cover all cities starting from city A. Random numbers for Ant 1 are 0.3, 0.8 and for Ant 2 are 0.5, 0.7. Also update the pheromone matrix after 1 iteration.** [10 marks · Final Nov 2025 Q7b] #### Given $$ D=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&4&2&4\\B&4&0&3&6\\C&2&3&0&2\\D&4&6&2&0\end{array}\qquad \tau=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&1&1&1\\B&1&0&1&1\\C&1&1&0&1\\D&1&1&1&0\end{array} $$ Assume $$ \alpha=\beta=1 $$ , $$ \rho=0.5 $$ , $$ Q=1 $$ (class convention; not stated in the question). #### To find Tour of each ant, tour lengths, updated pheromone matrix. Swipe across the diagram to read its labels ↔ ![Final 2025 Q7b: distances (all pheromones start at 1)](https://deep-learning-study-room.vercel.app/is2/fig/aco_final_graph.ac3cd4aeac.svg) Diagram: Final 2025 Q7b: distances (all pheromones start at 1). Labels: Final 2025 Q7b: distances (all pheromones start at 1) | 4 | 2 | 4 | 3 | 6 | 2 | A | B | C | D #### Formula and why it applies $$ P_{ij}=\frac{\tau_{ij}\,\eta_{ij}}{\sum_{l\in\text{allowed}}\tau_{il}\,\eta_{il}},\qquad\eta_{ij}=\frac1{d_{ij}} $$ $$ \tau_{ij}^{new}=(1-\rho)\tau_{ij}+\sum_{k:\,ij\in\mathrm{tour}_k}\frac{Q}{L_k} $$ The sum includes **only ants whose completed tour contains edge ij**. An unused edge receives no deposit but still evaporates. Here distances are symmetric, so a traversed edge updates both symmetric matrix cells. Both ants construct their tours using the old pheromones; update after both finish, including their return to A. #### Solution ##### Step 1. Probabilities from A (both ants start at A) $$ \tau\eta:\;A\to B=1\times\tfrac14=0.25,\quad A\to C=1\times\tfrac12=0.5,\quad A\to D=1\times\tfrac14=0.25;\quad\text{sum}=1 $$ | Move | P | Cumulative | Range | | --- | --- | --- | --- | | A→B | 0.25 | 0.25 | 0–0.25 | | A→C | 0.50 | 0.75 | 0.25–0.75 | | A→D | 0.25 | 1.00 | 0.75–1 | Ant 1: r = 0.3 → **C**. Ant 2: r = 0.5 → **C**. ##### Step 2. Probabilities from C (B and D left) $$ C\to B=\tfrac13=0.3333,\quad C\to D=\tfrac12=0.5;\quad\text{sum}=0.8333\Rightarrow P_{CB}=0.4,\;P_{CD}=0.6 $$ Cumulative in city order: B 0–0.4, D 0.4–1. Ant 1: r = 0.8 → **D**. Ant 2: r = 0.7 → **D**. ##### Step 3. Complete the tours Only B is left, then return to A. $$ \text{Ant 1}=\text{Ant 2}:\;A\to C\to D\to B\to A,\qquad L=2+2+6+4=14 $$ Swipe across the diagram to read its labels ↔ ![Both ants: A → C → D → B → A, length 14](https://deep-learning-study-room.vercel.app/is2/fig/aco_final_tour.1dbfdb8b98.svg) Diagram: Both ants: A → C → D → B → A, length 14. Labels: Both ants: A → C → D → B → A, length 14 | 4 | 2 | 4 | 3 | 6 | 2 | A | B | C | D ##### Step 4. Pheromone update Each ant deposits $$ Q/L=1/14=0.0714 $$ on each edge of its tour; two ants used the same four edges (AC, CD, DB, BA), so each gets $$ 2/14=0.1429 $$ . $$ \text{used edge: }(1-0.5)(1)+0.1429=0.6429\qquad\text{unused edge (AD, BC): }(1-0.5)(1)+0=0.5 $$ $$ \tau_{\text{new}}=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&0.6429&0.6429&0.5\\B&0.6429&0&0.5&0.6429\\C&0.6429&0.5&0&0.6429\\D&0.5&0.6429&0.6429&0\end{array} $$ **Final answer.** Both ants: A → C → D → B → A, length 14. Updated pheromone: 0.6429 on AB, AC, CD, BD; 0.5 on AD and BC. In the next iteration the ants will prefer the edges of this tour. **Official key note.** The key's matrix shows 0.643 on AC, CD and BD but leaves AB at 0.5, i.e. it forgot to deposit on the return edge B → A even though it counted its length (4) in L = 14. Your slides deposit on the return edge, so AB = 0.6429 is correct. If you wish, write one line: “the return edge B–A is part of the tour, so it also receives Q/L.” Official marking: path of Ant 1 (3) + path of Ant 2 (3) + pheromone matrix (4). --- ## TEE · Solved PYQ: ACO for a 4-city TSP (Re-exam 2026, 10 marks) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u3-aco--reexam-2026/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u3-aco/reexam-2026 Unit 3 · Ant colony optimization for TSP · asked as a 10-mark numerical in both 2025–26 papers **Question. For the four-city travelling salesman problem the distance matrix and pheromone matrix are given (all pheromones 1). Apply Ant Colony Optimization. Calculate the path travelled by 2 ants starting from city A; random numbers for Ant 1 are 0.3, 0.8 and for Ant 2 are 0.5, 0.7. Also update the pheromone matrix after 1 iteration.** [10 marks · Re-exam Jan 2026 Q6a] #### Given $$ D=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&1&2&3\\B&1&0&4&5\\C&2&4&0&6\\D&3&5&6&0\end{array},\qquad\tau_{ij}=1\;(i\ne j) $$ Assume $$ \alpha=\beta=1,\;\rho=0.5,\;Q=1 $$ . #### To find Both tours, their lengths and the pheromone matrix after one iteration. #### Formula and why it applies $$ P_{ij}=\frac{\tau_{ij}^{\alpha}(1/d_{ij})^\beta}{\sum_{k\in allowed}\tau_{ik}^{\alpha}(1/d_{ik})^\beta} $$ $$ \tau_{ij}^{new}=(1-\rho)\tau_{ij}+\sum_{k:\,ij\in tour_k}\frac{Q}{L_k} $$ Use unvisited cities in alphabetical order for cumulative selection. Every ant completes its tour using old pheromones; then evaporate and deposit on each undirected edge, including the return edge. #### Solution ##### Step 1. From A $$ \eta:\;B=1,\;C=0.5,\;D=0.3333;\;\text{sum}=1.8333\Rightarrow P=0.5455,\;0.2727,\;0.1818 $$ Cumulative: B 0–0.5455, C 0.5455–0.8182, D 0.8182–1. Ant 1 (0.3) → **B**; Ant 2 (0.5) → **B**. ##### Step 2. From B (C and D left) $$ \eta:\;C=0.25,\;D=0.2;\;\text{sum}=0.45\Rightarrow P_{BC}=0.5556,\;P_{BD}=0.4444 $$ Cumulative: C 0–0.5556, D 0.5556–1. Ant 1 (0.8) → **D**; Ant 2 (0.7) → **D**. ##### Step 3. Complete the tours $$ A\to B\to D\to C\to A,\qquad L=1+5+6+2=14\;\text{(both ants)} $$ Swipe across the diagram to read its labels ↔ ![Re-exam 2026 Q6a: both ants A → B → D → C → A, length 14](https://deep-learning-study-room.vercel.app/is2/fig/aco_reexam_graph.02195c6ee9.svg) Diagram: Re-exam 2026 Q6a: both ants A → B → D → C → A, length 14. Labels: Re-exam 2026 Q6a: both ants A → B → D → C → A, length 14 | 1 | 2 | 3 | 4 | 5 | 6 | A | B | C | D ##### Step 4. Pheromone update Used edges AB, BD, DC, CA get $$ 0.5(1)+2\times\frac1{14}=0.6429 $$ ; unused edges AD and BC become $$ 0.5 $$ . $$ \tau_{\text{new}}=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&0.6429&0.6429&0.5\\B&0.6429&0&0.5&0.6429\\C&0.6429&0.5&0&0.6429\\D&0.5&0.6429&0.6429&0\end{array} $$ **Final answer.** Both ants: A → B → D → C → A, length 14. Pheromone 0.6429 on AB, BD, CD, AC; 0.5 on AD, BC. **Official key note.** This key applies *no evaporation* ( $$ 1+0.1428=1.1428 $$ ) and its matrix is inconsistent: it increases the unused edge A–D, leaves the used return edge C–A at 1, and gives B–D and D–B different values. If your examiner follows the no-evaporation idea, the consistent version is 1.1429 on AB, BD, CD, AC and 1 on AD, BC. The paper does not specify ρ. State the convention you use; the resulting matrix depends on it. Grading depends on the examiner’s intended convention. --- ## TEE · Practice: the lecture's Ant System example (two different tours) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u3-aco--lecture-example/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u3-aco/lecture-example Unit 3 · Ant colony optimization for TSP · asked as a 10-mark numerical in both 2025–26 papers **Why do this:** in the papers both ants happened to take the same tour. In the class example they take different tours, which shows how the deposit adds up when two lengths differ. **Question. Distances: AB = 5, AC = 6, AD = 7, BC = 2, BD = 9, CD = 3; all pheromones 1; $$ \alpha=\beta=1,\;\rho=0.5,\;Q=1 $$ . Two ants start at A. Ant 1 uses random numbers 0.45 then 0.5; Ant 2 uses 0.83 then 0.5. Find both tours and the updated pheromone matrix.** [lecture example · 10 marks] #### Given Symmetric distances AB=5, AC=6, AD=7, BC=2, BD=9, CD=3. Initial off-diagonal pheromone=1. α=β=1, ρ=0.5, Q=1. Ant draws: [0.45,0.5] and [0.83,0.5]. #### To find Two complete tours and every updated edge pheromone. #### Formula and why it applies Prefer high pheromone and small distance, normalise over unvisited cities, then evaporate and deposit once per iteration. $$ P_{ij}=\frac{\tau_{ij}^{\alpha}(1/d_{ij})^{\beta}}{\sum_{k\in allowed}\tau_{ik}^{\alpha}(1/d_{ik})^{\beta}} $$ $$ \tau_{ij}^{new}=(1-\rho)\tau_{ij}^{old}+\sum_{ants\ using\ ij}Q/L $$ #### Solution Swipe across the diagram to read its labels ↔ ![Lecture example (Ant System): distances](https://deep-learning-study-room.vercel.app/is2/fig/aco_lecture_graph.629672bf56.svg) Diagram: Lecture example (Ant System): distances. Labels: Lecture example (Ant System): distances | 5 | 6 | 7 | 2 | 9 | 3 | A | B | C | D #### Solution ##### Step 1. From A $$ \eta=\tfrac15,\tfrac16,\tfrac17=0.2,\;0.1667,\;0.1429;\;\text{sum}=0.5095\Rightarrow P=0.3925,\;0.3271,\;0.2804 $$ Cumulative: B 0–0.3925, C 0.3925–0.7196, D 0.7196–1. Ant 1 (0.45) → C. Ant 2 (0.83) → D. ##### Step 2. Ant 1 from C (B, D left) $$ \eta=\tfrac12,\tfrac13\Rightarrow P_{CB}=0.6,\;P_{CD}=0.4 $$ r = 0.5 ≤ 0.6 → B; then D; then back to A. $$ \text{Ant 1: }A\to C\to B\to D\to A,\quad L_1=6+2+9+7=24 $$ ##### Step 3. Ant 2 from D (B, C left) $$ \eta=\tfrac19,\tfrac13\Rightarrow P_{DB}=0.25,\;P_{DC}=0.75 $$ Cumulative B 0–0.25, C 0.25–1; r = 0.5 → C; then B; then A. $$ \text{Ant 2: }A\to D\to C\to B\to A,\quad L_2=7+3+2+5=17 $$ ##### Step 4. Deposits | Edge | Used by | New τ | | --- | --- | --- | | AD, BC | both ants | 0.5 + 1/24 + 1/17 = 0.6005 | | AC, BD | Ant 1 only | 0.5 + 1/24 = 0.5417 | | AB, CD | Ant 2 only | 0.5 + 1/17 = 0.5588 | **Final answer.** Ant 1: A-C-B-D-A (24); Ant 2: A-D-C-B-A (17). Ant 2's shorter tour deposits more (1/17 > 1/24), so its edges grow faster. AD and BC, used by both, end highest at 0.6005. **Slide corrections.** The slide gives P(D→B) = 0.33 and P(D→C) = 0.66; the correct values are 0.25 and 0.75 (the choice of C does not change). The slide truncates 0.6005 and 0.5417 to 0.6004 and 0.5416. ## Write the requested final matrix Ant 1 deposits 1/24≈.041666667; ant 2 deposits 1/17≈.058823529. Evaporation leaves .5 on every edge. Their shared edges AD and BC become .600490196. Ant-1-only edges AC and BD become .541666667; ant-2-only edges AB and CD become .558823529. $$ \tau^{new}\approx\begin{bmatrix}0&.558823529&.541666667&.600490196\\.558823529&0&.600490196&.541666667\\.541666667&.600490196&0&.558823529\\.600490196&.541666667&.558823529&0\end{bmatrix} $$ Row and column order is A,B,C,D. The diagonal 0 marks excluded self-edges. Off-diagonal entries mirror each other because this problem is undirected and symmetric. --- ## TEE · 5/10-mark answer: applications of ACO (explain one) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u3-aco--applications/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u3-aco/applications Unit 3 · Ant colony optimization for TSP · asked as a 10-mark numerical in both 2025–26 papers **Asked as:** “List the applications of Ant Colony Optimization and explain any one application in detail.” [5 marks · Final 2025 Q1b; key: listing 3 + explanation 2] ### Definition (1 line) Ant Colony Optimization is a probabilistic, population-based metaheuristic (Dorigo, 1992) in which artificial ants build solutions on a graph and communicate through pheromone trails. In the update rule used here, evaporation reduces all stored trails; deposits then reinforce edges of selected tours according to solution quality. Evaporation does not occur only on poor paths. ### Applications 1. **Travelling salesman problem**: shortest tour visiting every city once. 2. **Vehicle routing**: delivery routes for a fleet with capacity and time windows (minimising distance, time, fuel). 3. **Network routing**: adaptive routing of data packets in communication and sensor networks (AntNet). 4. **Job-shop and project scheduling**: assigning jobs to machines to minimise completion time. 5. **Image processing**: edge detection, with ants moving over pixel-intensity graphs. 6. **Assignment problems**: quadratic assignment, timetabling, graph colouring. 7. **Feature selection and data mining**: Ant-Miner for classification rules. 8. **Load balancing** in distributed systems and cloud task allocation. ### One application in detail: vehicle routing (or TSP) 1. **Graph.** Depot and customers are nodes; travel times or distances are edge weights. 2. **Ants.** Each ant builds a complete set of routes. From its current customer it chooses the next with probability $$ \propto\tau_{ij}\,(1/d_{ij}) $$ , skipping customers already served or that would exceed the vehicle capacity; when the vehicle is full it returns to the depot. 3. **Evaluation.** Total distance (plus penalties for late deliveries). 4. **Learning.** Pheromone evaporates everywhere and is added in proportion to $$ Q/L $$ on the routes of good solutions, so short connections are reinforced. 5. **Result.** Repeated iterations search for low-cost routes; finite-run convergence to the optimum is not guaranteed. Companies use this for parcel delivery, waste collection and school-bus routing. ### Advantages and limitations | Advantages | Limitations | | --- | --- | | Naturally suited to graph and routing problems | Several parameters to tune (α, β, ρ, number of ants) | | Can share useful route information and adapt search to changing graphs | Convergence time is uncertain; may stagnate on a sub-optimal path | | Positive feedback + parallel ants | Theoretical analysis is difficult | **Yaad rakho:** ACO = “raaste wale problems” (TSP, routing, scheduling). PSO = “numbers wale problems” (continuous function, weights). --- ## TEE · Swarm intelligence, particles, velocity, pbest and gbest Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/pso-core--meaning/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/pso-core/meaning First understand position, velocity and memory; then solve complete two-iteration minimisation examples. ## Swarm intelligence **Swarm intelligence** is collective problem-solving behaviour that emerges from interactions among many relatively simple agents and their environment. It is inspired by systems such as bird flocks, fish schools and ant colonies. **Particle swarm optimization (PSO)** is a population-based search method inspired by social movement. Each particle represents a candidate solution. It moves through the search space using a velocity influenced by its own best previous position and the best position known to the swarm. ## Translate every PSO word into a concrete object | Term | Meaning | Example | | --- | --- | --- | | Particle | One candidate solution | Particle A | | Position x | The candidate’s coordinates | xA = 3 | | Objective f(x) | The problem’s score for that candidate | f(3) = 3² = 9 | | Velocity v | Proposed change of position | vA = −1 means move left by 1 | | pbest | Best position that this particle has visited | A once visited x=1 with score 1 | | gbest | Best position found by any particle | Among all personal bests, x=0.5 scores 0.25 | Swipe across the diagram to read its labels ↔ ![A best memory changes only when the new score improves.](https://deep-learning-study-room.vercel.app/is2/fig/v2_pso_memory.fcc9783a95.svg) Diagram: A best memory changes only when the new score improves.. Labels: Best memory survives a worse move | Visit 1 | Position x = 3 | Score x² = 9 | Remember pbest = 3 | Visit 2 | Position x = 1 | Score x² = 1 | Remember pbest = 1 | Visit 3 | Position x = 2 | Score x² = 4 | Remember pbest = 1 | Minimise x²: after visiting x = 2, the best past position is still x = 1. | Current position and remembered best position are separate records. A best memory changes only when the new score improves. ## Current position is not necessarily personal best For minimising x², suppose a particle visits x=3, then x=1, then x=2. Its current position is 2, but its personal best stays 1 because 1² is less than 2². The particle remembers a better past result even after a bad move. **A particle visits x=3, x=1, x=2 in that order while minimising f(x)=x². Find pbest after each visit.** [2 marks · Practice] #### Given Positions in chronological order: 3 → 1 → 2. #### To find The best-so-far position and score. #### Formula and why it applies Keep the location with the lowest objective encountered so far. #### Solution ##### Step 1. First visit $$ f(3)=9 $$ No history: pbest=3, best score=9. ##### Step 2. Second visit $$ f(1)=1<9 $$ Replace pbest by 1. ##### Step 3. Third visit $$ f(2)=4>1 $$ Keep pbest=1 even though the current position is 2. **Final answer.** Final current position=2; pbest=1; personal-best score=1. **Define swarm intelligence and briefly explain one nature-inspired algorithm.** [2 marks · Practice] #### Answer Swarm intelligence is collective problem-solving that emerges from interactions among simple agents. Particle swarm optimization is inspired by flocking behaviour. Each particle is a candidate solution with a position and velocity. It remembers its own best position and uses the swarm’s best position to guide movement. Repeated evaluation and movement search for a low-cost or high-fitness solution. **Yaad rakho:** Position abhi kahan ho; pbest apna best past; gbest sabka best past. --- ## TEE · Read the PSO equation and work three individual updates Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/pso-core--equation/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/pso-core/equation First understand position, velocity and memory; then solve complete two-iteration minimisation examples. ## First read the update in words **New velocity = retained old velocity + pull toward personal best + pull toward the group best.** After finding that velocity, add it to the old position. You are calculating a movement, then applying it. Swipe across the diagram to read its labels ↔ ![A negative difference pulls toward a best position on the left.](https://deep-learning-study-room.vercel.app/is2/fig/depth_pso.88e9c2a672.svg) Diagram: A negative difference pulls toward a best position on the left.. Labels: Read the direction before multiplying the pull | Best = 1 | Current = 4 | difference = 1 − 4 = −3 | coefficient 0.5 → pull 0.5 × (−3) = −1.5 (left) A negative difference pulls toward a best position on the left. Example: current x=4 and personal best=1 gives the difference 1−4=−3. A coefficient of .5 gives personal pull .5×(−3)=−1.5. The negative sign says “move left”; it does not mean a negative probability. If current x=−2 and best=1, the difference is 1−(−2)=1+2=3, which pulls right. In vi,jt, i chooses the particle, j chooses its coordinate, and t chooses the recorded iteration. For particle B, coordinate 2, iteration 0, look at row B and the second velocity entry in the initial table. The letters identify a stored number; they are not extra multiplication factors. ## The velocity equation is three contributions added together $$ v_{i,j}^{t+1}=w v_{i,j}^{t}+c_1r_1(pbest_{i,j}^{t}-x_{i,j}^{t})+c_2r_2(gbest_j^{t}-x_{i,j}^{t}) $$ $$ x_{i,j}^{t+1}=x_{i,j}^{t}+v_{i,j}^{t+1} $$ | Symbol | Meaning | | --- | --- | | i | Which particle | | j | Which coordinate of that particle | | t / t+1 | Before / after this iteration; these are labels, not powers | | w | Inertia weight: how much old velocity is kept | | c₁r₁ | Combined strength of the pull to personal best | | c₂r₂ | Combined strength of the pull to global best | | r₁,r₂ | Random factors, normally in [0,1]; use the values supplied in an exam | A best position to the left gives a negative difference and pulls left; a best position to the right gives a positive difference and pulls right. The multiplication sets the strength of each pull. **For one coordinate, x=4, v=0, pbest=4, gbest=2. With w=0.5, c₁=c₂=1 and r₁=r₂=0.5, find the next velocity and position.** [3 marks · Practice] #### Given x=4, v=0, pbest=4, gbest=2; w=.5; c₁r₁=.5; c₂r₂=.5. #### To find New velocity followed by new position. #### Formula and why it applies Use the three-term velocity equation, then add the new velocity to x. $$ v_{new}=0.5v+0.5(pbest-x)+0.5(gbest-x) $$ #### Solution ##### Step 1. Inertia $$ 0.5\times0=0 $$ ##### Step 2. Personal pull $$ 0.5(4-(4))=0 $$ ##### Step 3. Social pull $$ 0.5(2-(4))=-1 $$ ##### Step 4. New velocity $$ v_{new}=0+0+(-1)=-1 $$ ##### Step 5. New position $$ x_{new}=4+(-1)=3 $$ **Final answer.** vnew=-1; xnew=3. **For one coordinate, x=4, v=-2, pbest=3, gbest=1. With w=0.5, c₁=c₂=1 and r₁=r₂=0.5, find the next velocity and position.** [3 marks · Practice] #### Given x=4, v=-2, pbest=3, gbest=1; w=.5; c₁r₁=.5; c₂r₂=.5. #### To find New velocity followed by new position. #### Formula and why it applies Use the three-term velocity equation, then add the new velocity to x. $$ v_{new}=0.5v+0.5(pbest-x)+0.5(gbest-x) $$ #### Solution ##### Step 1. Inertia $$ 0.5\times(-2)=-1 $$ ##### Step 2. Personal pull $$ 0.5(3-(4))=-0.5 $$ ##### Step 3. Social pull $$ 0.5(1-(4))=-1.5 $$ ##### Step 4. New velocity $$ v_{new}=(-1)+(-0.5)+(-1.5)=-3 $$ ##### Step 5. New position $$ x_{new}=4+(-3)=1 $$ **Final answer.** vnew=-3; xnew=1. **For one coordinate, x=-2, v=1, pbest=-1, gbest=1. With w=0.5, c₁=c₂=1 and r₁=r₂=0.5, find the next velocity and position.** [3 marks · Practice] #### Given x=-2, v=1, pbest=-1, gbest=1; w=.5; c₁r₁=.5; c₂r₂=.5. #### To find New velocity followed by new position. #### Formula and why it applies Use the three-term velocity equation, then add the new velocity to x. $$ v_{new}=0.5v+0.5(pbest-x)+0.5(gbest-x) $$ #### Solution ##### Step 1. Inertia $$ 0.5\times1=0.5 $$ ##### Step 2. Personal pull $$ 0.5(-1-(-2))=0.5 $$ ##### Step 3. Social pull $$ 0.5(1-(-2))=1.5 $$ ##### Step 4. New velocity $$ v_{new}=0.5+0.5+1.5=2.5 $$ ##### Step 5. New position $$ x_{new}=(-2)+2.5=0.5 $$ **Final answer.** vnew=2.5; xnew=0.5. **First-iteration shortcut:** if pbest was just initialised to x, the personal difference is zero. This follows from the initialisation; it is not a permanent property of PSO. A gbest particle can still move because its inertia term can be nonzero. **Yaad rakho:** Old speed ka hissa + apne best ki pull + group best ki pull; phir new speed ko position mein add. --- ## TEE · Two particles, two iterations: start with the simplest minimisation Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/pso-core--two-particles/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/pso-core/two-particles First understand position, velocity and memory; then solve complete two-iteration minimisation examples. A synchronous iteration is a class photograph of the old swarm: every particle uses that same snapshot of gbest. First calculate all new positions, then compare all new scores and update memories. If particle A discovers a better position halfway through the calculations, B still uses the previous snapshot during this iteration. B can use the improved gbest in the next iteration. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. **Minimise $$ f(x)=x^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (-2) | (0) | | B | (4) | (0) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(-2)=(-2)^2=4=4 $$ **Particle B** $$ f(4)=4^2=16=16 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(-2) $$ Its score is 4. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (-2). pbest values are A: (-2), B: (4). ##### Step 4. Iteration 1: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=-2-(-2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-2-(-2)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=(-2)+0=-2 $$ ##### Step 5. Iteration 1: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=4-(4)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=-2-(4)=-6 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-6)=-3 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-3)=-3 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=4+(-3)=1 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(-2)=(-2)^2=4=4 $$ **Particle B** $$ f(1)=1^2=1=1 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 4 | 4 | Keep old: tie | (-2) | | B | 16 | 1 | Replace: new score is lower | (1) | Compare the retained pbest scores. gbest = **(1)**, with objective **1**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (-2) | (0) | (-2) | | B | (1) | (-3) | (1) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (1). pbest values are A: (-2), B: (1). ##### Step 10. Iteration 2: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=-2-(-2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(-2)=3 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(3)=1.5 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+1.5=1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=(-2)+1.5=-0.5 $$ ##### Step 11. Iteration 2: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-3)=-1.5 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=(-1.5)+0+0=-1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1+(-1.5)=-0.5 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(-0.5)=(-0.5)^2=0.25=0.25 $$ **Particle B** $$ f(-0.5)=(-0.5)^2=0.25=0.25 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 4 | 0.25 | Replace: new score is lower | (-0.5) | | B | 1 | 0.25 | Replace: new score is lower | (-0.5) | Compare the retained pbest scores. gbest = **(-0.5)**, with objective **0.25**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (-0.5) | (1.5) | (-0.5) | | B | (-0.5) | (-1.5) | (-0.5) | **Final answer.** After iteration 2, gbest = (-0.5), objective = 0.25. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. --- ## TEE · A worse move: pbest stays in the past Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/pso-core--memory-matters/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/pso-core/memory-matters First understand position, velocity and memory; then solve complete two-iteration minimisation examples. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. **Minimise $$ f(x)=x^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (1) | (4) | | B | (3) | (-1) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(1)=1^2=1=1 $$ **Particle B** $$ f(3)=3^2=9=9 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(1) $$ Its score is 1. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (1). pbest values are A: (1), B: (3). ##### Step 4. Iteration 1: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times4=2 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=2+0+0=2 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=1+2=3 $$ ##### Step 5. Iteration 1: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-1)=-0.5 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3-(3)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(3)=-2 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-2)=-1 $$ Add the three velocity contributions. $$ v^{(1)}=(-0.5)+0+(-1)=-1.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=3+(-1.5)=1.5 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(3)=3^2=9=9 $$ **Particle B** $$ f(1.5)=1.5^2=2.25=2.25 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 1 | 9 | Keep old: new score is worse | (1) | | B | 9 | 2.25 | Replace: new score is lower | (1.5) | Compare the retained pbest scores. gbest = **(1)**, with objective **1**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (3) | (2) | (1) | | B | (1.5) | (-1.5) | (1.5) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (1). pbest values are A: (1), B: (1.5). ##### Step 10. Iteration 2: update particle A **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times2=1 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(3)=-2 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(-2)=-1 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(3)=-2 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-2)=-1 $$ Add the three velocity contributions. $$ v^{(2)}=1+(-1)+(-1)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=3+(-1)=2 $$ ##### Step 11. Iteration 2: update particle B **Only coordinate** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-1.5)=-0.75 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1.5-(1.5)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1.5)=-0.5 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-0.5)=-0.25 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.75)+0+(-0.25)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1.5+(-1)=0.5 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(2)=2^2=4=4 $$ **Particle B** $$ f(0.5)=0.5^2=0.25=0.25 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 1 | 4 | Keep old: new score is worse | (1) | | B | 2.25 | 0.25 | Replace: new score is lower | (0.5) | Compare the retained pbest scores. gbest = **(0.5)**, with objective **0.25**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (2) | (-1) | (1) | | B | (0.5) | (-1) | (0.5) | **Final answer.** After iteration 2, gbest = (0.5), objective = 0.25. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. **Yaad rakho:** Naya result kharab ho toh position toh badlegi, pbest nahi. --- ## TEE · Two-dimensional PSO: keep the coordinates separate Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/pso-core--two-coordinates/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/pso-core/two-coordinates First understand position, velocity and memory; then solve complete two-iteration minimisation examples. A two-dimensional particle has two entries in its position and two in its velocity. Apply the scalar velocity calculation once to coordinate 1 and once to coordinate 2. Keep them in the same order to form the new pair. Only after both entries are ready do you insert the pair into f(x₁,x₂). Adding x₁ and x₂ into a single position would lose information. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. **Minimise $$ f(x)=x_1^2+x_2^2 $$ using 2 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [3 marks · Practice; expanded teaching solution] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (1, 1) | (0, 0) | | B | (3, 2) | (0, 0) | w=0.5, c₁=1, c₂=1, r₁=0.5, r₂=0.5. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(1, 1)=1^2+1^2=1+1=2 $$ **Particle B** $$ f(3, 2)=3^2+2^2=9+4=13 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(1, 1) $$ Its score is 2. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (1, 1). pbest values are A: (1, 1), B: (3, 2). ##### Step 4. Iteration 1: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=1+0=1 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=1+0=1 $$ ##### Step 5. Iteration 1: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3-(3)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(3)=-2 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-2)=-1 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-1)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=3+(-1)=2 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=2-(2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(2)=-1 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-1)=-0.5 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-0.5)=-0.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=2+(-0.5)=1.5 $$ ##### Step 6. Iteration 1: evaluate the new positions **Particle A** $$ f(1, 1)=1^2+1^2=1+1=2 $$ **Particle B** $$ f(2, 1.5)=2^2+1.5^2=4+2.25=6.25 $$ ##### Step 7. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 2 | 2 | Keep old: tie | (1, 1) | | B | 13 | 6.25 | Replace: new score is lower | (2, 1.5) | Compare the retained pbest scores. gbest = **(1, 1)**, with objective **2**. ##### Step 8. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (1, 1) | (0, 0) | (1, 1) | | B | (2, 1.5) | (-1, -0.5) | (2, 1.5) | ##### Step 9. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (1, 1). pbest values are A: (1, 1), B: (2, 1.5). ##### Step 10. Iteration 2: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1+0=1 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1-(1)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(0)=0 $$ Add the three velocity contributions. $$ v^{(2)}=0+0+0=0 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1+0=1 $$ ##### Step 11. Iteration 2: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-1)=-0.5 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=2-(2)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(2)=-1 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-1)=-0.5 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.5)+0+(-0.5)=-1 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=2+(-1)=1 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-0.5)=-0.25 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=1.5-(1.5)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1\times0.5=0.5 $$ $$ C=0.5\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=1-(1.5)=-0.5 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1\times0.5=0.5 $$ $$ S=0.5\times(-0.5)=-0.25 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.25)+0+(-0.25)=-0.5 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=1.5+(-0.5)=1 $$ ##### Step 12. Iteration 2: evaluate the new positions **Particle A** $$ f(1, 1)=1^2+1^2=1+1=2 $$ **Particle B** $$ f(1, 1)=1^2+1^2=1+1=2 $$ ##### Step 13. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 2 | 2 | Keep old: tie | (1, 1) | | B | 6.25 | 2 | Replace: new score is lower | (1, 1) | Compare the retained pbest scores. gbest = **(1, 1)**, with objective **2**. ##### Step 14. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (1, 1) | (0, 0) | (1, 1) | | B | (1, 1) | (-1, -0.5) | (1, 1) | **Final answer.** After iteration 2, gbest = (1, 1), objective = 2. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. --- ## TEE · Class PSO photograph: all three particles through iteration 2 Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/pso-core--class-three-particles/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/pso-core/class-three-particles First understand position, velocity and memory; then solve complete two-iteration minimisation examples. Work down the page: each particle is fully updated, then all scores and memories are compared. Keep at least six digits internally for longer decimal examples; the final table displays rounded values. The positions, velocities and coefficients are transcribed from the class photo. We use f(x₁,x₂)=x₁²+x₂²−x₁x₂, minimisation, initial pbest=x, fixed supplied random factors and synchronous updates. The photo does not itself specify all these conventions; they are stated here to make the two-iteration extension unambiguous. **Minimise $$ f(x)=x_1^2+x_2^2-x_1x_2 $$ using 3 particles. Use the initial data below and perform exactly two PSO iterations, reporting position, velocity, pbest and gbest after each iteration.** [10 marks · Class data; two-iteration minimisation extension] #### Given | Particle | Initial position x⁰ | Initial velocity v⁰ | | --- | --- | --- | | A | (3.14, 7.21) | (0.1, 0.3) | | B | (4.15, 7.31) | (0.5, 0.1) | | C | (6.17, 8.27) | (0.2, 0) | w=0.5, c₁=1.2, c₂=1.2, r₁=0.5, r₂=0.1. Initialise each pbest at its initial position. Use these random coefficients unchanged in both iterations. No bounds or velocity clamp are imposed in this example. Updates are synchronous: every particle uses the same previous-iteration gbest. #### To find The complete swarm state after iterations 1 and 2, and the best location found. #### Formula and why it applies PSO combines inertia, personal memory and shared memory. Subtract positions coordinate by coordinate. Evaluate the objective only after moving. For minimisation, replace a best only when its score decreases. $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ $$ x^{t+1}=x^t+v^{t+1} $$ #### Solution ##### Step 1. Evaluate every initial position **Particle A** $$ f(3.14, 7.21)=3.14^2+7.21^2-(3.14)(7.21) $$ $$ =9.8596+51.9841-22.6394=39.2043 $$ **Particle B** $$ f(4.15, 7.31)=4.15^2+7.31^2-(4.15)(7.31) $$ $$ =17.2225+53.4361-30.3365=40.3221 $$ **Particle C** $$ f(6.17, 8.27)=6.17^2+8.27^2-(6.17)(8.27) $$ $$ =38.0689+68.3929-51.0259=55.4359 $$ ##### Step 2. Initialise the memories pbest is each initial position because there is no earlier history. The smallest objective decides gbest. $$ gbest^0=(3.14, 7.21) $$ Its score is 39.2043. Store the position, not only the score. ##### Step 3. Iteration 1: freeze the previous memories All particles in this iteration use gbest = (3.14, 7.21). pbest values are A: (3.14, 7.21), B: (4.15, 7.31), C: (6.17, 8.27). ##### Step 4. Iteration 1: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.1=0.05 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3.14-(3.14)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(3.14)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0.05+0+0=0.05 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=3.14+0.05=3.19 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.3=0.15 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.21-(7.21)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.21)=0 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(0)=0 $$ Add the three velocity contributions. $$ v^{(1)}=0.15+0+0=0.15 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=7.21+0.15=7.36 $$ ##### Step 5. Iteration 1: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.5=0.25 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=4.15-(4.15)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(4.15)=-1.01 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-1.01)=-0.1212 $$ Add the three velocity contributions. $$ v^{(1)}=0.25+0+(-0.1212)=0.1288 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=4.15+0.1288=4.2788 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.1=0.05 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.31-(7.31)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.31)=-0.1 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.1)=-0.012 $$ Add the three velocity contributions. $$ v^{(1)}=0.05+0+(-0.012)=0.038 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=7.31+0.038=7.348 $$ ##### Step 6. Iteration 1: update particle C **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.2=0.1 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=6.17-(6.17)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(6.17)=-3.03 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-3.03)=-0.3636 $$ Add the three velocity contributions. $$ v^{(1)}=0.1+0+(-0.3636)=-0.2636 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=6.17+(-0.2636)=5.9064 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0=0 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=8.27-(8.27)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(8.27)=-1.06 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-1.06)=-0.1272 $$ Add the three velocity contributions. $$ v^{(1)}=0+0+(-0.1272)=-0.1272 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(1)}=8.27+(-0.1272)=8.1428 $$ ##### Step 7. Iteration 1: evaluate the new positions **Particle A** $$ f(3.19, 7.36)=3.19^2+7.36^2-(3.19)(7.36) $$ $$ =10.1761+54.1696-23.4784=40.8673 $$ **Particle B** $$ f(4.2788, 7.348)=4.2788^2+7.348^2-(4.2788)(7.348) $$ $$ =18.30812944+53.993104-31.4406224=40.86061104 $$ **Particle C** $$ f(5.9064, 8.1428)=5.9064^2+8.1428^2-(5.9064)(8.1428) $$ $$ =34.88556096+66.30519184-48.09463392=53.09611888 $$ ##### Step 8. Iteration 1: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 39.2043 | 40.8673 | Keep old: new score is worse | (3.14, 7.21) | | B | 40.3221 | 40.86061104 | Keep old: new score is worse | (4.15, 7.31) | | C | 55.4359 | 53.09611888 | Replace: new score is lower | (5.9064, 8.1428) | Compare the retained pbest scores. gbest = **(3.14, 7.21)**, with objective **39.2043**. ##### Step 9. Iteration 1: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (3.19, 7.36) | (0.05, 0.15) | (3.14, 7.21) | | B | (4.2788, 7.348) | (0.1288, 0.038) | (4.15, 7.31) | | C | (5.9064, 8.1428) | (-0.2636, -0.1272) | (5.9064, 8.1428) | ##### Step 10. Iteration 2: freeze the previous memories All particles in this iteration use gbest = (3.14, 7.21). pbest values are A: (3.14, 7.21), B: (4.15, 7.31), C: (5.9064, 8.1428). ##### Step 11. Iteration 2: update particle A **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.05=0.025 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=3.14-(3.19)=-0.05 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.05)=-0.03 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(3.19)=-0.05 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.05)=-0.006 $$ Add the three velocity contributions. $$ v^{(2)}=0.025+(-0.03)+(-0.006)=-0.011 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=3.19+(-0.011)=3.179 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.15=0.075 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.21-(7.36)=-0.15 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.15)=-0.09 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.36)=-0.15 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.15)=-0.018 $$ Add the three velocity contributions. $$ v^{(2)}=0.075+(-0.09)+(-0.018)=-0.033 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=7.36+(-0.033)=7.327 $$ ##### Step 12. Iteration 2: update particle B **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.1288=0.0644 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=4.15-(4.2788)=-0.1288 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.1288)=-0.07728 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(4.2788)=-1.1388 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-1.1388)=-0.136656 $$ Add the three velocity contributions. $$ v^{(2)}=0.0644+(-0.07728)+(-0.136656)=-0.149536 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=4.2788+(-0.149536)=4.129264 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times0.038=0.019 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=7.31-(7.348)=-0.038 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(-0.038)=-0.0228 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(7.348)=-0.138 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.138)=-0.01656 $$ Add the three velocity contributions. $$ v^{(2)}=0.019+(-0.0228)+(-0.01656)=-0.02036 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=7.348+(-0.02036)=7.32764 $$ ##### Step 13. Iteration 2: update particle C **Coordinate 1** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-0.2636)=-0.1318 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=5.9064-(5.9064)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=3.14-(5.9064)=-2.7664 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-2.7664)=-0.331968 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.1318)+0+(-0.331968)=-0.463768 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=5.9064+(-0.463768)=5.442632 $$ **Coordinate 2** Inertia = keep a fraction of the old velocity. $$ I=0.5\times(-0.1272)=-0.0636 $$ Personal pull: first subtract the current coordinate from the personal-best coordinate. $$ d_p=8.1428-(8.1428)=0 $$ Multiply the two personal coefficients, then multiply by the difference. $$ c_1r_1=1.2\times0.5=0.6 $$ $$ C=0.6\times(0)=0 $$ Social pull: subtract the current coordinate from the frozen global-best coordinate. $$ d_g=7.21-(8.1428)=-0.9328 $$ Multiply the social coefficients, then multiply by that difference. $$ c_2r_2=1.2\times0.1=0.12 $$ $$ S=0.12\times(-0.9328)=-0.111936 $$ Add the three velocity contributions. $$ v^{(2)}=(-0.0636)+0+(-0.111936)=-0.175536 $$ Add the new velocity to the old position. The parenthesised upper label is the iteration number. $$ x^{(2)}=8.1428+(-0.175536)=7.967264 $$ ##### Step 14. Iteration 2: evaluate the new positions **Particle A** $$ f(3.179, 7.327)=3.179^2+7.327^2-(3.179)(7.327) $$ $$ =10.106041+53.684929-23.292533=40.498437 $$ **Particle B** $$ f(4.129264, 7.32764)=4.129264^2+7.32764^2-(4.129264)(7.32764) $$ $$ =17.050821182+53.69430797-30.257760057=40.487369094 $$ **Particle C** $$ f(5.442632, 7.967264)=5.442632^2+7.967264^2-(5.442632)(7.967264) $$ $$ =29.622243087+63.477295646-43.362885999=49.736652734 $$ ##### Step 15. Iteration 2: update pbest, then gbest | Particle | Old pbest score | New score | Decision | pbest position | | --- | --- | --- | --- | --- | | A | 39.2043 | 40.498437 | Keep old: new score is worse | (3.14, 7.21) | | B | 40.3221 | 40.487369094 | Keep old: new score is worse | (4.15, 7.31) | | C | 53.09611888 | 49.736652734 | Replace: new score is lower | (5.442632, 7.967264) | Compare the retained pbest scores. gbest = **(3.14, 7.21)**, with objective **39.2043**. ##### Step 16. Iteration 2: record the complete state | Particle | Position | Velocity | pbest | | --- | --- | --- | --- | | A | (3.179, 7.327) | (-0.011, -0.033) | (3.14, 7.21) | | B | (4.129264, 7.32764) | (-0.149536, -0.02036) | (4.15, 7.31) | | C | (5.442632, 7.967264) | (-0.463768, -0.175536) | (5.442632, 7.967264) | **Final answer.** After iteration 2, gbest = (3.14, 7.21), objective = 39.2043. This is the best found in these two iterations, not a claim that PSO has proved the global optimum. Swipe across the diagram to read its labels ↔ ![Motion alone does not imply improvement. Evaluate the objective at the new location before updating a best memory.](https://deep-learning-study-room.vercel.app/is2/fig/v2_pso_class_paths.961831cda1.svg) Diagram: Motion alone does not imply improvement. Evaluate the objective at the new location before updating a best memory.. Labels: 3 | 3.5 | 4 | 4.5 | 5 | 5.5 | 6 | 6.5 | 7 | 7.5 | 8 | 8.5 | Class PSO: positions after each iteration | 0 = initial, 1 = iteration 1, 2 = iteration 2 | A0 | A1 | A2 | B0 | B1 | B2 | C0 | C1 | C2 | First coordinate x₁ | The best-so-far location remains A0 = (3.14, 7.21) after both iterations. Motion alone does not imply improvement. Evaluate the objective at the new location before updating a best memory. --- ## TEE · PSO algorithm, architecture, advantages and limitations Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/pso-core--theory/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/pso-core/theory First understand position, velocity and memory; then solve complete two-iteration minimisation examples. ## Algorithm you can write in an exam 1. Define the objective and whether to minimise or maximise it. Initialise particle positions and velocities. 2. Evaluate every particle. Set each pbest to its initial position and gbest to the best among them. 3. Use the previous state to calculate each new velocity. 4. Update each position using that new velocity. Apply bounds or a velocity limit only according to the stated policy. 5. Evaluate the new positions. Update a pbest only when the new objective is better; choose gbest from the personal bests. 6. Repeat until the iteration budget or another stopping criterion is reached. Initialise → Evaluate + memories → Velocity → Position → Evaluate + update memoriesrepeat until stopping | Advantages | Limitations | | --- | --- | | Simple position/velocity updates; does not require objective derivatives | Can converge prematurely to a local region | | Uses both individual and shared experience | Results depend on coefficients, initialisation and random factors | | Useful for many continuous optimisation problems | Constraints and discrete encodings need suitable handling | | Keeps best-so-far memories | Does not guarantee the global optimum in a finite run | **Explain one application of PSO.** [3 marks · Practice] #### Answer In controller tuning, a particle can represent the gain values of a controller. The objective is a performance cost, such as tracking error plus an overshoot penalty. Each proposed gain vector is evaluated in the model. PSO updates candidate gains using individual and swarm bests. The best feasible gains found are selected and then validated on the actual system. A lower training cost alone does not prove safe real-world behaviour. | Exam situation | Action | | --- | --- | | Minimisation | Lower f is better | | Maximisation | Higher f is better; do not reverse velocity differences | | Tie in best score | Keep the previous best unless a tie rule is supplied | | Different random factors each iteration | Read the relevant row; do not reuse old factors | | Synchronous update | Freeze gbest for all particles within one iteration | | Position/velocity outside a limit | Use the limit-handling rule explicitly supplied; state an assumption if absent | **Yaad rakho:** Velocity → position → score → pbest → gbest. Ye order har iteration mein same. --- ## TEE · Reinforcement learning: learning from rewards Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-rl--what-is-rl/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-rl/what-is-rl Unit 4 · Introduction to reinforcement learning · TEE ## One interaction, with every word attached to a value | Time | Current state | Chosen action | Observed reward | Next state | | --- | --- | --- | --- | --- | | 0 | Robot in room A | Move right | −1 for the move | Room B | | 1 | Room B | Pick up parcel | +5 for pickup | Carrying parcel | | 2 | Carrying parcel | Finish | +2 for delivery | Terminal | The agent is the robot; the environment supplies the next state and reward after an action. A policy tells the robot which action to take in each state. An episode is one run up to termination. The immediate reward at time 0 is −1; the undiscounted return for the whole displayed run is −1+5+2=6. A value estimates future return, not just the next reward. For ε-greedy with two actions, ε=.1 and a unique greedy action: exploit that action with probability .9; during the remaining .1, select uniformly from both actions. Thus the greedy action has probability .9+.1/2=.95 and the other .05. If a question uses a different exploration convention, follow that definition. **In one line:** an agent learns what to do by trying actions, receiving rewards, and adjusting its behaviour so that the total reward over time is as large as possible. ## Definition (write this in the exam) **Reinforcement learning (RL)** is a type of machine learning in which an **agent** learns a **policy**, a rule that maps situations (**states**) to **actions**, by interacting with an **environment**. After every action the environment returns a numerical **reward** and a new state. The agent's goal is to maximise the **expected cumulative (discounted) reward**, called the **return**. No labelled “correct action” is given; the agent discovers good actions by trial and error. Swipe across the diagram to read its labels ↔ ![At time t the agent sees state S t , chooses action A t ; the environment replies with reward R t+1 and next state S t+1 .](https://deep-learning-study-room.vercel.app/is2/fig/rl_loop_full.a9937cb961.svg) Diagram: At time t the agent sees state S t , chooses action A t ; the environment replies with reward R t+1 and next state S t+1 .. Labels: One time-step of reinforcement learning | Agent | policy π(a | s) | Environment | P(s′ | s, a), R | action Aₜ | next state Sₜ₊₁ and reward Rₜ₊₁ | The loop repeats: S₀, A₀, R₁, S₁, A₁, R₂, S₂, … At time t the agent sees state St, chooses action At; the environment replies with reward Rt+1 and next state St+1. ## The vocabulary you must know | Term | Meaning | Example: a robot vacuum | | --- | --- | --- | | **Agent** | The learner and decision maker. | The vacuum's controller. | | **Environment** | Everything outside the agent that responds to its actions. | The room, furniture, dust. | | **State $$ s $$ ** | A description of the current situation. | Position + battery level. | | **Action $$ a $$ ** | A choice the agent can make in a state. | Forward, turn left, return to dock. | | **Reward $$ r $$ ** | Immediate scalar feedback after an action. | +1 per dust patch, −10 for hitting a wall. | | **Policy $$ \pi $$ ** | The agent's behaviour: $$ \pi(a\mid s) $$ = probability of action $$ a $$ in state $$ s $$ . | “If battery < 15%, go to dock.” | | **Return $$ G_t $$ ** | Total discounted reward from time $$ t $$ onward. | All future dust collected, minus penalties. | | **Value $$ v(s) $$ ** | Expected return starting from state $$ s $$ . | How good it is to be at the centre with a full battery. | | **Model** | The agent's knowledge of $$ P(s'\mid s,a) $$ and $$ R $$ (optional). | A map of the room. | ## Four elements of an RL system (Sutton and Barto) 1. **Policy**: defines behaviour (state → action). 2. **Reward signal**: defines the goal; the immediate desirability of a step. 3. **Value function**: long-term desirability of a state (what reward is expected later). 4. **Model of the environment** (optional): predicts the next state and reward, used for planning. Methods with a model are *model-based*; without one, *model-free* (for example, Q-learning). ## How RL differs from other learning | | Supervised | Unsupervised | Reinforcement | | --- | --- | --- | --- | | Feedback | Correct label for every input | None | A reward, often delayed | | Goal | Predict labels | Find structure (clusters) | Maximise long-term return | | Data | Fixed dataset (i.i.d.) | Fixed dataset | Generated by the agent's own actions (sequential) | | Example | Spam classifier | Customer segments (GMM) | Game-playing agent | ## Two ideas that make RL special **Delayed reward.** A move in chess earns nothing now but may win the game 30 moves later. RL must assign credit to earlier actions. **Exploration versus exploitation.** To earn reward the agent should *exploit* the best action it knows, but to find better actions it must *explore* untried ones. A common rule is **ε-greedy**: with probability $$ 1-\varepsilon $$ take the best-known action; with probability $$ \varepsilon $$ take a random one. **Yaad rakho:** Agent karta hai, environment batata hai (reward + next state), policy decide karti hai, value future ka hisaab rakhti hai. --- ## TEE · Markov property, transition matrix and state diagram Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-markov--markov-chain/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-markov/markov-chain Unit 4 · Markov processes · asked in every recent paper (5 and 10 marks) A state is the current category, such as Savings or Current. A transition is one step from one category to another. The Markov assumption says that, once the current state is known, older states provide no additional information about the next-state probabilities in this model. It does not claim that the past never mattered; the current state must capture the relevant information. **In one line:** in a Markov process the next state depends only on the present state, so the whole system is described by one table of “from → to” probabilities. ## Definitions **Markov property.** A sequence of states $$ S_1, S_2, \dots $$ has the Markov property if the future is independent of the past given the present: $$ P(S_{t+1}\mid S_t, S_{t-1},\dots,S_1)=P(S_{t+1}\mid S_t) $$ **Markov process (Markov chain).** A memoryless random process given by a pair $$ \langle \mathcal S, P\rangle $$ : a finite set of states $$ \mathcal S $$ and a **transition probability matrix** $$ P $$ with entries $$ P_{ij}=P(S_{t+1}=j\mid S_t=i) $$ Row $$ i $$ is the state you are **in now** (“from”); column $$ j $$ is the state you go **to next**. Two rules always hold: 1. Every entry is between 0 and 1: $$ 0\le P_{ij}\le 1 $$ . 2. Every row sums to 1: $$ \sum_j P_{ij}=1 $$ (from any state you must go somewhere, possibly staying). A **state transition diagram** draws each state as a circle and each non-zero $$ P_{ij} $$ as an arrow from $$ i $$ to $$ j $$ labelled with the probability. Staying in the same state is a self-loop. **Exam trap.** Rows are “from”, columns are “to”. “20% of Savings switch to Current” goes in row Savings, column Current. Always check that each row adds to 1 before moving on; an inconsistent row needs clarification before it can represent a probability distribution. **Question. A survey shows that users switch between Brand X and Brand Y each year: 25% of Brand X users switch to Brand Y and 75% remain with X; 40% of Brand Y users switch to X and 60% remain with Y. (a) Write the transition probability matrix. (b) Draw the state transition diagram.** [5 marks · Final Nov 2025 Q1d] #### Given Two states: X, Y. From X: stay 0.75, to Y 0.25. From Y: to X 0.40, stay 0.60. #### To find Matrix $$ P $$ and its diagram. #### Formula and why it applies $$ P_{ij}=P(\text{next}=j\mid\text{now}=i) $$ . Each yearly switch depends only on the brand used this year, so the process is Markov. #### Solution ##### Step 1. Order the states and fill row by row $$ P=\begin{array}{c|cc} & X & Y\\\hline X & 0.75 & 0.25\\ Y & 0.40 & 0.60\end{array}\qquad\text{i.e. } P=\begin{bmatrix}0.75&0.25\\0.40&0.60\end{bmatrix} $$ ##### Step 2. Check the rows Row X: 0.75 + 0.25 = 1. Row Y: 0.40 + 0.60 = 1. Valid. ##### Step 3. Draw the diagram Swipe across the diagram to read its labels ↔ ![Self-loops carry the retention probabilities (0.75, 0.6); the arrows between states carry the switching probabilities.](https://deep-learning-study-room.vercel.app/is2/fig/mk_brand.c2c4be3e6a.svg) Diagram: Self-loops carry the retention probabilities (0.75, 0.6); the arrows between states carry the switching probabilities.. Labels: Brand switching each year | 0.25 | 0.4 | 0.75 | 0.6 | X | Y Self-loops carry the retention probabilities (0.75, 0.6); the arrows between states carry the switching probabilities. **Final answer.** $$ P=\begin{bmatrix}0.75&0.25\\0.40&0.60\end{bmatrix} $$ with the diagram above. Marking in the official key: matrix 3 marks, diagram 2 marks. **Question. A bank offers Savings and Current accounts. Each year 20% of Savings holders switch to Current and 80% stay; 10% of Current holders switch to Savings and 90% stay. (a) Construct the transition probability matrix. (b) Draw the state diagram.** [5 marks · Re-exam Jan 2026 Q1d] #### Given For Savings: stay 80%, switch 20%. For Current: switch to Savings 10%, stay 90%. State order: Savings, Current. #### To find The transition matrix and labelled state diagram. #### Formula and why it applies $$ P_{ij}=P(S_{t+1}=j\mid S_t=i) $$ Convert percentages to fractions of 100. A row fixes the present state, while columns list possible destinations. #### Solution ##### Step 1. Matrix (rows = this year, columns = next year) $$ P=\begin{array}{c|cc} & S & C\\\hline S & 0.80 & 0.20\\ C & 0.10 & 0.90\end{array} $$ ##### Step 2. Diagram Swipe across the diagram to read its labels ↔ ![Savings (S) and Current (C) accounts per year](https://deep-learning-study-room.vercel.app/is2/fig/mk_bank.879499f6eb.svg) Diagram: Savings (S) and Current (C) accounts per year. Labels: Savings (S) and Current (C) accounts per year | 0.2 | 0.1 | 0.8 | 0.9 | S | C **Final answer.** $$ P=\begin{bmatrix}0.8&0.2\\0.1&0.9\end{bmatrix} $$ ; diagram as shown. Each part carries 2.5 marks in the key. **Yaad rakho:** Row = “abhi kahan ho”, column = “kal kahan jaoge”. Har row ka total 1. --- ## TEE · Next-step and n-step distributions Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-markov--next-step/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-markov/next-step Unit 4 · Markov processes · asked in every recent paper (5 and 10 marks) **A bank has 60 Savings customers and 40 Current customers. In one period, 80% of Savings stay and 20% switch to Current; 10% of Current switch to Savings and 90% stay. Find expected counts after one and two periods.** [3 marks · Worked example] #### Given $$ P=\begin{bmatrix}0.8&0.2\\0.1&0.9\end{bmatrix} $$ Row/column order is Savings, Current; initial counts [60,40]. #### To find Two successive count vectors. #### Formula and why it applies Each destination receives incoming contributions from every origin. Multiply origin count by the corresponding transition probability, then add. #### Solution ##### Step 1. Period 1: contributions to Savings $$ 60\times0.8=48\qquad40\times0.1=4 $$ $$ S_1=48+4=52 $$ ##### Step 2. Period 1: contributions to Current $$ 60\times0.2=12\qquad40\times0.9=36 $$ $$ C_1=12+36=48 $$ ##### Step 3. Period 2: use the new counts [52,48] $$ S_2=52\times0.8+48\times0.1=41.6+4.8=46.4 $$ ##### Step 4. Period 2: second destination $$ C_2=52\times0.2+48\times0.9=10.4+43.2=53.6 $$ **Final answer.** After one period [52,48]; after two [46.4,53.6]. Each total is 100. Fractional counts are expectations, not split people. Matrix notation [60,40]P is shorthand for these destination-column calculations. P² means P multiplied by P using row-by-column multiplication; it does not mean squaring every individual entry. **In one line:** write today's numbers as a row, multiply by the matrix, and you get tomorrow's numbers. ## The rule Let $$ x_t $$ be a **row vector** of today's shares (or counts) in each state, in the same order as the matrix rows. Then $$ x_{t+1}=x_t\,P,\qquad x_{t+n}=x_t\,P^{\,n} $$ Entry $$ j $$ of $$ x_{t+1} $$ is the **column $$ j $$ ** calculation: everyone who arrives in $$ j $$ , from every state. $$ x_{t+1}(j)=\sum_i x_t(i)\,P_{ij} $$ $$ P^{\,n} $$ is the **n-step transition matrix**: $$ (P^{\,n})_{ij} $$ is the probability of being in $$ j $$ after $$ n $$ steps when starting in $$ i $$ (the Chapman–Kolmogorov result). Counts stay counts (the total is preserved); shares stay shares (they still add to 1). **Exam trap.** Multiply the row vector by the *columns* of $$ P $$ , not its rows. A quick check: the answer's total must equal the starting total (100 guests stay 100 guests). **Question. A hotel chain has properties in Paris (P), London (L) and Rome (R). Daily movement: from Paris 60% stay, 25% to London, 15% to Rome; from London 30% to Paris, 50% stay, 20% to Rome; from Rome 20% to Paris, 30% to London, 50% stay. Today's guest counts are [50, 30, 20] in P, L, R. Find the expected counts tomorrow.** [part of 10 marks · Final Nov 2025 Q5a] #### Given $$ P=\begin{bmatrix}0.60&0.25&0.15\\0.30&0.50&0.20\\0.20&0.30&0.50\end{bmatrix},\qquad x_0=[50,\;30,\;20] $$ #### To find $$ x_1=x_0P $$ . #### Formula and why it applies $$ x_{t+1}=x_tP,\qquad x_{t+2}=(x_tP)P $$ For each destination, multiply each present-state share by its probability of arriving there, then add all arrivals. #### Solution ##### Step 1. Paris tomorrow (column 1) $$ 50(0.60)+30(0.30)+20(0.20)=30+9+4=43 $$ ##### Step 2. London tomorrow (column 2) $$ 50(0.25)+30(0.50)+20(0.30)=12.5+15+6=33.5 $$ ##### Step 3. Rome tomorrow (column 3) $$ 50(0.15)+30(0.20)+20(0.50)=7.5+6+10=23.5 $$ ##### Step 4. Check the total 43 + 33.5 + 23.5 = 100 = 50 + 30 + 20. Correct. **Final answer.** Expected guests tomorrow: Paris 43, London 33.5, Rome 23.5. **Question. Using the brand-switching matrix $$ P=\begin{bmatrix}0.75&0.25\\0.40&0.60\end{bmatrix} $$ , a customer uses Brand X this year. Find the probability that the customer uses each brand after two years.** [practice · 3 marks] #### Given Transition matrix P is printed in the question. For the two-year brand example the initial state is certainly X, so x₀=[1,0]. #### To find The requested next-step or two-step distribution. #### Formula and why it applies $$ x_{t+1}=x_tP,\qquad x_{t+2}=(x_tP)P $$ For each destination, multiply each present-state share by its probability of arriving there, then add all arrivals. #### Solution ##### Step 1. Start vector $$ x_0=[1,\;0] $$ (certainly X now). ##### Step 2. One year $$ x_1=[1,0]P=[0.75,\;0.25] $$ ##### Step 3. Two years $$ x_2=x_1P=[0.75(0.75)+0.25(0.40),\;0.75(0.25)+0.25(0.60)]=[0.6625,\;0.3375] $$ **Final answer.** After two years: X with probability 0.6625, Y with probability 0.3375. (Equivalently, row X of $$ P^2=\begin{bmatrix}0.6625&0.3375\\0.54&0.46\end{bmatrix} $$ .) ## Other next-step results, with full working in Solved papers | Paper question | Start | Next step | | --- | --- | --- | | Learning levels B, I, A (Final Q6b) | [50, 30, 20] | [36, 32, 32] | | Booking channels W, A, C (Re-exam Q5b) | [0.40, 0.35, 0.25] | [0.39, 0.335, 0.275] | | Aircraft at hubs A, B, C (Re-exam Q6b) | [30, 40, 30] | [38.5, 32.5, 29] | The complete working for each is in [Solved papers](https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q6b/). **Yaad rakho:** Kal ka number = har jagah se aane wale log jodo (column ke saath multiply). --- ## TEE · Steady state and long-run gain or loss Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-markov--steady-state/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-markov/steady-state Unit 4 · Markov processes · asked in every recent paper (5 and 10 marks) **For transitions Savings→Current=0.2 and Current→Savings=0.1, derive the stationary shares.** [3 marks · Worked example] #### Given Savings→Current=0.2, so Savings→Savings=0.8. Current→Savings=0.1, so Current→Current=0.9. Let s and c be the Savings and Current shares; s+c=1. #### To find s and c that remain unchanged after a step. #### Formula and why it applies At stationarity total incoming share equals current share. #### Solution ##### Step 1. Write Savings balance $$ s=0.8s+0.1c $$ ##### Step 2. Subtract 0.8s from both sides $$ s-0.8s=0.1c\quad\Rightarrow\quad0.2s=0.1c $$ ##### Step 3. Divide both sides by 0.1 $$ 2s=c $$ ##### Step 4. Use the total share $$ s+c=1\quad\Rightarrow\quad s+2s=1\quad\Rightarrow\quad3s=1 $$ ##### Step 5. Divide and recover the other share $$ s=1/3\qquad c=2s=2/3 $$ **Final answer.** Stationary shares [1/3,2/3]. **In one line:** a stationary distribution is unchanged by one transition; that fixed row $$ \pi $$ satisfies $$ \pi=\pi P $$ , and comparing it with today's shares tells you who gains and who loses in the long run. ## Definition A **steady-state (stationary) distribution** is a row vector $$ \pi=[\pi_1,\dots,\pi_n] $$ such that $$ \pi=\pi P,\qquad \sum_i \pi_i=1,\qquad \pi_i\ge 0 $$ Once the system has these shares, one more step leaves them unchanged. For a chain in which every state can reach every other state and that does not cycle with a fixed period (true for every paper question so far), $$ x_t\to\pi $$ from *any* starting distribution. ## Method for any number of states 1. Write one balance equation per column: $$ \pi_j=\sum_i\pi_iP_{ij} $$ . 2. For the irreducible examples here, keep n−1 independent balance equations and add $$ \sum\pi_i=1 $$ . 3. Express every $$ \pi $$ in terms of one of them, then use the sum to find it. **Two-state shortcut.** When a+b>0, if $$ P=\begin{bmatrix}1-a&a\\b&1-b\end{bmatrix} $$ then $$ \pi=\left[\frac{b}{a+b},\;\frac{a}{a+b}\right] $$ Brand switching: $$ a=0.25,\;b=0.40\Rightarrow\pi=[0.40/0.65,\;0.25/0.65]=[0.6154,\;0.3846] $$ . Bank: $$ a=0.2,\;b=0.1\Rightarrow\pi=[1/3,\;2/3] $$ . ## How to interpret a transition matrix (asked as a 2–3 mark part) - **Diagonal entries $$ P_{ii} $$ ** = **retention**: the fraction that stays. - **Off-diagonal entries $$ P_{ij} $$ ** = **transfer/loss** from $$ i $$ to $$ j $$ ; looking down column $$ j $$ shows **inflow (gain)** into $$ j $$ . - **Long-run gainer/loser**: compare $$ \pi_j $$ with the current share. Larger → gains; smaller → loses. - An **absorbing state** has $$ P_{ii}=1 $$ : once entered it is never left. **Question. Learners move daily between Beginner (B), Intermediate (I) and Advanced (A): from B 60% stay, 30% to I, 10% to A; from I 20% to B, 50% stay, 30% to A; from A 10% drop to I, 80% stay and 10% stop learning (treated as remaining in A). Find the steady-state proportions and the level with the largest long-run share.** [part of 10 marks · Final Nov 2025 Q6b] #### Given $$ P=\begin{bmatrix}0.6&0.3&0.1\\0.2&0.5&0.3\\0&0.1&0.9\end{bmatrix} $$ Row A is 0 + 0.1 + (0.8 + 0.1) = 1. #### To find $$ \pi=[\pi_B,\pi_I,\pi_A] $$ . #### Formula and why it applies $$ \pi=\pi P,\quad\sum_i\pi_i=1 $$ At stationarity each destination receives exactly its current share. Solve the balance equations together with the total-probability equation. #### Solution ##### Step 1. Column B balance $$ \pi_B=0.6\pi_B+0.2\pi_I+0\cdot\pi_A\;\Rightarrow\;0.4\pi_B=0.2\pi_I\;\Rightarrow\;\pi_I=2\pi_B $$ ##### Step 2. Column A balance $$ \pi_A=0.1\pi_B+0.3\pi_I+0.9\pi_A\;\Rightarrow\;0.1\pi_A=0.1\pi_B+0.3(2\pi_B)=0.7\pi_B\;\Rightarrow\;\pi_A=7\pi_B $$ ##### Step 3. Use the sum $$ \pi_B+2\pi_B+7\pi_B=1\;\Rightarrow\;\pi_B=0.1,\quad\pi_I=0.2,\quad\pi_A=0.7 $$ ##### Step 4. Check with column I $$ 0.3(0.1)+0.5(0.2)+0.1(0.7)=0.03+0.10+0.07=0.20=\pi_I\;\checkmark $$ **Final answer.** Long run: 10% Beginner, 20% Intermediate, 70% Advanced. Advanced holds the largest share. It is not strictly absorbing (10% drop back to I each day), but it is the dominant level. **Official key correction.** The key prints 16.3%, 25.5%, 58.2%. Those numbers do not satisfy $$ \pi=\pi P $$ (column B gives $$ 0.6(0.163)+0.2(0.255)=0.149\ne0.163 $$ ). The correct steady state is 0.1, 0.2, 0.7. **Question. For the hotel chain (Paris, London, Rome) with today's counts [50, 30, 20], find the long-run distribution and say which city gains or loses guests.** [part of 10 marks · Final Nov 2025 Q5a] #### Given $$ P=\begin{bmatrix}0.60&0.25&0.15\\0.30&0.50&0.20\\0.20&0.30&0.50\end{bmatrix} $$ #### To find Stationary shares and the requested comparison with the initial distribution. #### Formula and why it applies $$ \pi=\pi P,\quad\sum_i\pi_i=1 $$ At stationarity each destination receives exactly its current share. Solve the balance equations together with the total-probability equation. #### Solution ##### Step 1. Column P balance $$ \pi_P=0.6\pi_P+0.3\pi_L+0.2\pi_R\;\Rightarrow\;\pi_P=0.75\pi_L+0.5\pi_R $$ ##### Step 2. Column L balance $$ \pi_L=0.25\pi_P+0.5\pi_L+0.3\pi_R\;\Rightarrow\;0.5\pi_L=0.25\pi_P+0.3\pi_R $$ Substitute Step 1: $$ 0.5\pi_L=0.1875\pi_L+0.125\pi_R+0.3\pi_R\Rightarrow0.3125\pi_L=0.425\pi_R\Rightarrow\pi_L=1.36\,\pi_R $$ . Then $$ \pi_P=0.75(1.36\pi_R)+0.5\pi_R=1.52\,\pi_R $$ . ##### Step 3. Use the sum $$ (1.52+1.36+1)\pi_R=3.88\,\pi_R=1\Rightarrow\pi_R=0.2577,\;\pi_L=0.3505,\;\pi_P=0.3918 $$ Exactly: $$ \pi=[38/97,\;34/97,\;25/97] $$ . ##### Step 4. Compare with today's shares [0.50, 0.30, 0.20] | City | Today | Long run | Verdict | | --- | --- | --- | --- | | Paris | 0.50 | 0.392 | Loses guests | | London | 0.30 | 0.351 | Gains | | Rome | 0.20 | 0.258 | Gains | **Final answer.** $$ \pi\approx[0.392,\;0.351,\;0.258] $$ : Paris loses share; London and Rome gain. (The key's 0.3928 for Paris is a rounding slip; the three values must add to 1.) ## Other steady states from the papers | Question | Steady state | Compared with now | | --- | --- | --- | | Brand X/Y (Final Q1d) | [8/13, 5/13] = [0.615, 0.385] | — | | Savings/Current (Re-exam Q1d) | [1/3, 2/3] | Current dominates | | Booking W, A, C (Re-exam Q5b) | [8/21, 1/3, 2/7] = [0.381, 0.333, 0.286] | Now [0.40, 0.35, 0.25]: Website and App shrink slightly; Counter grows. | | Aircraft A, B, C (Re-exam Q6b) | [25/63, 20/63, 2/7] = [0.397, 0.317, 0.286] | Now [0.3, 0.4, 0.3]: A gains; B loses; C loses slightly. | **Official key correction (Re-exam Q5b).** The key says the Website “may grow”. Its steady share 0.381 is below today's 0.40, so the Website actually shrinks slightly; Counter is the one that grows. **Yaad rakho:** Steady state = “ab kuch nahi badlega”: $$ \pi = \pi P $$ aur total 1. Gain/loss = steady share vs aaj ka share. --- ## TEE · Markov reward process, return and discount Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--mrp-return/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-bellman/mrp-return Unit 4 · Markov reward process, Markov decision process, value functions, Bellman equations, applications ## Discounting changes the weight of later rewards γ (gamma) is a discount factor. Immediate reward has weight 1; one-step-later reward has weight γ; two-step-later reward has weight γ². This is repeated multiplication, not subtracting γ from a reward. An expectation averages possible future trajectories. A return G is the discounted total along one realised trajectory. **In one line:** a Markov reward process is a Markov chain that also pays a reward in each state; the return adds those rewards, shrinking later ones by γ. ## Definition A **Markov reward process (MRP)** is a tuple $$ \langle\mathcal S,P,R,\gamma\rangle $$ : - $$ \mathcal S $$ : finite set of states; $$ P $$ : transition matrix, $$ P_{ss'}=P(S_{t+1}=s'\mid S_t=s) $$ ; - $$ R $$ : reward function, $$ R(s)=\mathbb E[R_{t+1}\mid S_t=s] $$ (the expected reward for leaving $$ s $$ ); - $$ \gamma\in[0,1] $$ : the **discount factor**. There are no actions yet; the process simply drifts and collects rewards. (Add actions and you get an MDP.) ## Return The **return** $$ G_t $$ is the total discounted reward from time $$ t $$ : $$ G_t=R_{t+1}+\gamma R_{t+2}+\gamma^2R_{t+3}+\dots=\sum_{k=0}^{\infty}\gamma^kR_{t+k+1} $$ **Why discount?** (1) With bounded rewards, it keeps the infinite discounted sum finite when $$ \gamma<1 $$ . (2) Future rewards are uncertain. (3) Immediate rewards are usually preferred (like interest on money). $$ \gamma=0 $$ makes the agent short-sighted (only the next reward counts); $$ \gamma\to1 $$ makes it far-sighted. **Question. An episode gives rewards $$ R_1=-2,\;R_2=-1,\;R_3=-2,\;R_4=0 $$ and then ends. Find $$ G_0 $$ with $$ \gamma=0.9 $$ .** [practice · 2 marks] #### Given Rewards [−2,−1,−2,0]; discount γ=0.9; episode ends after the fourth reward. #### To find Return G₀. #### Formula and why it applies The first reward is undiscounted. Each later reward receives one extra factor of γ. $$ G_0=R_1+\gamma R_2+\gamma^2R_3+\gamma^3R_4 $$ #### Solution ##### Step 1. Write the discounted sum $$ G_0=R_1+\gamma R_2+\gamma^2R_3+\gamma^3R_4 $$ ##### Step 2. Substitute $$ G_0=-2+0.9(-1)+0.81(-2)+0.729(0)=-2-0.9-1.62=-4.52 $$ **Final answer.** $$ G_0=-4.52 $$ . ## State-value function of an MRP The **value** of a state is the expected return when starting there: $$ v(s)=\mathbb E\,[\,G_t\mid S_t=s\,] $$ The return of one episode is random; the value is its average over all possible futures. **Yaad rakho:** Return = aaj ka reward + γ × kal ka + γ² × parson ka… γ chhota = jaldi chahiye, γ bada = lambi soch. **An episode gives rewards [2, 4, 8] and then ends. Find the return from the start with γ=0.5.** [2 marks · Worked example] #### Given Rewards in order: R₁=2, R₂=4, R₃=8. Discount factor γ=0.5. The episode then ends, so there are no later rewards. #### To find G₀: the discounted sum of all three rewards, measured from the start. #### Formula and why it applies $$ G_0=R_1+\gamma R_2+\gamma^2R_3 $$ Keep the first reward undiscounted; multiply each later reward by the appropriate power. #### Solution ##### Step 1. First reward contribution $$ 1\times(2)=2 $$ ##### Step 2. Second reward contribution $$ 0.5\times(4)=2 $$ ##### Step 3. Third reward contribution $$ 0.5^2\times(8)=2 $$ ##### Step 4. Add contributions $$ G_0=(2)+(2)+(2)=6 $$ **Final answer.** 6. **An episode gives rewards [-1, 5, 2] and then ends. Find the return from the start with γ=0.9.** [2 marks · Worked example] #### Given Rewards in order: R₁=-1, R₂=5, R₃=2. Discount factor γ=0.9. The episode then ends, so there are no later rewards. #### To find G₀: the discounted sum of all three rewards, measured from the start. #### Formula and why it applies $$ G_0=R_1+\gamma R_2+\gamma^2R_3 $$ Keep the first reward undiscounted; multiply each later reward by the appropriate power. #### Solution ##### Step 1. First reward contribution $$ 1\times(-1)=-1 $$ ##### Step 2. Second reward contribution $$ 0.9\times(5)=4.5 $$ ##### Step 3. Third reward contribution $$ 0.9^2\times(2)=1.62 $$ ##### Step 4. Add contributions $$ G_0=(-1)+(4.5)+(1.62)=5.12 $$ **Final answer.** 5.12. **An episode gives rewards [2, 4, 8] and then ends. Find the return from the start with γ=0.** [2 marks · Worked example] #### Given Rewards in order: R₁=2, R₂=4, R₃=8. Discount factor γ=0. The episode then ends, so there are no later rewards. #### To find G₀: the discounted sum of all three rewards, measured from the start. #### Formula and why it applies $$ G_0=R_1+\gamma R_2+\gamma^2R_3 $$ Keep the first reward undiscounted; multiply each later reward by the appropriate power. #### Solution ##### Step 1. First reward contribution $$ 1\times(2)=2 $$ ##### Step 2. Second reward contribution $$ 0\times(4)=0 $$ ##### Step 3. Third reward contribution $$ 0^2\times(8)=0 $$ ##### Step 4. Add contributions $$ G_0=(2)+(0)+(0)=2 $$ **Final answer.** 2. --- ## TEE · Bellman equation for an MRP (with a solved example) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--bellman-mrp/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-bellman/bellman-mrp Unit 4 · Markov reward process, Markov decision process, value functions, Bellman equations, applications **A state pays immediate reward 2. Next it reaches a state of value 10 with probability 0.6 or a state of value 0 with probability 0.4. With γ=0.5, calculate its Bellman value.** [2 marks · Worked example] #### Given Immediate reward R=2; discount γ=0.5. Next-state possibilities: value 10 with probability 0.6, or value 0 with probability 0.4. These supplied continuation values already include rewards after reaching those states. #### To find Immediate reward plus discounted expected next value. #### Formula and why it applies $$ v(s)=R(s)+\gamma\sum_{s'}P(s'\mid s)v(s') $$ s′ is read “next state”; the prime is not a derivative. The sum is a weighted average of next-state values. #### Solution ##### Step 1. First branch contribution $$ 0.6\times10=6 $$ ##### Step 2. Second branch contribution $$ 0.4\times0=0 $$ ##### Step 3. Average, then discount $$ 6+0=6\qquad0.5\times6=3 $$ ##### Step 4. Add immediate reward $$ v(s)=2+3=5 $$ **Final answer.** v(s)=5. In the matrix expression, I is the identity matrix (ones on the diagonal and zero elsewhere); multiplying by I leaves a vector unchanged. The inverse solves simultaneous equations when it exists. For the small exam examples below, ordinary substitution is sufficient. **In one line:** the value of a state = the reward you get now + γ × the average value of where you land next. ## Derivation (short and examinable) $$ \begin{aligned}v(s)&=\mathbb E[G_t\mid S_t=s]\\&=\mathbb E[R_{t+1}+\gamma(R_{t+2}+\gamma R_{t+3}+\dots)\mid S_t=s]\\&=\mathbb E[R_{t+1}+\gamma G_{t+1}\mid S_t=s]\\&=R(s)+\gamma\sum_{s'\in\mathcal S}P_{ss'}\,v(s')\end{aligned} $$ The key step is $$ G_t=R_{t+1}+\gamma G_{t+1} $$ : the return splits into the immediate reward plus the discounted return from the next state. That recursive split is the whole idea of every Bellman equation. **Matrix form.** Stack the values in a column $$ v $$ : $$ v=R+\gamma Pv\Rightarrow (I-\gamma P)v=R\Rightarrow v=(I-\gamma P)^{-1}R $$ . For small problems just solve the linear equations. Swipe across the diagram to read its labels ↔ ![Done is terminal (it loops to itself with reward 0), so v(Done) = 0.](https://deep-learning-study-room.vercel.app/is2/fig/mrp_student.be69eb4eb6.svg) Diagram: Done is terminal (it loops to itself with reward 0), so v(Done) = 0.. Labels: A three-state Markov reward process (rewards: Class −2, Lab −1, Done 0) | 0.5 | 0.5 | 0.4 | 0.6 | 1 | Class | Lab | Done Done is terminal (it loops to itself with reward 0), so v(Done) = 0. **Question. In the MRP shown, Class pays −2 and Lab pays −1 when left; Done pays 0 and is terminal. From Class: 0.5 to Lab, 0.5 to Done. From Lab: 0.4 to Class, 0.6 to Done. With $$ \gamma=0.9 $$ , find v(Class) and v(Lab).** [practice · 5 marks] #### Given $$ R(C)=-2,\;R(L)=-1,\;v(D)=0,\;\gamma=0.9 $$ . #### To find $$ v(C),\;v(L) $$ . #### Formula and why it applies $$ v(s)=R(s)+\gamma\sum_{s'}P_{ss'}v(s') $$ It holds for every state, so we get one equation per non-terminal state. #### Solution ##### Step 1. Equation for Class $$ v(C)=-2+0.9\,[\,0.5\,v(L)+0.5\,(0)\,]=-2+0.45\,v(L) $$ ##### Step 2. Equation for Lab $$ v(L)=-1+0.9\,[\,0.4\,v(C)+0.6\,(0)\,]=-1+0.36\,v(C) $$ ##### Step 3. Substitute Step 2 into Step 1 $$ v(C)=-2+0.45(-1+0.36\,v(C))=-2.45+0.162\,v(C)\Rightarrow0.838\,v(C)=-2.45 $$ ##### Step 4. Solve $$ v(C)=\frac{-2.45}{0.838}=-2.9236,\qquad v(L)=-1+0.36(-2.9236)=-2.0525 $$ ##### Step 5. Check Class $$ -2+0.45(-2.0525)=-2-0.9236=-2.9236\;\checkmark $$ **Final answer.** $$ v(\text{Class})\approx-2.924,\;v(\text{Lab})\approx-2.053,\;v(\text{Done})=0 $$ . Lab is “less bad” because it is more likely to finish quickly. **Yaad rakho:** Bellman = “abhi ka reward + γ × aage ka average value”. Har state ki ek equation, phir solve. ## The substitution algebra in slow motion $$ v_C=-2+0.45(-1+0.36v_C) $$ Multiply .45 into both bracket terms: .45×(−1)=−.45 and .45×.36=.162. $$ v_C=-2-0.45+0.162v_C=-2.45+0.162v_C $$ Subtract .162vC from both sides; the coefficient on the left is 1−.162=.838. $$ 0.838v_C=-2.45\quad\Rightarrow\quad v_C=-2.45/0.838\approx-2.923627685 $$ Now use the Lab equation: .36×(−2.923627685)≈−1.052505967; adding −1 gives vL≈−2.052505967. --- ## TEE · Markov decision process, policies and value functions Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--mdp-policy/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-bellman/mdp-policy Unit 4 · Markov reward process, Markov decision process, value functions, Bellman equations, applications | Read this | Meaning | Concrete example | | --- | --- | --- | | π(a|s) | Probability the policy chooses action a in state s | In room A: left .3, right .7 | | P(s′|s,a) | Probability of the environment’s next state after that action | Right action reaches room B with probability .8 | | vπ(s) | Expected return starting in state s and following policy π | A number scoring the state | | qπ(s,a) | Expected return if action a is taken first, then policy π is followed | One score for each possible first action | | π in this chapter | A policy | Unlike π in a Markov-chain stationary distribution; context distinguishes them | The policy chooses an action; the transition model describes what happens after the action. They are two separate probability distributions. The worked examples use stationary Markov policies, depending on the current state and unchanged over time. More general policies may depend on time or history. **In one line:** an MDP is an MRP in which the agent chooses actions; a policy says which action to choose, and value functions score how good a state or a state–action pair is under that policy. ## Markov decision process A **Markov decision process (MDP)** is a tuple $$ \langle\mathcal S,\mathcal A,P,R,\gamma\rangle $$ : - $$ \mathcal S $$ : states; $$ \mathcal A $$ : actions; - $$ P^{a}_{ss'}=P(S_{t+1}=s'\mid S_t=s,A_t=a) $$ : transition probabilities, now depending on the action; - $$ R^{a}_{s}=\mathbb E[R_{t+1}\mid S_t=s,A_t=a] $$ : expected reward for taking $$ a $$ in $$ s $$ ; - $$ \gamma $$ : discount factor. The Markov property still holds: the next state and reward depend only on the current state and action. An MDP is a common model for RL when the chosen state captures the relevant information; partially observed problems require additional treatment. Swipe across the diagram to read its labels ↔ ![In s₁ the agent chooses “stay” or “go” (black dots). The environment then decides the outcome: “go” reaches s₂ with probability 0.8 (reward +5) or falls back to s₁ with 0.2 (reward 0).](https://deep-learning-study-room.vercel.app/is2/fig/mdp_two.331bc03880.svg) Diagram: In s₁ the agent chooses “stay” or “go” (black dots). The environment then decides the outcome: “go” reaches s₂ with probability 0.8 (reward +5) or falls back to s₁ with 0.2 (reward 0).. Labels: A two-state MDP: actions branch, then the environment decides the next state | s₁ | s₂ | a = stay | a = go | p = 1, r = +1 | p = 0.8, r = +5 | p = 0.2, r = 0 In s₁ the agent chooses “stay” or “go” (black dots). The environment then decides the outcome: “go” reaches s₂ with probability 0.8 (reward +5) or falls back to s₁ with 0.2 (reward 0). ## Policy A **policy** $$ \pi $$ is a distribution over actions given states: $$ \pi(a\mid s)=P(A_t=a\mid S_t=s) $$ - **Deterministic** policy: one fixed action per state, $$ a=\pi(s) $$ . - **Stochastic** policy: probabilities, e.g. $$ \pi(\text{stay}\mid s_1)=0.5,\;\pi(\text{go}\mid s_1)=0.5 $$ . The policies used in these examples are stationary Markov policies. The term policy more generally also permits time-dependent or history-dependent decision rules. ## Value functions **State-value function**: expected return from state $$ s $$ when following $$ \pi $$ : $$ v_\pi(s)=\mathbb E_\pi[G_t\mid S_t=s] $$ **Action-value function (Q-function)**: expected return when taking action $$ a $$ in $$ s $$ and following $$ \pi $$ afterwards: $$ q_\pi(s,a)=\mathbb E_\pi[G_t\mid S_t=s,A_t=a] $$ They are linked by averaging over the policy's choices: $$ v_\pi(s)=\sum_{a\in\mathcal A}\pi(a\mid s)\,q_\pi(s,a) $$ **Optimal** value functions take the best policy: $$ v_*(s)=\max_\pi v_\pi(s) $$ and $$ q_*(s,a)=\max_\pi q_\pi(s,a) $$ . An optimal policy acts greedily on $$ q_* $$ : $$ \pi_*(s)=\arg\max_a q_*(s,a) $$ . | | Markov process | MRP | MDP | | --- | --- | --- | --- | | Tuple | $$ \langle\mathcal S,P\rangle $$ | $$ \langle\mathcal S,P,R,\gamma\rangle $$ | $$ \langle\mathcal S,\mathcal A,P,R,\gamma\rangle $$ | | Rewards | No | Yes | Yes | | Decisions | No | No | Yes (actions) | | Value | — | $$ v(s) $$ | $$ v_\pi(s),\;q_\pi(s,a) $$ | **Yaad rakho:** MP → +reward = MRP → +action = MDP. v = “state kitni achhi”, q = “is state mein yeh action kitna achha”. --- ## TEE · Bellman expectation and optimality equations (worked) Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--bellman-equations/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-bellman/bellman-equations Unit 4 · Markov reward process, Markov decision process, value functions, Bellman equations, applications **At a state, action Left has value 3 and action Right has value 7. A policy chooses Left with probability 0.25 and Right with probability 0.75. Find the value under this policy, the maximum action value and the greedy action.** [2 marks · Worked example] #### Given Action values: Left=3 and Right=7. Policy probabilities: Left=0.25 and Right=0.75. The supplied action values already include subsequent returns. #### To find A weighted average, a maximum and the label attaining that maximum. #### Formula and why it applies $$ v_\pi(s)=\sum_a\pi(a\mid s)q_\pi(s,a) $$ The expectation averages according to policy; maximisation selects a largest action score. #### Solution ##### Step 1. Weight Left $$ 0.25\times3=0.75 $$ ##### Step 2. Weight Right $$ 0.75\times7=5.25 $$ ##### Step 3. Add for the policy value $$ v_\pi=0.75+5.25=6 $$ ##### Step 4. Read max and argmax max(3,7)=7 is a number. argmax is Right, the action with that number. **Final answer.** Policy value 6; maximum action value 7; greedy action Right. **In one line:** the expectation equations score a given policy; the optimality equations replace the policy's average by a max and describe the best possible behaviour. ## Bellman expectation equations Split the return into “reward now + discounted value later”, and average over the policy and the environment: $$ v_\pi(s)=\sum_{a}\pi(a\mid s)\sum_{s'}P(s'\mid s,a)\,\big[\,R(s,a,s')+\gamma\,v_\pi(s')\,\big] $$ $$ q_\pi(s,a)=\sum_{s'}P(s'\mid s,a)\,\Big[\,R(s,a,s')+\gamma\sum_{a'}\pi(a'\mid s')\,q_\pi(s',a')\,\Big] $$ Read the first one aloud: *for each action the policy might take, for each next state the environment might give, add the reward and γ × the next state's value, weighted by both probabilities.* ## Bellman optimality equations $$ v_*(s)=\max_{a}\sum_{s'}P(s'\mid s,a)\,\big[\,R(s,a,s')+\gamma\,v_*(s')\,\big] $$ $$ q_*(s,a)=\sum_{s'}P(s'\mid s,a)\,\big[\,R(s,a,s')+\gamma\max_{a'}q_*(s',a')\,\big] $$ The only change is $$ \sum_a\pi(a\mid s)\to\max_a $$ : the optimal agent picks the best action instead of averaging. These equations are non-linear (because of max), so they are solved iteratively (value iteration, policy iteration, Q-learning). **Question. In the two-state MDP (figure in the previous lesson), s₂ is terminal with $$ v(s_2)=0 $$ and $$ \gamma=0.9 $$ . In s₁, “stay” returns to s₁ with reward +1; “go” reaches s₂ with probability 0.8 and reward +5, or returns to s₁ with probability 0.2 and reward 0. (a) Find $$ v_\pi(s_1) $$ for the random policy $$ \pi(\text{stay})=\pi(\text{go})=0.5 $$ . (b) Find $$ v_*(s_1) $$ and the optimal action.** [practice · 6 marks] #### Given $$ q(s_1,\text{stay})=1+0.9\,v(s_1) $$ ; $$ q(s_1,\text{go})=0.8(5+0.9\cdot0)+0.2(0+0.9\,v(s_1))=4+0.18\,v(s_1) $$ . #### To find The value under the random policy, the optimal value and the optimal action. #### Formula and why it applies $$ v_\pi(s)=\sum_a\pi(a\mid s)q_\pi(s,a),\qquad v_*(s)=\max_a q_*(s,a) $$ The expectation equation averages actions using the given policy. The optimality equation chooses the largest action value. Each action value already includes its expected immediate reward and discounted continuation. #### Solution ##### Step 1. (a) Expectation equation for the random policy $$ v=0.5(1+0.9v)+0.5(4+0.18v)=2.5+0.54v\Rightarrow0.46v=2.5\Rightarrow v_\pi(s_1)=5.4348 $$ ##### Step 2. (b) Try “go” always $$ v=4+0.18v\Rightarrow v=4/0.82=4.878 $$ Check whether switching helps: $$ q(\text{stay})=1+0.9(4.878)=5.390>4.878 $$ . So “go” is not optimal. ##### Step 3. Try “stay” always $$ v=1+0.9v\Rightarrow v=10 $$ Check: $$ q(\text{go})=4+0.18(10)=5.8<10=q(\text{stay}) $$ . No improvement possible. **Final answer.** (a) $$ v_\pi(s_1)\approx5.43 $$ . (b) $$ v_*(s_1)=10 $$ with $$ q_*(s_1,\text{stay})=10 $$ , $$ q_*(s_1,\text{go})=5.8 $$ : the optimal action is **stay**. A steady +1 forever is worth $$ 1/(1-0.9)=10 $$ , more than a one-time +5. ## Seeing the max at work: value iteration Start with $$ v_0(s_1)=0 $$ and apply the optimality equation repeatedly: $$ v_{k+1}=\max(1+0.9v_k,\;4+0.18v_k) $$ . | k | stay = 1 + 0.9vₖ | go = 4 + 0.18vₖ | vₖ₊₁ | best | | --- | --- | --- | --- | --- | | 0 | 1.000 | 4.000 | 4.000 | go | | 1 | 4.600 | 4.720 | 4.720 | go | | 2 | 5.248 | 4.850 | 5.248 | stay | | 3 | 5.723 | 4.945 | 5.723 | stay | | … | converges | | 10 | stay | Early on the agent prefers the quick +5; as it “looks further ahead”, the steady reward wins. This is why γ and long-term value matter. **Yaad rakho:** Expectation = policy ka average (Σπ). Optimality = best action (max). Baaki sab same: reward + γ × next value. ## Expand the random-policy equation For the full MDP question above, “stay” gives 1+.9v. “Go” has two possible transitions: .8×(5+.9×0)=4 and .2×(0+.9v)=.18v, so its total is 4+.18v. The random policy then averages these two action values. $$ v=0.5(1+0.9v)+0.5(4+0.18v) $$ $$ v=0.5+0.45v+2+0.09v=2.5+0.54v $$ Subtract .54v from both sides, then divide by .46. $$ (1-0.54)v=2.5\quad0.46v=2.5\quad v=2.5/0.46\approx5.434782609 $$ --- ## TEE · Using Bellman equations: policy and value iteration Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--solving/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-bellman/solving Unit 4 · Markov reward process, Markov decision process, value functions, Bellman equations, applications A **backup** means applying the Bellman equation once to improve a value estimate. In value iteration, each sweep uses the previous sweep’s values for all states. In policy iteration, first solve values for the current policy; then choose the action with the largest one-step action value using those solved continuation values. Repeat if the policy changes. The “try go, then improve to stay” calculation in the previous lesson is one complete policy-improvement example. **In one line:** policy/value iteration use Bellman equations with a model; Q-learning uses sampled experience. These are examples of methods, not a claim about every RL algorithm. ## 1. Policy evaluation Given $$ \pi $$ , compute $$ v_\pi $$ by repeatedly applying the expectation equation until values stop changing: $$ v_{k+1}(s)\leftarrow\sum_a\pi(a\mid s)\sum_{s'}P(s'\mid s,a)\,[R+\gamma v_k(s')] $$ ## 2. Policy iteration Start with any policy π → Evaluate: find $$ v_\pi $$ → Improve: $$ \pi'(s)=\arg\max_a q_\pi(s,a) $$ → Repeat until the policy stops changing For a finite discounted MDP with exact policy evaluation and consistent tie handling, policy improvement does not reduce value and policy iteration can terminate at an optimal policy. ## 3. Value iteration Skip full evaluation; apply the optimality equation directly: $$ v_{k+1}(s)\leftarrow\max_a\sum_{s'}P(s'\mid s,a)\,[R+\gamma v_k(s')] $$ Stop when the change is tiny, then act greedily. The table in the previous lesson is value iteration on one state. Optional extension: Q-learning (not an observed PYQ) The two supplied TEE papers do not ask a Q-learning update. The January 2026 paper asks an explanation of the Bellman equation and RL applications. This extra worked example can wait until those requirements are covered. ## Q-learning Q-learning is included to illustrate a Bellman target; it is not separately named in the supplied introductory RL syllabus. When $$ P $$ and $$ R $$ are unknown, the agent learns $$ q_* $$ from its own experience $$ (s,a,r,s') $$ : $$ Q(s,a)\leftarrow Q(s,a)+\alpha\,\big[\,r+\gamma\max_{a'}Q(s',a')-Q(s,a)\,\big] $$ $$ \alpha $$ is the learning rate. The bracket is the **temporal-difference error**: (Bellman target) − (current guess). The agent usually explores with ε-greedy actions while learning. **Question. A Q-table has $$ Q(s,a)=2 $$ . The agent takes $$ a $$ in $$ s $$ , receives $$ r=3 $$ and lands in $$ s' $$ where $$ \max_{a'}Q(s',a')=5 $$ . With $$ \alpha=0.5,\;\gamma=0.9 $$ , find the updated $$ Q(s,a) $$ .** [Optional authored example] #### Given Old Q=2, reward=3, best next-state Q=5, α=0.5, γ=0.9; the next state is nonterminal. #### To find The updated Q estimate. #### Formula and why it applies Move partway from the old estimate toward the one-step Bellman target. $$ Q_{new}=Q_{old}+\alpha[r+\gamma\max Q_{next}-Q_{old}] $$ #### Solution ##### Step 1. Target $$ r+\gamma\max_{a'}Q(s',a')=3+0.9(5)=7.5 $$ ##### Step 2. Update $$ Q\leftarrow2+0.5(7.5-2)=2+2.75=4.75 $$ **Final answer.** $$ Q(s,a)=4.75 $$ . **Yaad rakho:** Model pata hai → policy/value iteration. Model nahi pata → Q-learning (experience se seekho). --- ## TEE · 10-mark answer: Bellman equation and applications of RL Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--ten-mark-answer/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/read/u4-bellman/ten-mark-answer Unit 4 · Markov reward process, Markov decision process, value functions, Bellman equations, applications **Asked as:** “Explain Bellman equation in reinforcement learning. Also give applications of reinforcement learning.” [10 marks · Re-exam Jan 2026 Q7b; marking: Bellman explanation 5, applications 5] Write the answer below in this order. It fits in about two pages. ### Part A · Bellman equation (5 marks) **1. Setting.** Reinforcement learning problems are modelled as a Markov decision process $$ \langle\mathcal S,\mathcal A,P,R,\gamma\rangle $$ . The agent follows a policy $$ \pi(a\mid s) $$ and wants to maximise the return $$ G_t=\sum_{k\ge0}\gamma^kR_{t+k+1} $$ . **2. Value functions.** $$ v_\pi(s)=\mathbb E_\pi[G_t\mid S_t=s] $$ and $$ q_\pi(s,a)=\mathbb E_\pi[G_t\mid S_t=s,A_t=a] $$ . **3. Key idea.** The return is recursive: $$ G_t=R_{t+1}+\gamma G_{t+1} $$ . Taking expectations gives the **Bellman expectation equation**: $$ v_\pi(s)=\sum_a\pi(a\mid s)\sum_{s'}P(s'\mid s,a)\,[R(s,a,s')+\gamma v_\pi(s')] $$ $$ q_\pi(s,a)=\sum_{s'}P(s'\mid s,a)\,\Big[R(s,a,s')+\gamma\sum_{a'}\pi(a'\mid s')q_\pi(s',a')\Big] $$ **4. Optimality.** For the optimal policy the average over actions becomes a maximum (**Bellman optimality equation**): $$ v_*(s)=\max_a\sum_{s'}P(s'\mid s,a)\,[R+\gamma v_*(s')],\qquad q_*(s,a)=\sum_{s'}P(s'\mid s,a)\,[R+\gamma\max_{a'}q_*(s',a')] $$ **5. Meaning and use.** The value of a state equals the immediate reward plus the discounted value of the successor state. This breaks a long-horizon problem into one-step sub-problems (dynamic programming). It is the basis of policy evaluation, policy iteration, value iteration and Q-learning; the optimal policy is $$ \pi_*(s)=\arg\max_a q_*(s,a) $$ . **6. Small illustration** (adds clarity): if staying in a state gives +1 each step with $$ \gamma=0.9 $$ , then $$ v=1+0.9v\Rightarrow v=10 $$ . ### Part B · Applications of reinforcement learning (5 marks) 1. **Game playing**: AlphaGo (defeated Lee Sedol, 2016), Atari agents (DQN), and the Mario agent built in this course's lab. 2. **Robotics**: walking, grasping and balancing learned by trial and error in simulation. 3. **Autonomous driving**: lane changing, merging and parking decisions. 4. **Recommendation systems**: choosing content to maximise long-term engagement rather than single clicks. 5. **Finance**: trading and portfolio rebalancing as sequential decisions. 6. **Healthcare**: dynamic treatment regimes (dose adjustment over time). 7. **Resource and energy management**: data-centre cooling control, traffic-signal timing, inventory control. 8. **Language models**: reinforcement learning from human feedback (RLHF) for chatbots. #### One application in detail: traffic-signal control | | | | --- | --- | | Agent | Controller at an intersection. | | State | Queue length and waiting time on each approach, current signal phase. | | Actions | Keep the current phase or switch to the next green phase. | | Reward | Negative of total waiting time (or vehicles cleared per cycle). | | Learning | Q-learning updates $$ Q(s,a) $$ from observed traffic; ε-greedy exploration tries alternative timings. | | Benefit | Adapts to rush hours and incidents; fixed timers cannot. | | Challenge | Safety during exploration, so agents are trained in simulation first. | **Yaad rakho:** Bellman ka sar: “value = reward abhi + γ × value aage”. Application likho to S, A, R batao: examiner ko yahi chahiye. --- ## TEE · Fuzzy formulas, with meanings and worked-example links Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/formulas/m2-formulas-v2--fuzzy/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/formulas/m2-formulas-v2/fuzzy One place for the formulas, meanings, conditions and links to full worked examples. ## Fuzzy formulas ### Triangular membership $$ \mu(x)=\max\left(0,\min\left(\frac{x-a}{b-a},\frac{c-x}{c-b}\right)\right) $$ a=left zero, b=peak, c=right zero, with a0 sets the width. At x=c the exponent is zero and membership is 1. [Worked exponential calculations →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--smooth-memberships/) ### Generalised bell membership $$ \mu(x)=\frac{1}{1+|(x-c)/a|^{2b}} $$ a>0 controls width; b>0 controls steepness; c is the centre. First subtract c, divide by a, take absolute value, apply the power, add 1, then take its reciprocal. ### Alpha-cut $$ A_\alpha=\{x:\mu_A(x)\ge\alpha\} $$ α is the chosen membership threshold. For 0<α≤1 keep inputs meeting or exceeding it, including threshold-equality endpoints. [Discrete and triangular examples →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/fuzzy-core--properties/) --- ## TEE · GA and PSO formulas, symbols and memory rules Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/formulas/m2-formulas-v2--ga-pso/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/formulas/m2-formulas-v2/ga-pso One place for the formulas, meanings, conditions and links to full worked examples. ### Binary decoding $$ x=\sum_{i=1}^{n}b_i2^{n-i} $$ bᵢ is a bit; the maximum n-bit unsigned integer is 2ⁿ−1. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--encoding/) ### Roulette selection $$ P_i=F_i/\sum_jF_j\quad C_i=\sum_{j=1}^iP_j $$ Fitness weights must be nonnegative with a positive total. For minimisation use the specified transformation or selection method. C is cumulative probability. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--selection/) ### Population average $$ \bar f=\frac{f_1+\cdots+f_N}{N} $$ N is the number of individuals. Compare both best and average; mutation need not improve either. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--two-generations/) ### PSO velocity $$ v^{t+1}=wv^t+c_1r_1(pbest^t-x^t)+c_2r_2(gbest^t-x^t) $$ w weights inertia; c₁r₁ weights personal pull; c₂r₂ weights social pull. Apply the same equation separately to each coordinate. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--equation/) ### PSO position $$ x^{t+1}=x^t+v^{t+1} $$ Use the new velocity, not the old one. Then reevaluate the objective. [Full explanation and worked solutions →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/pso-core--two-particles/) **Memory updates:** minimise → replace a pbest only for a lower score; maximise → only for a higher score. gbest is a position selected from the personal bests. Store the corresponding objective score alongside it. Keep previous best on ties unless instructed otherwise. **Yaad rakho:** Formula ke pehle objective direction aur old state likho. Iteration 2 mein fresh positions aur retained memories use karo. ## Encoding and minimisation conventions $$ x=L+\frac{d}{2^n-1}(U-L) $$ For the specified uniform real-range encoding: decode n bits into unsigned integer d first. L and U are the physical limits. [Endpoint examples →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--encoding/) $$ F=\frac{1}{1+f(x)} $$ This is the stated roulette transformation in our nonnegative-cost minimisation example. It is not required for every GA. It converts a smaller cost into a larger selection weight. [Complete minimisation example →](https://deep-learning-study-room.vercel.app/read/is2/m2/read/ga-core--minimisation/) --- ## TEE · Unit 1 formulas: probability, Bayes, naive Bayes, Bayesian networks, GMM Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/formulas/tee-formulas--unit1/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/formulas/tee-formulas/unit1 Probability, Bayes, GMM/EM, Markov chains and Bellman equations (Units 2–3 are in the M2 sheet) ## Conditions and notation for this revision sheet Conditional probabilities require nonzero evidence probability. “Prior” means the class probability before the features; “likelihood” is the feature probability given a class; “posterior” is the class probability after evidence. Σ adds alternatives; Π multiplies factors; argmax returns the winning label. For Laplace smoothing, Nc is the class count and k is the number of possible values of the particular feature. For the Jensen expression below, λ weights are nonnegative, sum to 1, and the logged arguments are positive. For a Gaussian, σ>0. GMM weights πk are nonnegative and sum to 1; γik or λki are responsibilities, not the mixing weights themselves. Worked explanations: [Naive Bayes](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-nb--model/) · [CPTs and Bayesian networks](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-bn--semantics/) · [Mean, variance and likelihood](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-gmm--centre-spread/) · [EM update](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u1-gmm--em/). | | | | --- | --- | | Conditional | $$ P(a\mid b)=P(a\land b)/P(b) $$ | | Product rule | $$ P(a\land b)=P(a\mid b)P(b) $$ | | Marginalisation | $$ P(Y)=\sum_zP(Y,z) $$ | | Normalisation | $$ \mathbf P(X\mid e)=\alpha\,\mathbf P(X,e),\;\alpha=1/\sum_xP(x,e) $$ | | Bayes' rule | $$ P(A\mid B)=\dfrac{P(B\mid A)P(A)}{P(B\mid A)P(A)+P(B\mid\lnot A)P(\lnot A)} $$ | | Naive Bayes | $$ v_{NB}=\arg\max_cP(c)\prod_iP(a_i\mid c) $$ ; Laplace: $$ \frac{n_{ic}+1}{N_c+k} $$ | | BN chain rule | $$ P(x_1..x_n)=\prod_iP(x_i\mid parents(X_i)) $$ | | BN query | $$ P(X\mid e)=\alpha\sum_{\text{hidden}}\prod P(\cdot\mid parents) $$ | | CPT size | Boolean node with k Boolean parents: independent probability parameters = $$ 2^k $$ numbers | | GMM density | $$ p(x)=\sum_k\pi_k\mathcal N(x\mid\mu_k,\sigma_k^2) $$ | | Gaussian | $$ \mathcal N(x\mid\mu,\sigma^2)=\frac1{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$ | | Log-likelihood | $$ \log L=\sum_i\log\sum_k\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2) $$ | | Jensen (log concave) | $$ \log\sum_k\lambda_ka_k\ge\sum_k\lambda_k\log a_k $$ | | E-step | $$ \lambda_k^i=\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)/\sum_l\pi_l\mathcal N(x_i\mid\mu_l,\sigma_l^2) $$ | | M-step | $$ \mu_k=\frac{\sum\lambda_k^ix_i}{\sum\lambda_k^i},\;\sigma_k^2=\frac{\sum\lambda_k^i(x_i-\mu_k)^2}{\sum\lambda_k^i},\;\pi_k=\frac{\sum_i\lambda_k^i}{n} $$ | | GMM parameters | 1-D: $$ 3K-1 $$ (K = 5 → 14) | --- ## TEE · Unit 4 formulas: Markov chains, MRP, MDP, Bellman Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/formulas/tee-formulas--unit4/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/formulas/tee-formulas/unit4 Probability, Bayes, GMM/EM, Markov chains and Bellman equations (Units 2–3 are in the M2 sheet) ## Read these symbols in context P is a transition matrix with origin rows and destination columns. π denotes a stationary distribution in the first equations, but a policy in the MDP equations. s′ is a next state; a′ a next action. R is the specified immediate or expected reward; γ discounts future rewards. E means expectation (probability-weighted average). I is the identity matrix. The two-state stationary shortcut needs a+b>0. The inverse (I−γP)⁻¹ exists for a finite stochastic P with 0≤γ<1; at γ=1, check the problem’s terminal structure instead of assuming the full matrix is invertible. The continuing discounted-return examples use γ<1. A finite episode can also use γ=1. Q-learning is supplementary here; the explicitly listed course focus is the Markov/MDP and Bellman material. α in Q-learning is a learning rate; α in Bayes normalisation was a different constant. Worked explanations: [Matrix multiplication](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-markov--next-step/) · [Stationary algebra](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-markov--steady-state/) · [Discounted return](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--mrp-return/) · [Bellman equations](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--bellman-equations/). | | | | --- | --- | | Markov property | $$ P(S_{t+1}\mid S_t,\dots,S_1)=P(S_{t+1}\mid S_t) $$ | | Transition matrix | $$ P_{ij}=P(\text{next}=j\mid\text{now}=i) $$ ; rows sum to 1 | | Next step | $$ x_{t+1}=x_tP $$ (row vector); $$ x_{t+n}=x_tP^n $$ | | Steady state | $$ \pi=\pi P,\;\sum\pi_i=1 $$ ; two states $$ \begin{bmatrix}1-a&a\\b&1-b\end{bmatrix}\Rightarrow\pi=\left[\frac b{a+b},\frac a{a+b}\right] $$ | | Return | $$ G_t=\sum_{k\ge0}\gamma^kR_{t+k+1} $$ | | MRP Bellman | $$ v(s)=R(s)+\gamma\sum_{s'}P_{ss'}v(s') $$ ; $$ v=(I-\gamma P)^{-1}R $$ | | Policy | $$ \pi(a\mid s)=P(A_t=a\mid S_t=s) $$ | | Value functions | $$ v_\pi(s)=\mathbb E_\pi[G_t\mid s] $$ ; $$ q_\pi(s,a)=\mathbb E_\pi[G_t\mid s,a] $$ ; $$ v_\pi=\sum_a\pi(a\mid s)q_\pi(s,a) $$ | | Bellman expectation | $$ v_\pi(s)=\sum_a\pi(a\mid s)\sum_{s'}P(s'\mid s,a)[R+\gamma v_\pi(s')] $$ | | Bellman optimality | $$ v_*(s)=\max_a\sum_{s'}P(s'\mid s,a)[R+\gamma v_*(s')] $$ ; $$ q_*(s,a)=\sum_{s'}P(s'\mid s,a)[R+\gamma\max_{a'}q_*(s',a')] $$ | Optional further reading: the [Q-learning extension](https://deep-learning-study-room.vercel.app/read/is2/tee/read/u4-bellman--solving/) is kept with the Bellman lesson, outside this core formula list. --- ## TEE · TEE-only formulas: ACO and GA for TSP Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/formulas/tee-formulas--aco-tsp/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/formulas/tee-formulas/aco-tsp Probability, Bayes, GMM/EM, Markov chains and Bellman equations (Units 2–3 are in the M2 sheet) ## Ant colony optimisation | | | | --- | --- | | Transition probability | $$ P_{ij}=\dfrac{\tau_{ij}^\alpha\eta_{ij}^\beta}{\sum_{l\in\text{allowed}}\tau_{il}^\alpha\eta_{il}^\beta},\;\eta_{ij}=1/d_{ij} $$ | | Choice rule | cumulative probabilities in **city order**; first city with cumulative ≥ r | | Pheromone update | $$ \tau_{ij}\leftarrow(1-\rho)\tau_{ij}+\sum_{k:\,ij\in tour_k} Q/L_k $$ on every edge ant k used (return edge included); symmetric only for an undirected symmetric TSP | | Values assumed in the worked PYQs | $$ \alpha=\beta=1,\;\rho=0.5,\;Q=1,\;\tau_0=1 $$ | α weights pheromone influence; β weights inverse-distance influence; ρ is the evaporation rate; Q is the deposit constant; L is the complete tour length. These must be supplied or explicitly assumed. Do not treat the example values as universal defaults. $$ L=\sum_{i=1}^{n-1}d(c_i,c_{i+1})+d(c_n,c_1) $$ $$ F_{TSP}=1/L $$ The return edge is part of the tour. Permutation crossover/mutation must keep each city exactly once. For positive lengths, shorter length gives larger reciprocal fitness. --- ## TEE · Q1(a) Joint probability table [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q1a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q1a Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. A supermarket recorded the joint distribution of Product type (Y: premium = 1, regular = 0), Payment mode (X: credit = 1, debit = 0) and Gender (Z: female = 1, male = 0). (a) If a customer bought a premium item, find the probability that (1) they are female, (2) they paid by credit card. (b) Find (1) P(premium) and (2) P(credit card). (c) Given a male customer paid by debit card, find the probability he bought a regular item.** [5 marks] ### Answer #### Given The printed eight entries are reproduced below. Their total is 1.002, not 1. Therefore the table is inconsistent as a probability distribution. | | Credit, female | Credit, male | Debit, female | Debit, male | | --- | --- | --- | --- | --- | | **Premium (Y=1)** | 0.08 | 0.10 | 0.072 | 0.05 | | **Regular (Y=0)** | 0.10 | 0.15 | 0.25 | 0.20 | #### To find The five requested probabilities, stating how the inconsistent data are interpreted. #### Formula and why it applies For this worked solution, treat all printed entries as nonnegative weights and normalise by their total 1.002. This preserves the printed ratios. It is a declared interpretation, not a claim about the examiner’s intended correction. $$ P(E)=\frac{\text{sum of weights for }E}{\text{sum of all weights}} $$ $$ P(A\mid E)=\frac{\text{sum of weights for }A\cap E}{\text{sum of weights for }E} $$ #### Solution ##### Step 1. Validate the full table $$ 0.08+0.10+0.072+0.05+0.10+0.15+0.25+0.20=1.002 $$ Use total weight 1.002 consistently for unconditional probabilities. ##### Step 2. Premium row total $$ w(premium)=0.08+0.10+0.072+0.05=0.302 $$ ##### Step 3. Female given premium $$ P(female\mid premium)=\frac{0.08+0.072}{0.302}=\frac{0.152}{0.302}\approx0.503311 $$ ##### Step 4. Credit given premium $$ P(credit\mid premium)=\frac{0.08+0.10}{0.302}=\frac{0.18}{0.302}\approx0.596026 $$ ##### Step 5. Normalised premium marginal $$ P(premium)=\frac{0.302}{1.002}\approx0.301397 $$ ##### Step 6. Normalised credit marginal $$ P(credit)=\frac{0.08+0.10+0.10+0.15}{1.002}=\frac{0.43}{1.002}\approx0.429142 $$ ##### Step 7. Regular given male and debit $$ P(regular\mid male,debit)=\frac{0.20}{0.05+0.20}=0.8 $$ **Final answer.** Under common normalisation: (a) 0.503311 and 0.596026; (b) 0.301397 and 0.429142; (c) 0.8. **Alternative correction:** if 0.072 is corrected to 0.070, the total becomes 1 and the corresponding answers are (a) 0.50 and 0.60; (b) 0.30 and 0.43; (c) 0.8. This agrees with the interpretation used in the supplied key. State the correction; do not call raw totals 0.302 and 0.43 valid marginals of the original inconsistent table. --- ## TEE · Q1(b) Applications of ACO [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q1b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q1b Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **List the applications of Ant Colony Optimization and explain any one application in detail.** [5 marks] ### Answer ### Definition (1 line) Ant Colony Optimization is a probabilistic, population-based metaheuristic (Dorigo, 1992) in which artificial ants build solutions on a graph and communicate through pheromone trails. In the update rule used here, evaporation reduces all stored trails; deposits then reinforce edges of selected tours according to solution quality. Evaporation does not occur only on poor paths. ### Applications 1. **Travelling salesman problem**: shortest tour visiting every city once. 2. **Vehicle routing**: delivery routes for a fleet with capacity and time windows (minimising distance, time, fuel). 3. **Network routing**: adaptive routing of data packets in communication and sensor networks (AntNet). 4. **Job-shop and project scheduling**: assigning jobs to machines to minimise completion time. 5. **Image processing**: edge detection, with ants moving over pixel-intensity graphs. 6. **Assignment problems**: quadratic assignment, timetabling, graph colouring. 7. **Feature selection and data mining**: Ant-Miner for classification rules. 8. **Load balancing** in distributed systems and cloud task allocation. ### One application in detail: vehicle routing (or TSP) 1. **Graph.** Depot and customers are nodes; travel times or distances are edge weights. 2. **Ants.** Each ant builds a complete set of routes. From its current customer it chooses the next with probability $$ \propto\tau_{ij}\,(1/d_{ij}) $$ , skipping customers already served or that would exceed the vehicle capacity; when the vehicle is full it returns to the depot. 3. **Evaluation.** Total distance (plus penalties for late deliveries). 4. **Learning.** Pheromone evaporates everywhere and is added in proportion to $$ Q/L $$ on the routes of good solutions, so short connections are reinforced. 5. **Result.** Repeated iterations search for low-cost routes; finite-run convergence to the optimum is not guaranteed. Companies use this for parcel delivery, waste collection and school-bus routing. ### Advantages and limitations | Advantages | Limitations | | --- | --- | | Naturally suited to graph and routing problems | Several parameters to tune (α, β, ρ, number of ants) | | Can share useful route information and adapt search to changing graphs | Convergence time is uncertain; may stagnate on a sub-optimal path | | Positive feedback + parallel ants | Theoretical analysis is difficult | **Yaad rakho:** ACO = “raaste wale problems” (TSP, routing, scheduling). PSO = “numbers wale problems” (continuous function, weights). --- ## TEE · Q1(c) Fuzzy set operations [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q1c/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q1c Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Consider two fuzzy sets X = {(c1, 0.6), (c2, 0.8), (c3, 0.3), (c4, 0.5), (c5, 0.7)} and Y = {(c1, 0.4), (c2, 0.5), (c3, 0.9), (c4, 0.6), (c5, 0.2)}. Find 1) X ∪ Y 2) X ∩ Y 3) ¬X ∪ ¬Y 4) ¬X ∩ ¬Y.** [5 marks] ### Answer #### Given The aligned X and Y memberships printed in this question. #### To find The four requested membership lists in the original element order. #### Formula and why it applies Use max for OR, min for AND, and 1−μ for NOT. $$ \mu_{X\cup Y}=\max(\mu_X,\mu_Y),\quad\mu_{X\cap Y}=\min(\mu_X,\mu_Y) $$ Complement both operands before applying the operator in ¬X∪¬Y or ¬X∩¬Y. #### Solution ##### Step 1. Complements ¬X = {0.4, 0.2, 0.7, 0.5, 0.3}; ¬Y = {0.6, 0.5, 0.1, 0.4, 0.8}. ##### Step 2. Table | | c1 | c2 | c3 | c4 | c5 | | --- | --- | --- | --- | --- | --- | | X | 0.6 | 0.8 | 0.3 | 0.5 | 0.7 | | Y | 0.4 | 0.5 | 0.9 | 0.6 | 0.2 | | **X ∪ Y = max** | 0.6 | 0.8 | 0.9 | 0.6 | 0.7 | | **X ∩ Y = min** | 0.4 | 0.5 | 0.3 | 0.5 | 0.2 | | **¬X ∪ ¬Y** | 0.6 | 0.5 | 0.7 | 0.5 | 0.8 | | **¬X ∩ ¬Y** | 0.4 | 0.2 | 0.1 | 0.4 | 0.3 | ##### Step 3. Check with De Morgan's law ¬X ∪ ¬Y must equal 1 − (X ∩ Y) = {0.6, 0.5, 0.7, 0.5, 0.8} ✓, and ¬X ∩ ¬Y must equal 1 − (X ∪ Y) = {0.4, 0.2, 0.1, 0.4, 0.3} ✓. **Final answer.** X ∪ Y = {(c1,0.6),(c2,0.8),(c3,0.9),(c4,0.6),(c5,0.7)}; X ∩ Y = {(c1,0.4),(c2,0.5),(c3,0.3),(c4,0.5),(c5,0.2)}; ¬X ∪ ¬Y = {(c1,0.6),(c2,0.5),(c3,0.7),(c4,0.5),(c5,0.8)}; ¬X ∩ ¬Y = {(c1,0.4),(c2,0.2),(c3,0.1),(c4,0.4),(c5,0.3)}. Official marking: 1 + 1 + 1.5 + 1.5. ## Calculate every element separately ### Element c1 $$ \neg X:1-0.6=0.4\qquad\neg Y:1-0.4=0.6 $$ $$ X\cup Y:\max(0.6,0.4)=0.6 $$ $$ X\cap Y:\min(0.6,0.4)=0.4 $$ $$ \neg X\cup\neg Y:\max(0.4,0.6)=0.6 $$ $$ \neg X\cap\neg Y:\min(0.4,0.6)=0.4 $$ ### Element c2 $$ \neg X:1-0.8=0.2\qquad\neg Y:1-0.5=0.5 $$ $$ X\cup Y:\max(0.8,0.5)=0.8 $$ $$ X\cap Y:\min(0.8,0.5)=0.5 $$ $$ \neg X\cup\neg Y:\max(0.2,0.5)=0.5 $$ $$ \neg X\cap\neg Y:\min(0.2,0.5)=0.2 $$ ### Element c3 $$ \neg X:1-0.3=0.7\qquad\neg Y:1-0.9=0.1 $$ $$ X\cup Y:\max(0.3,0.9)=0.9 $$ $$ X\cap Y:\min(0.3,0.9)=0.3 $$ $$ \neg X\cup\neg Y:\max(0.7,0.1)=0.7 $$ $$ \neg X\cap\neg Y:\min(0.7,0.1)=0.1 $$ ### Element c4 $$ \neg X:1-0.5=0.5\qquad\neg Y:1-0.6=0.4 $$ $$ X\cup Y:\max(0.5,0.6)=0.6 $$ $$ X\cap Y:\min(0.5,0.6)=0.5 $$ $$ \neg X\cup\neg Y:\max(0.5,0.4)=0.5 $$ $$ \neg X\cap\neg Y:\min(0.5,0.4)=0.4 $$ ### Element c5 $$ \neg X:1-0.7=0.3\qquad\neg Y:1-0.2=0.8 $$ $$ X\cup Y:\max(0.7,0.2)=0.7 $$ $$ X\cap Y:\min(0.7,0.2)=0.2 $$ $$ \neg X\cup\neg Y:\max(0.3,0.8)=0.8 $$ $$ \neg X\cap\neg Y:\min(0.3,0.8)=0.3 $$ --- ## TEE · Q1(d) Brand-switching Markov chain [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q1d/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q1d Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Users switch between Brand X and Brand Y each year: 25% of X users switch to Y and 75% stay; 40% of Y users switch to X and 60% stay. (a) Write the transition probability matrix. (b) Draw the state transition diagram.** [5 marks] ### Answer #### Given All transition values are given in the full question. State order: X, Y. #### To find The transition matrix and state diagram. #### Formula and why it applies Rows mean current state; columns mean next state. For each destination, multiply each origin amount by its transition probability, then add. #### Solution ##### Step 1. Convert every origin row to probabilities | From / To | X | Y | | --- | --- | --- | | X | 0.75 | 0.25 | | Y | 0.4 | 0.6 | Each row sums to 1 because it lists all possible destinations for that origin. A percentage is divided by 100: 25%=.25. ##### Step 2. Interpret entries and draw the transitions Diagonal entries are the fractions staying at their current state; off-diagonal entries are fractions moving to a different state. Each arrow uses its origin row and destination column. These fractions have different origin populations, so adding raw column probabilities does not give the number arriving. Swipe across the diagram to read its labels ↔ ![Each arrow is labelled with the probability of that transition.](https://deep-learning-study-room.vercel.app/is2/fig/mk_brand.c2c4be3e6a.svg) Diagram: Each arrow is labelled with the probability of that transition.. Labels: Brand switching each year | 0.25 | 0.4 | 0.75 | 0.6 | X | Y Each arrow is labelled with the probability of that transition. **Final answer.** The transition matrix and state diagram are given above. --- ## TEE · Q2(a) PSO, one iteration [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q2a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q2a Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Maximize $$ F(x_1,x_2)=2x_1^2-3x_1x_2+x_2^2+5x_1+2x_2+7 $$ using Particle Swarm Optimization. $$ X=\begin{bmatrix}2.5&-3.0\\3.5&-1.5\end{bmatrix},\;V=\begin{bmatrix}0.2&0.3\\0.1&0.25\end{bmatrix} $$ . Take $$ W=0.9,\;c_1=c_2=1.5,\;r_1=0.5949,\;r_2=0.0855 $$ . Optimize and give the solution after 1 iteration showing the step-by-step process.** [10 marks] ### Answer #### Given Particle 1: $$ x=(2.5,\,-3.0) $$ , $$ v=(0.2,\,0.3) $$ . Particle 2: $$ x=(3.5,\,-1.5) $$ , $$ v=(0.1,\,0.25) $$ . $$ c_1r_1=1.5\times0.5949=0.89235,\qquad c_2r_2=1.5\times0.0855=0.12825 $$ #### To find Fitness, pbest, gbest; new velocities and positions; updated pbest and gbest. #### Formula and why it applies $$ v_{ij}\leftarrow wv_{ij}+c_1r_1(p^b_{ij}-x_{ij})+c_2r_2(p^g_j-x_{ij}),\qquad x_{ij}\leftarrow x_{ij}+v_{ij} $$ Each particle is pulled by its own memory and by the swarm's best; we maximise, so the larger $$ F $$ is better. #### Solution ##### Step 1. Fitness of particle 1 $$ F(2.5,-3)=2(6.25)-3(2.5)(-3)+9+5(2.5)+2(-3)+7=12.5+22.5+9+12.5-6+7=57.5 $$ ##### Step 2. Fitness of particle 2 $$ F(3.5,-1.5)=2(12.25)-3(3.5)(-1.5)+2.25+17.5-3+7=24.5+15.75+2.25+17.5-3+7=64 $$ ##### Step 3. pbest and gbest | Particle | Position | F | pbest | | --- | --- | --- | --- | | 1 | (2.5, −3.0) | 57.5 | (2.5, −3.0) | | 2 | (3.5, −1.5) | 64 | (3.5, −1.5) | No previous iteration, so pbest = current position. Largest fitness is 64, so **gbest = (3.5, −1.5)**. ##### Step 4. Velocity and position of particle 1 $$ v_{11}=0.9(0.2)+0.89235(2.5-2.5)+0.12825(3.5-2.5)=0.18+0+0.12825=0.30825 $$ $$ x_{11}=2.5+0.30825=2.80825 $$ $$ v_{12}=0.9(0.3)+0.89235(0)+0.12825(-1.5-(-3.0))=0.27+0.192375=0.462375 $$ $$ x_{12}=-3.0+0.462375=-2.537625 $$ ##### Step 5. Velocity and position of particle 2 (it is gbest, so both pull terms are 0) $$ v_{21}=0.9(0.1)+0+0=0.09,\qquad x_{21}=3.5+0.09=3.59 $$ $$ v_{22}=0.9(0.25)+0+0=0.225,\qquad x_{22}=-1.5+0.225=-1.275 $$ ##### Step 6. New fitness $$ F(2.80825,-2.537625)=15.772536+21.378856+6.439541+14.04125-5.07525+7=59.5569 $$ $$ F(3.59,-1.275)=25.7762+13.73175+1.625625+17.95-2.55+7=63.5336 $$ ##### Step 7. Update pbest and gbest | Particle | New position | New F | Old pbest F | pbest after iteration | | --- | --- | --- | --- | --- | | 1 | (2.80825, −2.537625) | 59.5569 | 57.5 | (2.80825, −2.537625) — improved | | 2 | (3.59, −1.275) | 63.5336 | 64 | (3.5, −1.5) — kept | The best pbest fitness is still 64, so gbest stays (3.5, −1.5). **Final answer.** $$ V=\begin{bmatrix}0.30825&0.462375\\0.09&0.225\end{bmatrix} $$ , $$ X=\begin{bmatrix}2.80825&-2.537625\\3.59&-1.275\end{bmatrix} $$ . After one iteration particle 1 improved to F = 59.5569; the best solution so far is gbest = (3.5, −1.5) with F = 64. Official key: fitness and gbest (5 marks), velocity and position updates (5 marks). The calculations above use the question data. The key's summary matrix writes 0.462 in place of 2.808 for $$ x_{11} $$ ; that is a copying slip. --- ## TEE · Q2(b) GMM: compare with k-means, log-likelihood, parameters [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q2b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q2b Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **With reference to the Gaussian Mixture Model (GMM): compare GMM with the k-means algorithm; derive the equation for the log-likelihood function; state the number of parameters to be computed if K = 5.** [10 marks] ### Answer ## Read the derivation’s symbols first “i.i.d.” means the observations are modelled as independent draws from the same distribution. The product Π runs over observations; the inner sum Σ runs over components. λki in some notes is the same responsibility that other sections call γik. A superscript i here labels the observation, not an exponent. Jensen’s inequality gives a lower bound on the log of the component sum. EM chooses responsibilities making that bound touch the current likelihood, then updates the parameters to maximise the bound with those responsibilities fixed. This explains the two alternating steps; λ values are not arbitrary new model parameters. | Parameter | Number in a 1D K-component GMM | Why | | --- | --- | --- | | Means | K | One centre for each component | | Variances | K | One spread for each component | | Independent mixing weights | K−1 | The last weight is 1 minus the others | | Total | 3K−1 | K+K+(K−1) | For K=2, choose μ₁,μ₂,σ₁²,σ₂² and π₁: five free numbers. Then π₂=1−π₁. For K=5, 5+5+4=14. In 2D with a full covariance matrix, each component instead has two means and three independent covariance entries (two variances and one shared off-diagonal covariance). Five such components have 5×(2+3)+4=29 free parameters. ### 1. GMM versus k-means | Aspect | k-means | GMM | | --- | --- | --- | | Assignment | Hard: each point in exactly one cluster | Soft: probability $$ \lambda_k^i $$ of belonging to each cluster | | Model | Geometric: nearest centroid | Probabilistic generative model $$ \sum_k\pi_k\mathcal N(\mu_k,\Sigma_k) $$ | | Cluster shape | Spherical, similar size | Elliptical, any size/orientation; overlapping clusters | | Parameters | K centroids | Weights, means, covariances (1-D: 3K − 1) | | Algorithm / objective | Assign–update; minimise squared distances | EM; maximise log-likelihood | | Output | Hard labels and fitted centroids | Labels, membership probabilities and a density (useful for anomaly detection) | | Relation | The hard-assignment, equal spherical variance, small-variance limit of a Gaussian-mixture objective leads to the k-means objective. Ordinary finite-variance EM is not the same algorithm as k-means. Both need K in advance and depend on initialisation. | | ### 2. Derivation of the log-likelihood **Step 1. Model.** Each point is generated by choosing component k with probability $$ \pi_k $$ and drawing $$ x_i\sim\mathcal N(\mu_k,\sigma_k^2) $$ , so its density is $$ f_{mix}(x_i)=\sum_k\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2) $$ . **Step 2. Likelihood.** Points are i.i.d., so the likelihood is a product (θ = all parameters): $$ L(\theta)=\prod_{i=1}^{n}\Big[\sum_{k=1}^{K}\pi_k\frac{1}{\sqrt{2\pi}\,\sigma_k}e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}}\Big] $$ **Step 3. Log-likelihood.** Logs turn the product into a sum: $$ \log L(\theta)=\sum_{i=1}^{n}\log\Big(\sum_{k=1}^{K}\pi_k\frac{1}{\sqrt{2\pi}\,\sigma_k}e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}}\Big) $$ **Step 4. Why it cannot be solved directly.** The sum sits inside the log, so setting derivatives to zero gives no closed-form solution. **Step 5. Jensen's inequality.** log is concave, so for weights $$ \lambda_k\ge0,\;\sum\lambda_k=1 $$ : $$ \log\big(\sum_k\lambda_ka_k\big)\ge\sum_k\lambda_k\log a_k $$ . Introduce $$ \lambda_k^i $$ for every point ( $$ \sum_k\lambda_k^i=1 $$ ), multiply and divide inside the log: $$ \log L(\theta)=\sum_i\log\Big(\sum_k\lambda_k^i\,\frac{\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)}{\lambda_k^i}\Big)\;\ge\;\sum_i\sum_k\lambda_k^i\log\frac{\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)}{\lambda_k^i} $$ **Step 6. Estimation (EM).** Maximise the lower bound alternately. Fixing θ gives the E-step $$ \lambda_k^i=\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)/\sum_l\pi_l\mathcal N(x_i\mid\mu_l,\sigma_l^2) $$ . Fixing λ and differentiating (with a Lagrange multiplier for $$ \sum\pi_k=1 $$ ) gives the M-step: $$ \hat\mu_k=\frac{\sum_i\lambda_k^ix_i}{\sum_i\lambda_k^i},\qquad\hat\sigma_k^2=\frac{\sum_i\lambda_k^i(x_i-\hat\mu_k)^2}{\sum_i\lambda_k^i},\qquad\hat\pi_k=\frac{\sum_i\lambda_k^i}{n} $$ ### 3. Number of parameters for K = 5 The dimension and covariance structure must also be specified. For the one-dimensional, independent-variance model used in the derivation: $$ 2K+(K-1)=3K-1=3(5)-1=\mathbf{14} $$ (5 means + 5 variances + 4 free weights, since the 5 weights must add to 1.) In two dimensions with full covariance it would be 29. **Slide note.** One slide puts the convex inequality under the “concave” heading. Correct: log is *concave*, so $$ f(\sum\lambda a)\ge\sum\lambda f(a) $$ (≥, not >). --- ## TEE · Q3(a) Bayesian network: wet grass given no rain [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q3a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q3a Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Cloudy → Sprinkler, Cloudy → Rain, Sprinkler and Rain → Wet Grass. P(Cloudy) = 0.5; P(Sprinkler | C) = 0.10, P(Sprinkler | ¬C) = 0.50; P(Rain | C) = 0.80, P(Rain | ¬C) = 0.20; P(Wet | S, R) = 0.99, P(Wet | S, ¬R) = 0.90, P(Wet | ¬S, R) = 0.90, P(Wet | ¬S, ¬R) = 0.01. Calculate the probability of the grass being wet given it doesn't rain.** [10 marks] ### Answer The following table enumerates the values of Cloudy and Sprinkler that were not fixed by the question. Two Boolean choices give four rows: true/true, true/false, false/true, false/false. Each row is one mutually exclusive complete case. Read the CPT entry using that row’s parent values, multiply its factors, and add the resulting row weights. The denominator must include all cases matching the evidence, including both possible values of the queried outcome. #### Given C→S, C→R, (S,R)→W. Root and conditional probabilities are listed in the question. Evidence R=false. #### To find P(W=true | R=false). Hidden variables C and S must be summed out. #### Formula and why it applies Use the network joint and divide by the evidence probability. $$ P(W\mid\neg R)=\frac{\sum_{c,s}P(c)P(s\mid c)P(\neg R\mid c)P(W\mid s,\neg R)}{\sum_cP(c)P(\neg R\mid c)} $$ #### Solution Swipe across the diagram to read its labels ↔ ![Final 2025 Q3a: sprinkler network](https://deep-learning-study-room.vercel.app/is2/fig/bn_sprinkler.a251583272.svg) Diagram: Final 2025 Q3a: sprinkler network. Labels: Final 2025 Q3a: sprinkler network | Cloudy | P(C) = 0.5 | Sprinkler | P(S|C) = 0.1, P(S|¬C) = 0.5 | Rain | P(R|C) = 0.8, P(R|¬C) = 0.2 | Wet grass | P(W|S,R): .99 · .90 · .90 · .01 | (TT · TF · FT · FF) ##### Step 1. Joint by the chain rule $$ P(C,S,R,W)=P(C)\,P(S\mid C)\,P(R\mid C)\,P(W\mid S,R) $$ ##### Step 2. Sum over C and S with R = false, W = true | C | S | P(C) | P(S|C) | P(¬R|C) | P(W|S,¬R) | Product | | --- | --- | --- | --- | --- | --- | --- | | T | T | 0.5 | 0.1 | 0.2 | 0.90 | 0.0090 | | T | F | 0.5 | 0.9 | 0.2 | 0.01 | 0.0009 | | F | T | 0.5 | 0.5 | 0.8 | 0.90 | 0.1800 | | F | F | 0.5 | 0.5 | 0.8 | 0.01 | 0.0020 | | **P(W, ¬R)** | | | | | | **0.1919** | ##### Step 3. Evidence $$ P(\lnot R)=P(C)P(\lnot R\mid C)+P(\lnot C)P(\lnot R\mid\lnot C)=0.5(0.2)+0.5(0.8)=0.5 $$ ##### Step 4. Divide $$ P(W\mid\lnot R)=\frac{0.1919}{0.5}=0.3838 $$ **Final answer.** P(grass wet | no rain) ≈ **0.384**. Without rain, the grass is wet mainly when the sprinkler runs (more likely on non-cloudy days). **Keep the requested conditional probability separate.** The supplied key discusses P(Wet) and P(Rain | Wet), whereas this question asks P(Wet | no rain). The answer to the printed question is 0.3838. For comparison, the exact marginal from the printed CPT is calculated below; it is not 0.6505. #### Optional comparison: calculate P(Wet) from the same network The four no-rain cases already total 0.1919. Now add the four rain cases, keeping the same Cloudy/Sprinkler order. $$ P(W,R)=0.5(0.1)(0.8)(0.99)+0.5(0.9)(0.8)(0.90)+0.5(0.5)(0.2)(0.99)+0.5(0.5)(0.2)(0.90) $$ $$ P(W,R)=0.0396+0.324+0.0495+0.045=0.4581 $$ $$ P(W)=P(W,R)+P(W,\neg R)=0.4581+0.1919=0.6500 $$ $$ P(R\mid W)=\frac{0.4581}{0.6500}\approx0.704769231 $$ **Meaning:** the unconditional chance of Wet is 0.65, the chance of Rain among wet cases is about 0.704769, and the requested chance of Wet among no-rain cases is 0.3838. These are three different questions. **Trap.** Do not plug in the marginal P(Sprinkler) = 0.3; sprinkler and rain both depend on Cloudy, so you must sum over Cloudy (given no rain, P(S) = 0.42). --- ## TEE · Q3(b) Fuzzy loan-approval system [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q3b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q3b Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **A bank uses fuzzy decision support with Credit Score and Income Stability, both on 1–100. Loan Approval is a score on 1–100 with labels Reject, Consider and Approve. Define triangular membership functions, construct nine rules, explain Mamdani inference and defuzzify for Credit Score=55 and Income Stability=35.** [10 marks] ### Answer #### Given | Variable | Crisp value | | --- | --- | | Credit Score | 55 | | Income Stability | 35 | The question asks us to design the membership functions. This solution chooses Low=Tri(1,25,50), Medium=Tri(25,50,75), High=Tri(50,75,100) for both inputs. Output labels use the same triangles in their respective order. These are explicit design choices, not additional printed givens. #### To find A complete stated design and its continuous Mamdani centroid. #### Formula and why it applies Use min for AND and implication, max for aggregation and the true area centroid of the resulting shape. A peak-only weighted average is a different defuzzifier. $$ z^*=\frac{\int z\mu_{out}(z)\,dz}{\int\mu_{out}(z)\,dz} $$ #### Solution ##### Step 1. Specify the membership functions | Input / output position | Triangle | Peak | | --- | --- | --- | | Low / Reject | (1,25,50) | 25 | | Medium / Consider | (25,50,75) | 50 | | High / Approve | (50,75,100) | 75 | Use 0 outside each triangle, the rising fraction before the peak and the falling fraction after it. This design is evaluated at the requested input; a deployed controller would also need coverage/fallback at all boundary values. ##### Step 2. Write all nine rules | Credit Score / Income Stability | Low | Medium | High | | --- | --- | --- | --- | | Low | Reject | Reject | Consider | | Medium | Reject | Consider | Approve | | High | Consider | Approve | Approve | Read each cell as an IF–AND–THEN rule. For example, the centre cell combines Medium with Medium. ##### Step 3. Fuzzify Credit Score = 55 $$ \mu_{Low}(55)=0 $$ $$ \mu_{Medium}(55)=\frac{75-55}{75-50}=0.8 $$ $$ \mu_{High}(55)=\frac{55-50}{75-50}=0.2 $$ ##### Step 4. Fuzzify Income Stability = 35 $$ \mu_{Low}(35)=\frac{50-35}{50-25}=0.6 $$ $$ \mu_{Medium}(35)=\frac{35-25}{50-25}=0.4 $$ $$ \mu_{High}(35)=0 $$ ##### Step 5. Evaluate all rules and aggregate matching consequents | Credit Score | Income Stability | Firing strength | Consequent | | --- | --- | --- | --- | | Low | Low | min(0,0.6) = 0 | Reject | | Low | Medium | min(0,0.4) = 0 | Reject | | Low | High | min(0,0) = 0 | Consider | | Medium | Low | min(0.8,0.6) = 0.6 | Reject | | Medium | Medium | min(0.8,0.4) = 0.4 | Consider | | Medium | High | min(0.8,0) = 0 | Approve | | High | Low | min(0.2,0.6) = 0.2 | Consider | | High | Medium | min(0.2,0.4) = 0.2 | Approve | | High | High | min(0.2,0) = 0 | Approve | Aggregated label strengths: Reject = 0.6, Consider = 0.4, Approve = 0.2. $$ \mu_{out}(z)=\max_r\min(\alpha_r,\mu_{C_r}(z)) $$ ##### Step 6. Draw the aggregated output Swipe across the diagram to read its labels ↔ ![The shaded upper envelope counts any overlap only once.](https://deep-learning-study-room.vercel.app/is2/fig/v2_loan_aggregate.2c5ffa32a7.svg) Diagram: The shaded upper envelope counts any overlap only once.. Labels: Chosen triangular output sets and their max-aggregate | 0 | 0.5 | 1 | 1 | 25 | 50 | 75 | 100 | μ | Output score | Reject | Consider | Approve | centroid ≈ 42.78 The shaded upper envelope counts any overlap only once. ##### Step 7. Calculate area and moment of each straight-line strip For a strip from a to b with endpoint heights u and v, its trapezoid area is A. Its first moment M is the integral of z times the line height. $$ A=(b-a)(u+v)/2 $$ $$ M=\frac{b-a}{6}\big[(2a+b)u+(a+2b)v\big] $$ | a | b | u | v | Area A | Moment M | | --- | --- | --- | --- | --- | --- | | 1 | 15.4 | 0 | 0.6 | 4.32 | 45.792 | | 15.4 | 25 | 0.6 | 0.6 | 5.76 | 116.352 | | 25 | 35 | 0.6 | 0.6 | 6 | 180 | | 35 | 40 | 0.6 | 0.4 | 2.5 | 93.333333333 | | 40 | 50 | 0.4 | 0.4 | 4 | 180 | | 50 | 55 | 0.4 | 0.4 | 2 | 105 | | 55 | 65 | 0.4 | 0.4 | 4 | 240 | | 65 | 70 | 0.4 | 0.2 | 1.5 | 100.833333333 | | 70 | 75 | 0.2 | 0.2 | 1 | 72.5 | | 75 | 95 | 0.2 | 0.2 | 4 | 340 | | 95 | 100 | 0.2 | 0 | 0.5 | 48.333333333 | ##### Step 8. Divide total moment by total area $$ \sum A=35.58 $$ $$ \sum M=1522.144 $$ $$ z^*=\frac{1522.144}{35.58}\approx42.780888139 $$ **Final answer.** For these stated triangles and rules, the centroid is 42.780888139. This is a fuzzy suitability score; the category memberships can be evaluated at this score. The model alone is not a complete loan-decision policy. Because the paper asks for a design, a different explicitly specified membership/rule system can yield a different numerical output. Explain the choices and use the chosen defuzzification method consistently; no particular examiner mark outcome is promised. ## How the centroid table’s breakpoints are obtained For each output triangle, solve “line height = clip height”. Reject is clipped at .6: rising endpoint 1+.6(25−1)=15.4 and falling endpoint 50−.6(50−25)=35. Consider is clipped at .4: endpoints 25+.4(50−25)=35 and 75−.4(75−50)=65. Approve is clipped at .2: endpoints 50+.2(75−50)=55 and 100−.2(100−75)=95. Now take the maximum of the three clipped curves. The declining Reject edge reaches the Consider height .4 at (50−z)/25=.4, hence z=40. The declining Consider edge reaches the Approve height .2 at (75−z)/25=.2, hence z=70. These crossover locations must be included when splitting the aggregate. Original peaks and other clip endpoints may add extra splits without changing the total area. **For the first aggregate strip from a=1 to b=15.4, endpoint heights are u=0 and v=0.6. Find its area and first moment.** [3 marks · Worked example] #### Given Strip endpoints a=1 and b=15.4; left height u=0 and right height v=0.6. This is a straight rising edge of the clipped Reject set. #### To find A and M for one table row. #### Formula and why it applies $$ A=(b-a)(u+v)/2 $$ $$ M=\frac{b-a}{6}[(2a+b)u+(a+2b)v] $$ A is trapezoid area. M weights every small area by its horizontal position; this formula integrates a straight line. #### Solution ##### Step 1. Width and height sum $$ b-a=15.4-1=14.4\qquad u+v=0.6 $$ ##### Step 2. Area $$ A=14.4\times0.6/2=8.64/2=4.32 $$ ##### Step 3. First bracket term $$ (2a+b)u=(2+15.4)\times0=0 $$ ##### Step 4. Second bracket term $$ (a+2b)v=(1+30.8)\times0.6=31.8\times0.6=19.08 $$ ##### Step 5. Moment $$ M=(14.4/6)(0+19.08)=2.4\times19.08=45.792 $$ **Final answer.** Area=4.32; moment=45.792. Repeat the same arithmetic for each row, then sum the areas and moments separately. ### Interpret the loan result using an explicit decision rule Choose the output label having the greatest membership at the centroid; this is an additional stated interpretation rule. At z=42.780888139: $$ \mu_{Reject}=(50-z)/25\approx0.288764474,\quad\mu_{Consider}=(z-25)/25\approx0.711235526,\quad\mu_{Approve}=0 $$ The resulting recommendation is **Consider / manual review**; this design does not recommend automatic approval. If a binary approve/reject answer is required, the question must supply a threshold or you must state the threshold you adopt. A different decision threshold is separate from defuzzification. --- ## TEE · Q4(a) Cartesian product and max–min composition [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q4a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q4a Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Consider fuzzy sets A = {0.4/small + 0.3/medium + 0.1/large} and B = {0.2/slow + 0.7/moderate + 0.3/fast}. 1. Find R = A × B. 2. Given C = {0.3/small + 0.1/medium + 0.4/large}, find S = C × B. 3. Calculate the max–min composition S ∘ R.** [10 marks] ### Answer #### Given The fuzzy sets or relation matrices and their listed row/column order. #### To find The relation matrix/matrices and the composition specified in the question. #### Formula and why it applies For Cartesian products take min of row and column membership. For the ordered matrix composition shown here: $$ T_{ik}=\max_j\min(S_{ij},R_{jk}) $$ If max–product is asked instead, replace the inner min by multiplication. #### Solution ##### Step 1. R = A × B (rows small, medium, large; columns slow, moderate, fast) $$ R=\begin{bmatrix}\min(.4,.2)&\min(.4,.7)&\min(.4,.3)\\\min(.3,.2)&\min(.3,.7)&\min(.3,.3)\\\min(.1,.2)&\min(.1,.7)&\min(.1,.3)\end{bmatrix}=\begin{bmatrix}0.2&0.4&0.3\\0.2&0.3&0.3\\0.1&0.1&0.1\end{bmatrix} $$ ##### Step 2. S = C × B $$ S=\begin{bmatrix}0.2&0.3&0.3\\0.1&0.1&0.1\\0.2&0.4&0.3\end{bmatrix} $$ ##### Step 3. S ∘ R by max–min (row of S with column of R) The printed relation domains do not directly compose: both relate size to speed. For the numerical exercise, explicitly assume positional alignment and compute row of S with column of R. Equal dimensions alone are not a justification for semantic relation composition. $$ (1,1)=\max[\min(.2,.2),\min(.3,.2),\min(.3,.1)]=\max(.2,.2,.1)=0.2 $$ $$ (1,2)=\max[\min(.2,.4),\min(.3,.3),\min(.3,.1)]=\max(.2,.3,.1)=0.3 $$ $$ (1,3)=\max[\min(.2,.3),\min(.3,.3),\min(.3,.1)]=0.3 $$ Row 2 of S is all 0.1, so every entry of row 2 is 0.1. $$ (3,1)=\max[\min(.2,.2),\min(.4,.2),\min(.3,.1)]=0.2,\;(3,2)=\max[.2,.3,.1]=0.3,\;(3,3)=\max[.2,.3,.1]=0.3 $$ **Final answer.** $$ S\circ R=\begin{bmatrix}0.2&0.3&0.3\\0.1&0.1&0.1\\0.2&0.3&0.3\end{bmatrix} $$ . **Official key note.** The key first printed a wrong matrix with row 2 = (0.2, 0.3, 0.3), then corrected it by hand to (0.1, 0.1, 0.1). The corrected matrix above is right. Marking: R 3, S 3, composition 4. **A careful remark you can add.** Strictly, composition needs S: X → Y and R: Y → Z, while here both relate size to speed. The numerical matrix calculation above assumes positional identification of the three middle labels; that is a declared repair, not a valid size-to-speed relation composition without an extra mapping. ## Every cell without the row shortcuts Both printed relations run from size to speed; their middle labels do not match for genuine relation composition. The numerical working below follows the question’s intended row-of-S with column-of-R operation using explicitly assumed positional alignment of the three middle entries. ### R: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | R11 | 0.4 | 0.2 | 0.2 | | R12 | 0.4 | 0.7 | 0.4 | | R13 | 0.4 | 0.3 | 0.3 | | R21 | 0.3 | 0.2 | 0.2 | | R22 | 0.3 | 0.7 | 0.3 | | R23 | 0.3 | 0.3 | 0.3 | | R31 | 0.1 | 0.2 | 0.1 | | R32 | 0.1 | 0.7 | 0.1 | | R33 | 0.1 | 0.3 | 0.1 | ### S: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | S11 | 0.3 | 0.2 | 0.2 | | S12 | 0.3 | 0.7 | 0.3 | | S13 | 0.3 | 0.3 | 0.3 | | S21 | 0.1 | 0.2 | 0.1 | | S22 | 0.1 | 0.7 | 0.1 | | S23 | 0.1 | 0.3 | 0.1 | | S31 | 0.4 | 0.2 | 0.2 | | S32 | 0.4 | 0.7 | 0.4 | | S33 | 0.4 | 0.3 | 0.3 | **Result cell (1,1)** $$ \max\{\min(0.2,0.2),\min(0.3,0.2),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (1,2)** $$ \max\{\min(0.2,0.4),\min(0.3,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ **Result cell (1,3)** $$ \max\{\min(0.2,0.3),\min(0.3,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ **Result cell (2,1)** $$ \max\{\min(0.1,0.2),\min(0.1,0.2),\min(0.1,0.1)\} $$ $$ =\max\{0.1,0.1,0.1\}=0.1 $$ **Result cell (2,2)** $$ \max\{\min(0.1,0.4),\min(0.1,0.3),\min(0.1,0.1)\} $$ $$ =\max\{0.1,0.1,0.1\}=0.1 $$ **Result cell (2,3)** $$ \max\{\min(0.1,0.3),\min(0.1,0.3),\min(0.1,0.1)\} $$ $$ =\max\{0.1,0.1,0.1\}=0.1 $$ **Result cell (3,1)** $$ \max\{\min(0.2,0.2),\min(0.4,0.2),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (3,2)** $$ \max\{\min(0.2,0.4),\min(0.4,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ **Result cell (3,3)** $$ \max\{\min(0.2,0.3),\min(0.4,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.2,0.3,0.1\}=0.3 $$ --- ## TEE · Q4(b) Naive Bayes: flu [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q4b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q4b Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. A hospital wants to predict flu from Fever (Yes/No), Cough (Yes/No) and Temperature Level (Low/Normal/High). Training data: (Yes, Yes, High, Yes), (No, No, Normal, No), (Yes, No, Low, No), (Yes, Yes, High, Yes), (No, Yes, Normal, No), (Yes, Yes, High, Yes), (No, No, Low, No), (Yes, No, Normal, No). Using naive Bayes, predict whether a patient with Fever = Yes, Cough = Yes, Temperature = Normal has flu.** [10 marks] ### Answer #### Given The full training data are in Step 1. Query: Fever=Yes, Cough=Yes, Temperature=Normal. The main calculation uses unsmoothed counts. #### To find Both class scores and the predicted class. #### Formula and why it applies $$ score(C)=P(C)\prod_jP(a_j\mid C) $$ C is a class label. Each aⱼ is one query feature value. First count inside a class, then multiply the prior and all three likelihoods. #### Solution ##### Step 1. Write the training rows as a table | Row | Fever | Cough | Temperature | Class | | --- | --- | --- | --- | --- | | 1 | Yes | Yes | High | Yes | | 2 | No | No | Normal | No | | 3 | Yes | No | Low | No | | 4 | Yes | Yes | High | Yes | | 5 | No | Yes | Normal | No | | 6 | Yes | Yes | High | Yes | | 7 | No | No | Low | No | | 8 | Yes | No | Normal | No | ##### Step 2. Separate the two class groups | Class | Rows in this group | Class count | Prior | | --- | --- | --- | --- | | Yes | 1, 4, 6 | 3 | 3/8 = 0.375 | | No | 2, 3, 5, 7, 8 | 5 | 5/8 = 0.625 | ##### Step 3. For class Yes, count Fever=Yes Look only at class Yes rows 1, 4, 6. The matching rows are 1, 4, 6. $$ P(\text{Fever}=\text{Yes}\mid C=\text{Yes})=\frac{3}{3}=1 $$ ##### Step 4. For class Yes, count Cough=Yes Look only at class Yes rows 1, 4, 6. The matching rows are 1, 4, 6. $$ P(\text{Cough}=\text{Yes}\mid C=\text{Yes})=\frac{3}{3}=1 $$ ##### Step 5. For class Yes, count Temperature=Normal Look only at class Yes rows 1, 4, 6. No row in this class has that feature value. $$ P(\text{Temperature}=\text{Normal}\mid C=\text{Yes})=\frac{0}{3}=0 $$ ##### Step 6. Multiply the Yes score one factor at a time $$ P(Yes)=3/8=0.375 $$ $$ 0.375\times1=0.375 $$ $$ 0.375\times1=0.375 $$ $$ 0.375\times0=0 $$ ##### Step 7. For class No, count Fever=Yes Look only at class No rows 2, 3, 5, 7, 8. The matching rows are 3, 8. $$ P(\text{Fever}=\text{Yes}\mid C=\text{No})=\frac{2}{5}=0.4 $$ ##### Step 8. For class No, count Cough=Yes Look only at class No rows 2, 3, 5, 7, 8. The matching rows are 5. $$ P(\text{Cough}=\text{Yes}\mid C=\text{No})=\frac{1}{5}=0.2 $$ ##### Step 9. For class No, count Temperature=Normal Look only at class No rows 2, 3, 5, 7, 8. The matching rows are 2, 5, 8. $$ P(\text{Temperature}=\text{Normal}\mid C=\text{No})=\frac{3}{5}=0.6 $$ ##### Step 10. Multiply the No score one factor at a time $$ P(No)=5/8=0.625 $$ $$ 0.625\times0.4=0.25 $$ $$ 0.25\times0.2=0.05 $$ $$ 0.05\times0.6=0.03 $$ ##### Step 11. Compare the scores $$ score(Yes)=0\qquad score(No)=0.03 $$ The larger score belongs to No. Normalise only if a posterior probability is requested: divide each score by their total 0.03. **Final answer.** Predict No. Yes score=0; No score=0.03. The zero Yes score is caused by an absent training combination. If Laplace smoothing is explicitly required, apply it to all feature likelihoods in both classes. That is a different calculation and may change the prediction; do not mix smoothed and unsmoothed sides. **Yaad rakho:** Class ke rows alag karo. Har feature ka count usi class ke total se divide. Saare factors multiply karo. --- ## TEE · Q5(a) Hotel guest movement (Markov chain) [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q5a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q5a Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. A hotel chain has properties in Paris (P), London (L) and Rome (R). Of guests in Paris, 60% stay, 25% go to London, 15% to Rome; in London 30% go to Paris, 50% stay, 20% go to Rome; in Rome 20% go to Paris, 30% to London, 50% stay. 1. Construct the transition matrix. 2. Explain the diagonal and off-diagonal entries. 3. Draw the state diagram. 4. Today's counts are [50, 30, 20]; find tomorrow's expected counts. 5. Which city gains or loses guests in the long run?** [10 marks] ### Answer #### Given All transition values are given in the full question. State order: P, L, R. Initial vector [50, 30, 20]. #### To find The matrix, diagram, next-step vector and long-run comparison. #### Formula and why it applies Rows mean current state; columns mean next state. For each destination, multiply each origin amount by its transition probability, then add. $$ x_{next}=x_{now}P\qquad\pi=\pi P,\quad\sum_i\pi_i=1 $$ #### Solution ##### Step 1. Convert every origin row to probabilities | From / To | P | L | R | | --- | --- | --- | --- | | P | 0.6 | 0.25 | 0.15 | | L | 0.3 | 0.5 | 0.2 | | R | 0.2 | 0.3 | 0.5 | Each row sums to 1 because it lists all possible destinations for that origin. A percentage is divided by 100: 25%=.25. ##### Step 2. Interpret entries and draw the transitions Diagonal entries are the fractions staying at their current state; off-diagonal entries are fractions moving to a different state. Each arrow uses its origin row and destination column. These fractions have different origin populations, so adding raw column probabilities does not give the number arriving. Swipe across the diagram to read its labels ↔ ![Each arrow is labelled with the probability of that transition.](https://deep-learning-study-room.vercel.app/is2/fig/mk_hotel.55f5204329.svg) Diagram: Each arrow is labelled with the probability of that transition.. Labels: Daily guest movement: Paris, London, Rome | 0.25 | 0.15 | 0.3 | 0.2 | 0.2 | 0.3 | 0.6 | 0.5 | 0.5 | P | L | R Each arrow is labelled with the probability of that transition. ##### Step 3. Next step: collect all contributions to P | Origin | Current count/share | Transition to P | Contribution | | --- | --- | --- | --- | | P | 50 | 0.6 | 50 × 0.6 = 30 | | L | 30 | 0.3 | 30 × 0.3 = 9 | | R | 20 | 0.2 | 20 × 0.2 = 4 | $$ P_{new}=30+9+4=43 $$ ##### Step 4. Next step: collect all contributions to L | Origin | Current count/share | Transition to L | Contribution | | --- | --- | --- | --- | | P | 50 | 0.25 | 50 × 0.25 = 12.5 | | L | 30 | 0.5 | 30 × 0.5 = 15 | | R | 20 | 0.3 | 20 × 0.3 = 6 | $$ L_{new}=12.5+15+6=33.5 $$ ##### Step 5. Next step: collect all contributions to R | Origin | Current count/share | Transition to R | Contribution | | --- | --- | --- | --- | | P | 50 | 0.15 | 50 × 0.15 = 7.5 | | L | 30 | 0.2 | 30 × 0.2 = 6 | | R | 20 | 0.5 | 20 × 0.5 = 10 | $$ R_{new}=7.5+6+10=23.5 $$ ##### Step 6. Stationarity: write the Paris equation $$ p=0.6p+0.3l+0.2r $$ $$ 0.4p=0.3l+0.2r $$ Divide every term by .4. $$ p=0.75l+0.5r $$ ##### Step 7. Write the London equation and substitute p $$ l=0.25p+0.5l+0.3r $$ $$ 0.5l=0.25(0.75l+0.5r)+0.3r $$ $$ 0.5l=0.1875l+0.125r+0.3r $$ ##### Step 8. Collect like terms $$ (0.5-0.1875)l=(0.125+0.3)r $$ $$ 0.3125l=0.425r\quad\Rightarrow\quad l=1.36r $$ $$ p=0.75(1.36r)+0.5r=1.52r $$ ##### Step 9. Use total probability to get all three shares $$ p+l+r=1\quad\Rightarrow\quad1.52r+1.36r+r=1 $$ $$ 3.88r=1\quad r=1/3.88=25/97 $$ $$ l=1.36(25/97)=34/97\quad p=1.52(25/97)=38/97 $$ ##### Step 10. Compare the stationary shares with the initial shares | State | Initial share | Stationary share | Change in share | Conclusion | | --- | --- | --- | --- | --- | | P | 0.5 | 0.391752577 | -0.108247423 | Loses | | L | 0.3 | 0.350515464 | 0.050515464 | Gains | | R | 0.2 | 0.257731959 | 0.057731959 | Gains | Compare like with like: divide counts by their total before comparing with stationary probabilities. “Stationary” means one more step leaves the shares unchanged. **Final answer.** Next-step vector [43.0, 33.5, 23.5]. Stationary shares ≈ [0.391752577, 0.350515464, 0.257731959]. P loses, L gains, R gains. --- ## TEE · Q5(b) Genetic algorithm [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q5b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q5b Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Implement a GA to maximise $$ f(x)=2x^3-5x^2+7x $$ for x in [0,15]. Take 4-bit representations of 13, 22, 18, 23. Crossover after the second bit and mutate by flipping any one bit. Produce two offspring and comment on fitness.** [10 marks] ### Answer **The printed data are inconsistent.** 22, 18 and 23 are outside 0–15 and cannot fit into four bits. Increasing the bit count would encode the numbers but would still violate the stated range. There is no single exact solution satisfying all printed conditions. The complete worked variant below explicitly changes the invalid parents to their last four bits (values modulo 16), producing [13,6,2,7]. This is one declared repair, not a silent interpretation of the printed question. If the instructor clarifies different corrected data, follow those data. #### Given Printed parents: [13, 22, 18, 23]. Worked variant: [13, 6, 2, 7] with 4 bits. For the unspecified selection step, select the two highest-fitness candidates. For the unspecified mutation location, flip the last bit of each child. Crossover is after bit 2. #### To find Two valid children under the declared assumptions and their fitness. #### Formula and why it applies Encoding must respect the candidate range. Crossover swaps tails; mutation flips the chosen bit. Fitness is evaluated after mutation. $$ f(x)=2x^3-5x^2+7x $$ #### Solution ##### Step 1. Encode and evaluate the declared parent population | x | Binary | Fitness substitution | | --- | --- | --- | | 13 | 1101 | 2(13³) − 5(13²) + 7(13) = 3640 | | 6 | 0110 | 2(6³) − 5(6²) + 7(6) = 294 | | 2 | 0010 | 2(2³) − 5(2²) + 7(2) = 10 | | 7 | 0111 | 2(7³) − 5(7²) + 7(7) = 490 | ##### Step 2. Select the two fittest under the stated assumption [13, 7] with fitness [3640, 490]. ##### Step 3. Cut after bit 2 and exchange tails | Parent 1 | Parent 2 | Child 1 | Child 2 | | --- | --- | --- | --- | | 11 | 01 | 01 | 11 | 1111 | 0101 | ##### Step 4. Flip the last bit of each child | Before | After | Decoded x | | --- | --- | --- | | 1111 | 1110 | 14 | | 0101 | 0100 | 4 | ##### Step 5. Evaluate both children | x | Fitness calculation | | --- | --- | | 14 | 2(2744) − 5(196) + 7(14) = 4606 | | 4 | 2(64) − 5(16) + 7(4) = 76 | ##### Step 6. Compare with the selected parents Best parent score = 3640; best child score = 4606. Parent average = 2065; child average = 2341. Both the best score and the average score increased in this worked variant. An individual child can still be worse than a parent; GA does not guarantee improvement after every random operator. **Final answer.** Under these declared repairs and operator choices, children are [14, 4] with fitness [4606, 76]. These are not unique answers to the inconsistent and under-specified original question. The printed wording about “two offspring” is not an explicit requirement for two complete generations. The M2 examples separately work two full generations because your class update asks for iteration 2. ## Evaluate the polynomial without skipping the powers Calculate x²=x×x, then x³=x²×x. Multiply each by its coefficient and combine with the written signs. A bit-string repair in the worked answer is an explicitly stated response to inconsistent printed givens; it is not a new instruction in the original paper. ### x=13 $$ x^2=13\times13=169\qquad x^3=169\times13=2197 $$ $$ 2x^3=4394\quad 5x^2=845\quad 7x=91 $$ $$ f(13)=4394-845+91=3640 $$ ### x=6 $$ x^2=6\times6=36\qquad x^3=36\times6=216 $$ $$ 2x^3=432\quad 5x^2=180\quad 7x=42 $$ $$ f(6)=432-180+42=294 $$ ### x=2 $$ x^2=2\times2=4\qquad x^3=4\times2=8 $$ $$ 2x^3=16\quad 5x^2=20\quad 7x=14 $$ $$ f(2)=16-20+14=10 $$ ### x=7 $$ x^2=7\times7=49\qquad x^3=49\times7=343 $$ $$ 2x^3=686\quad 5x^2=245\quad 7x=49 $$ $$ f(7)=686-245+49=490 $$ ### x=15 $$ x^2=15\times15=225\qquad x^3=225\times15=3375 $$ $$ 2x^3=6750\quad 5x^2=1125\quad 7x=105 $$ $$ f(15)=6750-1125+105=5730 $$ ### x=5 $$ x^2=5\times5=25\qquad x^3=25\times5=125 $$ $$ 2x^3=250\quad 5x^2=125\quad 7x=35 $$ $$ f(5)=250-125+35=160 $$ ### x=14 $$ x^2=14\times14=196\qquad x^3=196\times14=2744 $$ $$ 2x^3=5488\quad 5x^2=980\quad 7x=98 $$ $$ f(14)=5488-980+98=4606 $$ ### x=4 $$ x^2=4\times4=16\qquad x^3=16\times4=64 $$ $$ 2x^3=128\quad 5x^2=80\quad 7x=28 $$ $$ f(4)=128-80+28=76 $$ “Modulo 16” means keep the remainder after dividing by 16: 22=1×16+6, so its remainder is 6; 18=1×16+2; 23=1×16+7. This is only the declared repair used here, not an unspoken general encoding rule. --- ## TEE · Q6(a) Applications of fuzzy logic [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q6a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q6a Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **List the applications of fuzzy logic. Explain any one application in detail.** [10 marks] ### Answer ### Definition Fuzzy logic is a multi-valued logic, introduced by Lotfi Zadeh (1965), in which truth values range over [0, 1]; it lets machines reason with vague, human-like terms such as “slightly dirty” or “very hot”. ### Applications | Area | Examples | | --- | --- | | Home appliances | Washing machines (wash time from dirt and load), air conditioners, refrigerators, rice cookers, vacuum cleaners | | Automotive | Automatic transmission, anti-lock braking (ABS), cruise control, engine control | | Transportation | Traffic-signal control, Sendai subway train control (smooth braking and acceleration) | | Consumer electronics | Camera autofocus and auto-exposure, image stabilisation | | Industrial control | Cement kilns, pH, temperature and distillation control in chemical plants | | Aerospace | Satellite and spacecraft attitude control | | Medicine | Diagnosis support, anaesthesia dosage control | | Finance and decisions | Loan approval, credit and fire-risk scoring, stock trading rules | | AI and robotics | Robot navigation, natural-language interfaces, expert systems | ### One application in detail: fuzzy washing machine 1. **Problem.** Wash time depends on how dirty and how greasy the clothes are. These are vague (“a bit dirty”), and a fixed timer wastes water and power. 2. **Inputs and outputs.** Sensors measure dirt (turbidity of water) and grease; the output is wash time. Each is described by fuzzy sets: dirt Small/Medium/High, grease Small/Medium/High, time Very Short … Very Long. 3. **Fuzzification.** Sensor readings are converted to membership degrees, e.g. dirt 60% → Medium 0.8, High 0.2. 4. **Rule base.** 9 expert rules, e.g. IF dirt is High AND grease is High THEN time is Very Long; IF dirt is Small AND grease is Small THEN time is Very Short. 5. **Inference.** Mamdani min–max: rule strength = min of the two memberships; outputs combined by max. 6. **Defuzzification.** Use the specified defuzzifier. For a separate representative-weighted illustration, strengths [.6,.4,.2] at times [25,40,60] give (.6×25+.4×40+.2×60)/(.6+.4+.2)=(15+16+12)/1.2≈35.833 minutes. These illustrative strengths are supplied here, not derived from unspecified input graphs. A Mamdani area centroid requires the full aggregate shape. 7. **Benefits.** Smooth adjustment to every load, saves water and electricity, easy to design from expert knowledge without an exact mathematical model. ### Advantages and disadvantages (bonus) **Advantages:** handles imprecise or noisy data; mimics human reasoning; easy to design and modify; no exact model needed; low computational cost. **Disadvantages:** rules and membership functions depend on expert judgement; results are hard to verify formally; tuning can take time; not suited where very high precision is required. **Yaad rakho:** Application explain karte waqt 5 cheezein: inputs, fuzzy sets, rules, inference (min/max), defuzzification + ek number ka example. --- ## TEE · Q6(b) Learner progression (Markov chain) [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q6b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q6b Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Students are Beginner (B), Intermediate (I) or Advanced (A). Of Beginners, 60% remain, 30% become Intermediate, 10% go to Advanced. Of Intermediates, 20% fall back, 50% remain, 30% progress. Of Advanced, 10% drop to Intermediate, 80% remain and 10% stop (treated as remaining in A). 1. Construct the matrix. 2. Interpret retention, progression and regression. 3. Draw the diagram. 4. For [50, 30, 20], find the distribution after one day. 5. Which level has the largest steady-state share?** [10 marks] ### Answer #### Given All transition values are given in the full question. State order: B, I, A. Initial vector [50, 30, 20]. #### To find The matrix, diagram, next-step vector and long-run comparison. #### Formula and why it applies Rows mean current state; columns mean next state. For each destination, multiply each origin amount by its transition probability, then add. $$ x_{next}=x_{now}P\qquad\pi=\pi P,\quad\sum_i\pi_i=1 $$ #### Solution ##### Step 1. Convert every origin row to probabilities | From / To | B | I | A | | --- | --- | --- | --- | | B | 0.6 | 0.3 | 0.1 | | I | 0.2 | 0.5 | 0.3 | | A | 0 | 0.1 | 0.9 | Each row sums to 1 because it lists all possible destinations for that origin. A percentage is divided by 100: 25%=.25. ##### Step 2. Interpret entries and draw the transitions Diagonal entries are the fractions staying at their current state; off-diagonal entries are fractions moving to a different state. Each arrow uses its origin row and destination column. These fractions have different origin populations, so adding raw column probabilities does not give the number arriving. Swipe across the diagram to read its labels ↔ ![Each arrow is labelled with the probability of that transition.](https://deep-learning-study-room.vercel.app/is2/fig/mk_learning.61a789edc7.svg) Diagram: Each arrow is labelled with the probability of that transition.. Labels: Daily learner movement: Beginner, Intermediate, Advanced | 0.3 | 0.1 | 0.2 | 0.3 | 0.1 | 0.6 | 0.5 | 0.9 | B | I | A Each arrow is labelled with the probability of that transition. ##### Step 3. Next step: collect all contributions to B | Origin | Current count/share | Transition to B | Contribution | | --- | --- | --- | --- | | B | 50 | 0.6 | 50 × 0.6 = 30 | | I | 30 | 0.2 | 30 × 0.2 = 6 | | A | 20 | 0 | 20 × 0 = 0 | $$ B_{new}=30+6+0=36 $$ ##### Step 4. Next step: collect all contributions to I | Origin | Current count/share | Transition to I | Contribution | | --- | --- | --- | --- | | B | 50 | 0.3 | 50 × 0.3 = 15 | | I | 30 | 0.5 | 30 × 0.5 = 15 | | A | 20 | 0.1 | 20 × 0.1 = 2 | $$ I_{new}=15+15+2=32 $$ ##### Step 5. Next step: collect all contributions to A | Origin | Current count/share | Transition to A | Contribution | | --- | --- | --- | --- | | B | 50 | 0.1 | 50 × 0.1 = 5 | | I | 30 | 0.3 | 30 × 0.3 = 9 | | A | 20 | 0.9 | 20 × 0.9 = 18 | $$ A_{new}=5+9+18=32 $$ ##### Step 6. Stationarity: balance the Beginner column $$ b=0.6b+0.2i $$ $$ 0.4b=0.2i\quad\Rightarrow\quad i=2b $$ ##### Step 7. Balance the Advanced column $$ a=0.1b+0.3i+0.9a $$ $$ 0.1a=0.1b+0.3(2b)=0.7b $$ $$ a=0.7b/0.1=7b $$ ##### Step 8. Use the sum of shares $$ b+i+a=1\quad b+2b+7b=1 $$ $$ 10b=1\quad b=0.1\quad i=0.2\quad a=0.7 $$ Advanced is not an absorbing state: its self-transition is .9, not 1. ##### Step 9. Compare the stationary shares with the initial shares | State | Initial share | Stationary share | Change in share | Conclusion | | --- | --- | --- | --- | --- | | B | 0.5 | 0.1 | -0.4 | Loses | | I | 0.3 | 0.2 | -0.1 | Loses | | A | 0.2 | 0.7 | 0.5 | Gains | Compare like with like: divide counts by their total before comparing with stationary probabilities. “Stationary” means one more step leaves the shares unchanged. **Final answer.** Next-step vector [36.0, 32.0, 32.0]. Stationary shares ≈ [0.1, 0.2, 0.7]. B loses, I loses, A gains. --- ## TEE · Q7(a) Applications of PSO [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q7a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q7a Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **List the applications of Particle Swarm Optimization and explain any one application in detail.** [10 marks] ### Answer ### Definition (1 line) Particle Swarm Optimization is a population-based metaheuristic, inspired by bird flocking and fish schooling, in which particles move through the search space using their own best position (pbest) and the swarm's best position (gbest) to find optimal or near-optimal solutions. ### Applications 1. **Function optimisation**: maximising or minimising complex non-linear, multimodal functions where derivatives are unavailable. 2. **Neural-network training**: optimising weights and biases (your Lab 8 uses the pyswarms library for this). 3. **Hyper-parameter tuning** of ML models (learning rate, number of hidden units, SVM C and γ). 4. **Feature selection**: choosing the most useful subset of features (binary PSO). 5. **Image processing**: threshold selection for segmentation, edge detection, image enhancement. 6. **Path planning** for robots and drones: shortest collision-free path. 7. **Power systems**: economic load dispatch, optimal placement of capacitors. 8. **Control engineering**: tuning PID controller gains. 9. **Scheduling and wireless sensor networks**: job scheduling, sensor node placement and clustering. ### One application in detail: training a neural network with PSO Training data → Neural network with weights = particle position → Error (fitness) → PSO moves the weights → Best weights = gbest 1. **Encoding.** Flatten all weights and biases of the network into one vector. Each particle is one complete set of weights; a network with 50 weights gives a 50-dimensional search space. 2. **Fitness.** Run the training data through the network with that particle's weights and compute the error (for example mean squared error or cross-entropy). Lower error = better particle (minimisation). 3. **Search.** Apply the velocity and position updates. Each particle is pulled towards the weight set that gave its own lowest error (pbest) and towards the lowest-error weights found by the swarm (gbest). 4. **Stop** after a fixed number of iterations or when the error stops improving; gbest is the trained network. 5. **Why PSO here:** it needs no gradients, so it works with non-differentiable activations and can explore candidate weights without differentiating the objective. It can still stagnate; comparison with gradient methods depends on the problem. **Limitation:** large networks create a high-dimensional search space, and evaluating many complete candidate networks can be computationally expensive. Gradient-based training is often more efficient when useful gradients are available; there is no universal speed ordering for every problem and implementation. ### Advantages and disadvantages (add for 10 marks) | Advantages | Disadvantages | | --- | --- | | Simple to implement; few parameters (w, c₁, c₂) | Can converge prematurely to a local optimum | | No derivatives needed | Sensitive to parameter choice | | Fast convergence in early iterations; memory (pbest, gbest) | No guarantee of the global optimum | | Objective evaluations can be parallel; particles share information through gbest | Less suited to discrete problems without modification | Marking pattern used for ACO in the same paper: listing 3 marks + explanation 2 marks (5-mark version); listing 5 + explanation 5 (10-mark version). **Yaad rakho:** Application likhte waqt teen cheezein: particle kya hai, fitness kya hai, gbest ka matlab kya hai. --- ## TEE · Q7(b) ACO for TSP [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-final-2025--q7b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-final-2025/q7b Actual TEE paper · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. For the four-city travelling salesman problem, the distance and pheromone matrices (A, B, C, D) are given. Apply Ant Colony Optimization. Calculate the path travelled by 2 ants to cover all cities starting from city A. Random numbers for Ant 1 are 0.3, 0.8 and for Ant 2 are 0.5, 0.7. Also update the pheromone matrix after 1 iteration.** [10 marks] ### Answer #### Given $$ D=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&4&2&4\\B&4&0&3&6\\C&2&3&0&2\\D&4&6&2&0\end{array}\qquad \tau=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&1&1&1\\B&1&0&1&1\\C&1&1&0&1\\D&1&1&1&0\end{array} $$ Assume $$ \alpha=\beta=1 $$ , $$ \rho=0.5 $$ , $$ Q=1 $$ (class convention; not stated in the question). #### To find Tour of each ant, tour lengths, updated pheromone matrix. Swipe across the diagram to read its labels ↔ ![Final 2025 Q7b: distances (all pheromones start at 1)](https://deep-learning-study-room.vercel.app/is2/fig/aco_final_graph.ac3cd4aeac.svg) Diagram: Final 2025 Q7b: distances (all pheromones start at 1). Labels: Final 2025 Q7b: distances (all pheromones start at 1) | 4 | 2 | 4 | 3 | 6 | 2 | A | B | C | D #### Formula and why it applies $$ P_{ij}=\frac{\tau_{ij}\,\eta_{ij}}{\sum_{l\in\text{allowed}}\tau_{il}\,\eta_{il}},\qquad\eta_{ij}=\frac1{d_{ij}} $$ $$ \tau_{ij}^{new}=(1-\rho)\tau_{ij}+\sum_{k:\,ij\in\mathrm{tour}_k}\frac{Q}{L_k} $$ The sum includes **only ants whose completed tour contains edge ij**. An unused edge receives no deposit but still evaporates. Here distances are symmetric, so a traversed edge updates both symmetric matrix cells. Both ants construct their tours using the old pheromones; update after both finish, including their return to A. #### Solution ##### Step 1. Probabilities from A (both ants start at A) $$ \tau\eta:\;A\to B=1\times\tfrac14=0.25,\quad A\to C=1\times\tfrac12=0.5,\quad A\to D=1\times\tfrac14=0.25;\quad\text{sum}=1 $$ | Move | P | Cumulative | Range | | --- | --- | --- | --- | | A→B | 0.25 | 0.25 | 0–0.25 | | A→C | 0.50 | 0.75 | 0.25–0.75 | | A→D | 0.25 | 1.00 | 0.75–1 | Ant 1: r = 0.3 → **C**. Ant 2: r = 0.5 → **C**. ##### Step 2. Probabilities from C (B and D left) $$ C\to B=\tfrac13=0.3333,\quad C\to D=\tfrac12=0.5;\quad\text{sum}=0.8333\Rightarrow P_{CB}=0.4,\;P_{CD}=0.6 $$ Cumulative in city order: B 0–0.4, D 0.4–1. Ant 1: r = 0.8 → **D**. Ant 2: r = 0.7 → **D**. ##### Step 3. Complete the tours Only B is left, then return to A. $$ \text{Ant 1}=\text{Ant 2}:\;A\to C\to D\to B\to A,\qquad L=2+2+6+4=14 $$ Swipe across the diagram to read its labels ↔ ![Both ants: A → C → D → B → A, length 14](https://deep-learning-study-room.vercel.app/is2/fig/aco_final_tour.1dbfdb8b98.svg) Diagram: Both ants: A → C → D → B → A, length 14. Labels: Both ants: A → C → D → B → A, length 14 | 4 | 2 | 4 | 3 | 6 | 2 | A | B | C | D ##### Step 4. Pheromone update Each ant deposits $$ Q/L=1/14=0.0714 $$ on each edge of its tour; two ants used the same four edges (AC, CD, DB, BA), so each gets $$ 2/14=0.1429 $$ . $$ \text{used edge: }(1-0.5)(1)+0.1429=0.6429\qquad\text{unused edge (AD, BC): }(1-0.5)(1)+0=0.5 $$ $$ \tau_{\text{new}}=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&0.6429&0.6429&0.5\\B&0.6429&0&0.5&0.6429\\C&0.6429&0.5&0&0.6429\\D&0.5&0.6429&0.6429&0\end{array} $$ **Final answer.** Both ants: A → C → D → B → A, length 14. Updated pheromone: 0.6429 on AB, AC, CD, BD; 0.5 on AD and BC. In the next iteration the ants will prefer the edges of this tour. **Official key note.** The key's matrix shows 0.643 on AC, CD and BD but leaves AB at 0.5, i.e. it forgot to deposit on the return edge B → A even though it counted its length (4) in L = 14. Your slides deposit on the return edge, so AB = 0.6429 is correct. If you wish, write one line: “the return edge B–A is part of the tour, so it also receives Q/L.” Official marking: path of Ant 1 (3) + path of Ant 2 (3) + pheromone matrix (4). --- ## TEE · Q1(a) Disease, fever and test joint table [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q1a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q1a Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. For a disease Y with fever X as a symptom, a test Z is conducted. Joint probabilities: Disease (Y=1): fever & test+ 0.576, fever & test− 0.144, no fever & test+ 0.072, no fever & test− 0.016; No disease (Y=0): 0.008, 0.064, 0.012, 0.108. (1) If fever occurs and the test is positive, what is the probability of disease? (2) What is the probability that a random person tests positive? (3) Given no fever, what is the probability of still having the disease?** [5 marks] ### Answer #### Given The complete joint table or cell values printed in this question. The rows and columns identify the event combinations. #### To find All requested marginal and conditional probabilities. #### Formula and why it applies Marginal: add cells satisfying the event. Conditional: retain the evidence cells and divide the joint-event sum by the evidence sum. $$ P(A\mid E)=\frac{P(A\cap E)}{P(E)},\quad P(E)>0 $$ #### Solution ##### Step 1. (1) $$ P(Y{=}1\mid X{=}1,Z{=}1)=\frac{0.576}{0.576+0.008}=\frac{0.576}{0.584}=0.986 $$ ##### Step 2. (2) Add every test-positive cell $$ P(Z{=}1)=0.576+0.072+0.008+0.012=0.668 $$ ##### Step 3. (3) $$ P(X{=}0)=0.072+0.016+0.012+0.108=0.208,\qquad P(Y{=}1\mid X{=}0)=\frac{0.072+0.016}{0.208}=\frac{0.088}{0.208}=0.423 $$ **Final answer.** (1) 0.986; (2) 0.668; (3) 0.423. **Key correction.** The key writes 0.088/0.076 = 1.158 for part (3). A probability cannot exceed 1; the denominator must be P(no fever) = 0.208. --- ## TEE · Q1(b) Applications of PSO [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q1b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q1b Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **List the applications of Particle Swarm Optimization and explain any one application in detail.** [5 marks] ### Answer ### Definition (1 line) Particle Swarm Optimization is a population-based metaheuristic, inspired by bird flocking and fish schooling, in which particles move through the search space using their own best position (pbest) and the swarm's best position (gbest) to find optimal or near-optimal solutions. ### Applications 1. **Function optimisation**: maximising or minimising complex non-linear, multimodal functions where derivatives are unavailable. 2. **Neural-network training**: optimising weights and biases (your Lab 8 uses the pyswarms library for this). 3. **Hyper-parameter tuning** of ML models (learning rate, number of hidden units, SVM C and γ). 4. **Feature selection**: choosing the most useful subset of features (binary PSO). 5. **Image processing**: threshold selection for segmentation, edge detection, image enhancement. 6. **Path planning** for robots and drones: shortest collision-free path. 7. **Power systems**: economic load dispatch, optimal placement of capacitors. 8. **Control engineering**: tuning PID controller gains. 9. **Scheduling and wireless sensor networks**: job scheduling, sensor node placement and clustering. ### One application in detail: training a neural network with PSO Training data → Neural network with weights = particle position → Error (fitness) → PSO moves the weights → Best weights = gbest 1. **Encoding.** Flatten all weights and biases of the network into one vector. Each particle is one complete set of weights; a network with 50 weights gives a 50-dimensional search space. 2. **Fitness.** Run the training data through the network with that particle's weights and compute the error (for example mean squared error or cross-entropy). Lower error = better particle (minimisation). 3. **Search.** Apply the velocity and position updates. Each particle is pulled towards the weight set that gave its own lowest error (pbest) and towards the lowest-error weights found by the swarm (gbest). 4. **Stop** after a fixed number of iterations or when the error stops improving; gbest is the trained network. 5. **Why PSO here:** it needs no gradients, so it works with non-differentiable activations and can explore candidate weights without differentiating the objective. It can still stagnate; comparison with gradient methods depends on the problem. **Limitation:** large networks create a high-dimensional search space, and evaluating many complete candidate networks can be computationally expensive. Gradient-based training is often more efficient when useful gradients are available; there is no universal speed ordering for every problem and implementation. ### Advantages and disadvantages (add for 10 marks) | Advantages | Disadvantages | | --- | --- | | Simple to implement; few parameters (w, c₁, c₂) | Can converge prematurely to a local optimum | | No derivatives needed | Sensitive to parameter choice | | Fast convergence in early iterations; memory (pbest, gbest) | No guarantee of the global optimum | | Objective evaluations can be parallel; particles share information through gbest | Less suited to discrete problems without modification | Marking pattern used for ACO in the same paper: listing 3 marks + explanation 2 marks (5-mark version); listing 5 + explanation 5 (10-mark version). **Yaad rakho:** Application likhte waqt teen cheezein: particle kya hai, fitness kya hai, gbest ka matlab kya hai. --- ## TEE · Q1(c) Fuzzy set operations [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q1c/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q1c Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. X = {(c1, 0.7), (c2, 0.9), (c3, 0.2), (c4, 0.4), (c5, 0.9)}, Y = {(c1, 0.6), (c2, 0.3), (c3, 0.8), (c4, 0.7), (c5, 0.1)}. Find X ∪ Y, X ∩ Y, ¬X ∪ ¬Y, ¬X ∩ ¬Y.** [5 marks] ### Answer #### Given The aligned X and Y memberships printed in this question. #### To find The four requested membership lists in the original element order. #### Formula and why it applies Use max for OR, min for AND, and 1−μ for NOT. $$ \mu_{X\cup Y}=\max(\mu_X,\mu_Y),\quad\mu_{X\cap Y}=\min(\mu_X,\mu_Y) $$ Complement both operands before applying the operator in ¬X∪¬Y or ¬X∩¬Y. #### Solution | | c1 | c2 | c3 | c4 | c5 | | --- | --- | --- | --- | --- | --- | | ¬X | 0.3 | 0.1 | 0.8 | 0.6 | 0.1 | | ¬Y | 0.4 | 0.7 | 0.2 | 0.3 | 0.9 | | **X ∪ Y** | 0.7 | 0.9 | 0.8 | 0.7 | 0.9 | | **X ∩ Y** | 0.6 | 0.3 | 0.2 | 0.4 | 0.1 | | **¬X ∪ ¬Y** | 0.4 | 0.7 | 0.8 | 0.6 | 0.9 | | **¬X ∩ ¬Y** | 0.3 | 0.1 | 0.2 | 0.3 | 0.1 | **Final answer.** As in the last four rows. De Morgan check: ¬X ∪ ¬Y = 1 − (X ∩ Y) and ¬X ∩ ¬Y = 1 − (X ∪ Y) ✓. ## Calculate every element separately ### Element c1 $$ \neg X:1-0.7=0.3\qquad\neg Y:1-0.6=0.4 $$ $$ X\cup Y:\max(0.7,0.6)=0.7 $$ $$ X\cap Y:\min(0.7,0.6)=0.6 $$ $$ \neg X\cup\neg Y:\max(0.3,0.4)=0.4 $$ $$ \neg X\cap\neg Y:\min(0.3,0.4)=0.3 $$ ### Element c2 $$ \neg X:1-0.9=0.1\qquad\neg Y:1-0.3=0.7 $$ $$ X\cup Y:\max(0.9,0.3)=0.9 $$ $$ X\cap Y:\min(0.9,0.3)=0.3 $$ $$ \neg X\cup\neg Y:\max(0.1,0.7)=0.7 $$ $$ \neg X\cap\neg Y:\min(0.1,0.7)=0.1 $$ ### Element c3 $$ \neg X:1-0.2=0.8\qquad\neg Y:1-0.8=0.2 $$ $$ X\cup Y:\max(0.2,0.8)=0.8 $$ $$ X\cap Y:\min(0.2,0.8)=0.2 $$ $$ \neg X\cup\neg Y:\max(0.8,0.2)=0.8 $$ $$ \neg X\cap\neg Y:\min(0.8,0.2)=0.2 $$ ### Element c4 $$ \neg X:1-0.4=0.6\qquad\neg Y:1-0.7=0.3 $$ $$ X\cup Y:\max(0.4,0.7)=0.7 $$ $$ X\cap Y:\min(0.4,0.7)=0.4 $$ $$ \neg X\cup\neg Y:\max(0.6,0.3)=0.6 $$ $$ \neg X\cap\neg Y:\min(0.6,0.3)=0.3 $$ ### Element c5 $$ \neg X:1-0.9=0.1\qquad\neg Y:1-0.1=0.9 $$ $$ X\cup Y:\max(0.9,0.1)=0.9 $$ $$ X\cap Y:\min(0.9,0.1)=0.1 $$ $$ \neg X\cup\neg Y:\max(0.1,0.9)=0.9 $$ $$ \neg X\cap\neg Y:\min(0.1,0.9)=0.1 $$ --- ## TEE · Q1(d) Savings and current accounts (Markov chain) [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q1d/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q1d Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Each year 20% of Savings holders switch to Current and 80% stay; 10% of Current holders switch to Savings and 90% stay. (a) Construct the transition matrix. (b) Draw the state diagram.** [5 marks] ### Answer #### Given All transition values are given in the full question. State order: S, C. #### To find The transition matrix and state diagram. #### Formula and why it applies Rows mean current state; columns mean next state. For each destination, multiply each origin amount by its transition probability, then add. #### Solution ##### Step 1. Convert every origin row to probabilities | From / To | S | C | | --- | --- | --- | | S | 0.8 | 0.2 | | C | 0.1 | 0.9 | Each row sums to 1 because it lists all possible destinations for that origin. A percentage is divided by 100: 25%=.25. ##### Step 2. Interpret entries and draw the transitions Diagonal entries are the fractions staying at their current state; off-diagonal entries are fractions moving to a different state. Each arrow uses its origin row and destination column. These fractions have different origin populations, so adding raw column probabilities does not give the number arriving. Swipe across the diagram to read its labels ↔ ![Each arrow is labelled with the probability of that transition.](https://deep-learning-study-room.vercel.app/is2/fig/mk_bank.879499f6eb.svg) Diagram: Each arrow is labelled with the probability of that transition.. Labels: Savings (S) and Current (C) accounts per year | 0.2 | 0.1 | 0.8 | 0.9 | S | C Each arrow is labelled with the probability of that transition. **Final answer.** The transition matrix and state diagram are given above. --- ## TEE · Q2(a) Fuzzy fire-hazard system [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q2a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q2a Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **A building safety department wants a fuzzy logic system to predict fire hazards. Inputs are Temperature (1–100°C) and Smoke Level (1–100). The output Fire Risk (1–100) has labels Low, Moderate and High. Define triangular membership functions, construct at least nine rules, explain Mamdani inference and defuzzify for Temperature=65°C and Smoke=80.** [10 marks] ### Answer #### Given | Variable | Crisp value | | --- | --- | | Temperature | 65 | | Smoke Level | 80 | The question asks us to design the membership functions. This solution chooses Low=Tri(1,25,50), Medium=Tri(25,50,75), High=Tri(50,75,100) for both inputs. Output labels use the same triangles in their respective order. These are explicit design choices, not additional printed givens. #### To find A complete stated design and its continuous Mamdani centroid. #### Formula and why it applies Use min for AND and implication, max for aggregation and the true area centroid of the resulting shape. A peak-only weighted average is a different defuzzifier. $$ z^*=\frac{\int z\mu_{out}(z)\,dz}{\int\mu_{out}(z)\,dz} $$ #### Solution ##### Step 1. Specify the membership functions | Input / output position | Triangle | Peak | | --- | --- | --- | | Low / Low | (1,25,50) | 25 | | Medium / Moderate | (25,50,75) | 50 | | High / High | (50,75,100) | 75 | Use 0 outside each triangle, the rising fraction before the peak and the falling fraction after it. This design is evaluated at the requested input; a deployed controller would also need coverage/fallback at all boundary values. ##### Step 2. Write all nine rules | Temperature / Smoke Level | Low | Medium | High | | --- | --- | --- | --- | | Low | Low | Low | Moderate | | Medium | Low | Moderate | High | | High | Moderate | High | High | Read each cell as an IF–AND–THEN rule. For example, the centre cell combines Medium with Medium. ##### Step 3. Fuzzify Temperature = 65 $$ \mu_{Low}(65)=0 $$ $$ \mu_{Medium}(65)=\frac{75-65}{75-50}=0.4 $$ $$ \mu_{High}(65)=\frac{65-50}{75-50}=0.6 $$ ##### Step 4. Fuzzify Smoke Level = 80 $$ \mu_{Low}(80)=0 $$ $$ \mu_{Medium}(80)=0 $$ $$ \mu_{High}(80)=\frac{100-80}{100-75}=0.8 $$ ##### Step 5. Evaluate all rules and aggregate matching consequents | Temperature | Smoke Level | Firing strength | Consequent | | --- | --- | --- | --- | | Low | Low | min(0,0) = 0 | Low | | Low | Medium | min(0,0) = 0 | Low | | Low | High | min(0,0.8) = 0 | Moderate | | Medium | Low | min(0.4,0) = 0 | Low | | Medium | Medium | min(0.4,0) = 0 | Moderate | | Medium | High | min(0.4,0.8) = 0.4 | High | | High | Low | min(0.6,0) = 0 | Moderate | | High | Medium | min(0.6,0) = 0 | High | | High | High | min(0.6,0.8) = 0.6 | High | Aggregated label strengths: Low = 0, Moderate = 0, High = 0.6. $$ \mu_{out}(z)=\max_r\min(\alpha_r,\mu_{C_r}(z)) $$ ##### Step 6. Draw the aggregated output Swipe across the diagram to read its labels ↔ ![The shaded upper envelope counts any overlap only once.](https://deep-learning-study-room.vercel.app/is2/fig/v2_fire_aggregate.e0cab4ad42.svg) Diagram: The shaded upper envelope counts any overlap only once.. Labels: Chosen triangular output sets and their max-aggregate | 0 | 0.5 | 1 | 1 | 25 | 50 | 75 | 100 | μ | Output score | Low | Moderate | High | centroid ≈ 75 The shaded upper envelope counts any overlap only once. ##### Step 7. Calculate area and moment of each straight-line strip For a strip from a to b with endpoint heights u and v, its trapezoid area is A. Its first moment M is the integral of z times the line height. $$ A=(b-a)(u+v)/2 $$ $$ M=\frac{b-a}{6}\big[(2a+b)u+(a+2b)v\big] $$ | a | b | u | v | Area A | Moment M | | --- | --- | --- | --- | --- | --- | | 50 | 65 | 0 | 0.6 | 4.5 | 270 | | 65 | 85 | 0.6 | 0.6 | 12 | 900 | | 85 | 100 | 0.6 | 0 | 4.5 | 405 | ##### Step 8. Divide total moment by total area $$ \sum A=21 $$ $$ \sum M=1575 $$ $$ z^*=\frac{1575}{21}\approx75 $$ **Final answer.** For these stated triangles and rules, the centroid is 75. This is in the high-risk region of this illustrative fuzzy design. Because the paper asks for a design, a different explicitly specified membership/rule system can yield a different numerical output. Explain the choices and use the chosen defuzzification method consistently; no particular examiner mark outcome is promised. ## Why the clipped High shape is symmetric about 75 Only the High output survives, at height .6. Its triangle is (50,75,100). Solve its rising edge: (z−50)/25=.6, so z=50+15=65. Solve its falling edge: (100−z)/25=.6, so z=100−15=85. | Piece | Area calculation | Centre | Moment | | --- | --- | --- | --- | | Left triangle [50,65] | ½×15×.6=4.5 | (50+65+65)/3=60 | 4.5×60=270 | | Rectangle [65,85] | 20×.6=12 | (65+85)/2=75 | 12×75=900 | | Right triangle [85,100] | ½×15×.6=4.5 | (85+85+100)/3=90 | 4.5×90=405 | The horizontal centroid of a triangle is the average of the three vertex x-coordinates. Add areas: 4.5+12+4.5=21. Add moments:270+900+405=1575. Divide 1575/21=75, agreeing with symmetry. --- ## TEE · Q2(b) GMM: compare with k-means, log-likelihood, parameters [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q2b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q2b Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **With reference to the Gaussian Mixture Model (GMM): compare GMM with the k-means algorithm; derive the equation for the log-likelihood function; state the number of parameters to be computed if K = 5.** [10 marks] ### Answer ## Read the derivation’s symbols first “i.i.d.” means the observations are modelled as independent draws from the same distribution. The product Π runs over observations; the inner sum Σ runs over components. λki in some notes is the same responsibility that other sections call γik. A superscript i here labels the observation, not an exponent. Jensen’s inequality gives a lower bound on the log of the component sum. EM chooses responsibilities making that bound touch the current likelihood, then updates the parameters to maximise the bound with those responsibilities fixed. This explains the two alternating steps; λ values are not arbitrary new model parameters. | Parameter | Number in a 1D K-component GMM | Why | | --- | --- | --- | | Means | K | One centre for each component | | Variances | K | One spread for each component | | Independent mixing weights | K−1 | The last weight is 1 minus the others | | Total | 3K−1 | K+K+(K−1) | For K=2, choose μ₁,μ₂,σ₁²,σ₂² and π₁: five free numbers. Then π₂=1−π₁. For K=5, 5+5+4=14. In 2D with a full covariance matrix, each component instead has two means and three independent covariance entries (two variances and one shared off-diagonal covariance). Five such components have 5×(2+3)+4=29 free parameters. ### 1. GMM versus k-means | Aspect | k-means | GMM | | --- | --- | --- | | Assignment | Hard: each point in exactly one cluster | Soft: probability $$ \lambda_k^i $$ of belonging to each cluster | | Model | Geometric: nearest centroid | Probabilistic generative model $$ \sum_k\pi_k\mathcal N(\mu_k,\Sigma_k) $$ | | Cluster shape | Spherical, similar size | Elliptical, any size/orientation; overlapping clusters | | Parameters | K centroids | Weights, means, covariances (1-D: 3K − 1) | | Algorithm / objective | Assign–update; minimise squared distances | EM; maximise log-likelihood | | Output | Hard labels and fitted centroids | Labels, membership probabilities and a density (useful for anomaly detection) | | Relation | The hard-assignment, equal spherical variance, small-variance limit of a Gaussian-mixture objective leads to the k-means objective. Ordinary finite-variance EM is not the same algorithm as k-means. Both need K in advance and depend on initialisation. | | ### 2. Derivation of the log-likelihood **Step 1. Model.** Each point is generated by choosing component k with probability $$ \pi_k $$ and drawing $$ x_i\sim\mathcal N(\mu_k,\sigma_k^2) $$ , so its density is $$ f_{mix}(x_i)=\sum_k\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2) $$ . **Step 2. Likelihood.** Points are i.i.d., so the likelihood is a product (θ = all parameters): $$ L(\theta)=\prod_{i=1}^{n}\Big[\sum_{k=1}^{K}\pi_k\frac{1}{\sqrt{2\pi}\,\sigma_k}e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}}\Big] $$ **Step 3. Log-likelihood.** Logs turn the product into a sum: $$ \log L(\theta)=\sum_{i=1}^{n}\log\Big(\sum_{k=1}^{K}\pi_k\frac{1}{\sqrt{2\pi}\,\sigma_k}e^{-\frac{(x_i-\mu_k)^2}{2\sigma_k^2}}\Big) $$ **Step 4. Why it cannot be solved directly.** The sum sits inside the log, so setting derivatives to zero gives no closed-form solution. **Step 5. Jensen's inequality.** log is concave, so for weights $$ \lambda_k\ge0,\;\sum\lambda_k=1 $$ : $$ \log\big(\sum_k\lambda_ka_k\big)\ge\sum_k\lambda_k\log a_k $$ . Introduce $$ \lambda_k^i $$ for every point ( $$ \sum_k\lambda_k^i=1 $$ ), multiply and divide inside the log: $$ \log L(\theta)=\sum_i\log\Big(\sum_k\lambda_k^i\,\frac{\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)}{\lambda_k^i}\Big)\;\ge\;\sum_i\sum_k\lambda_k^i\log\frac{\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)}{\lambda_k^i} $$ **Step 6. Estimation (EM).** Maximise the lower bound alternately. Fixing θ gives the E-step $$ \lambda_k^i=\pi_k\mathcal N(x_i\mid\mu_k,\sigma_k^2)/\sum_l\pi_l\mathcal N(x_i\mid\mu_l,\sigma_l^2) $$ . Fixing λ and differentiating (with a Lagrange multiplier for $$ \sum\pi_k=1 $$ ) gives the M-step: $$ \hat\mu_k=\frac{\sum_i\lambda_k^ix_i}{\sum_i\lambda_k^i},\qquad\hat\sigma_k^2=\frac{\sum_i\lambda_k^i(x_i-\hat\mu_k)^2}{\sum_i\lambda_k^i},\qquad\hat\pi_k=\frac{\sum_i\lambda_k^i}{n} $$ ### 3. Number of parameters for K = 5 The dimension and covariance structure must also be specified. For the one-dimensional, independent-variance model used in the derivation: $$ 2K+(K-1)=3K-1=3(5)-1=\mathbf{14} $$ (5 means + 5 variances + 4 free weights, since the 5 weights must add to 1.) In two dimensions with full covariance it would be 29. **Slide note.** One slide puts the convex inequality under the “concave” heading. Correct: log is *concave*, so $$ f(\sum\lambda a)\ge\sum\lambda f(a) $$ (≥, not >). --- ## TEE · Q3(a) Bayesian network: fever given no flu [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q3a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q3a Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Weather → Flu, Weather → Cough, Flu and Cough → Fever. P(Weather) = 0.75; P(Flu | W) = 0.2, P(Flu | ¬W) = 0.5; P(Cough | W) = 0.70, P(Cough | ¬W) = 0.25; P(Fever | Flu, Cough) = 0.60, P(Fever | Flu, ¬Cough) = 0.02, P(Fever | ¬Flu, Cough) = 0.60, P(Fever | ¬Flu, ¬Cough) = 0.01. Calculate the probability of a person having fever given he doesn't have flu.** [10 marks] ### Answer The following table enumerates the values of Weather and Cough that were not fixed by the question. Two Boolean choices give four rows: true/true, true/false, false/true, false/false. Each row is one mutually exclusive complete case. Read the CPT entry using that row’s parent values, multiply its factors, and add the resulting row weights. The denominator must include all cases matching the evidence, including both possible values of the queried outcome. #### Given W→F, W→C, (F,C)→V. The complete probabilities are in the question. Evidence F=false. #### To find P(V=true | F=false), summing hidden W and C. #### Formula and why it applies Use the network joint and normalise by the no-Flu evidence. $$ P(V\mid\neg F)=\frac{\sum_{w,c}P(w)P(\neg F\mid w)P(c\mid w)P(V\mid\neg F,c)}{\sum_wP(w)P(\neg F\mid w)} $$ #### Solution Swipe across the diagram to read its labels ↔ ![Re-exam 2026 Q3a: weather–flu–cough–fever network](https://deep-learning-study-room.vercel.app/is2/fig/bn_flu.1d732f0fb6.svg) Diagram: Re-exam 2026 Q3a: weather–flu–cough–fever network. Labels: Re-exam 2026 Q3a: weather–flu–cough–fever network | Weather | P(W) = 0.75 | Flu | P(F|W) = 0.2, P(F|¬W) = 0.5 | Cough | P(C|W) = 0.7, P(C|¬W) = 0.25 | Fever | P(V|F,C): .60 · .02 · .60 · .01 | (TT · TF · FT · FF) ##### Step 1. Chain rule $$ P(W,F,C,V)=P(W)\,P(F\mid W)\,P(C\mid W)\,P(V\mid F,C) $$ ##### Step 2. Sum over W and C with F = false, V = true | W | C | P(W) | P(¬F|W) | P(C|W) | P(V|¬F,C) | Product | | --- | --- | --- | --- | --- | --- | --- | | T | T | 0.75 | 0.8 | 0.70 | 0.60 | 0.252 | | T | F | 0.75 | 0.8 | 0.30 | 0.01 | 0.0018 | | F | T | 0.25 | 0.5 | 0.25 | 0.60 | 0.01875 | | F | F | 0.25 | 0.5 | 0.75 | 0.01 | 0.0009375 | | **P(V, ¬F)** | | | | | | **0.2734875** | ##### Step 3. Evidence $$ P(\lnot F)=0.75(0.8)+0.25(0.5)=0.6+0.125=0.725 $$ ##### Step 4. Divide $$ P(V\mid\lnot F)=\frac{0.2734875}{0.725}=0.3772 $$ **Final answer.** P(fever | no flu) ≈ **0.377** (fever without flu comes mostly from cough). **Key correction.** The key uses P(Flu) = 0.05 and P(Fever | ¬Flu) = 0.2, which are not in this network, and answers 0.2. Marking scheme (still applies): write the equation with hidden variables (2), take true/false cases (3), substitute (2), answer (3). --- ## TEE · Q3(b) Fuzzy relations and composition [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q3b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q3b Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. A = {0.6/small + 0.4/medium + 0.1/large}, B = {0.4/slow + 0.8/moderate + 0.3/fast}, C = {0.5/small + 0.2/medium + 0.4/large}. Find R = A × B, S = C × B and the max–min composition S ∘ R.** [10 marks] ### Answer #### Given The fuzzy sets or relation matrices and their listed row/column order. #### To find The relation matrix/matrices and the composition specified in the question. #### Formula and why it applies For Cartesian products take min of row and column membership. For the ordered matrix composition shown here: $$ T_{ik}=\max_j\min(S_{ij},R_{jk}) $$ If max–product is asked instead, replace the inner min by multiplication. #### Solution ##### Step 1. R and S $$ R=\begin{bmatrix}0.4&0.6&0.3\\0.4&0.4&0.3\\0.1&0.1&0.1\end{bmatrix},\qquad S=\begin{bmatrix}0.4&0.5&0.3\\0.2&0.2&0.2\\0.4&0.4&0.3\end{bmatrix} $$ ##### Step 2. S ∘ R $$ (1,1)=\max[\min(.4,.4),\min(.5,.4),\min(.3,.1)]=\max(.4,.4,.1)=0.4 $$ $$ (1,2)=\max[\min(.4,.6),\min(.5,.4),\min(.3,.1)]=\max(.4,.4,.1)=0.4 $$ $$ (1,3)=\max[\min(.4,.3),\min(.5,.3),\min(.3,.1)]=0.3 $$ $$ \text{row 2 (all 0.2)}=(0.2,0.2,0.2);\quad(3,1)=\max(.4,.4,.1)=0.4,\;(3,2)=\max(.4,.4,.1)=0.4,\;(3,3)=0.3 $$ **Final answer.** $$ R=\begin{bmatrix}.4&.6&.3\\.4&.4&.3\\.1&.1&.1\end{bmatrix} $$ , $$ S=\begin{bmatrix}.4&.5&.3\\.2&.2&.2\\.4&.4&.3\end{bmatrix} $$ , $$ S\circ R=\begin{bmatrix}0.4&0.4&0.3\\0.2&0.2&0.2\\0.4&0.4&0.3\end{bmatrix} $$ . (The key gives R and S only.) ## Every cell without the row shortcuts Both printed relations run from size to speed; their middle labels do not match for genuine relation composition. The numerical working below follows the question’s intended row-of-S with column-of-R operation using explicitly assumed positional alignment of the three middle entries. ### R: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | R11 | 0.6 | 0.4 | 0.4 | | R12 | 0.6 | 0.8 | 0.6 | | R13 | 0.6 | 0.3 | 0.3 | | R21 | 0.4 | 0.4 | 0.4 | | R22 | 0.4 | 0.8 | 0.4 | | R23 | 0.4 | 0.3 | 0.3 | | R31 | 0.1 | 0.4 | 0.1 | | R32 | 0.1 | 0.8 | 0.1 | | R33 | 0.1 | 0.3 | 0.1 | ### S: Cartesian-product cells | Cell | Row degree | Column degree | min result | | --- | --- | --- | --- | | S11 | 0.5 | 0.4 | 0.4 | | S12 | 0.5 | 0.8 | 0.5 | | S13 | 0.5 | 0.3 | 0.3 | | S21 | 0.2 | 0.4 | 0.2 | | S22 | 0.2 | 0.8 | 0.2 | | S23 | 0.2 | 0.3 | 0.2 | | S31 | 0.4 | 0.4 | 0.4 | | S32 | 0.4 | 0.8 | 0.4 | | S33 | 0.4 | 0.3 | 0.3 | **Result cell (1,1)** $$ \max\{\min(0.4,0.4),\min(0.5,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (1,2)** $$ \max\{\min(0.4,0.6),\min(0.5,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (1,3)** $$ \max\{\min(0.4,0.3),\min(0.5,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.3,0.3,0.1\}=0.3 $$ **Result cell (2,1)** $$ \max\{\min(0.2,0.4),\min(0.2,0.4),\min(0.2,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (2,2)** $$ \max\{\min(0.2,0.6),\min(0.2,0.4),\min(0.2,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (2,3)** $$ \max\{\min(0.2,0.3),\min(0.2,0.3),\min(0.2,0.1)\} $$ $$ =\max\{0.2,0.2,0.1\}=0.2 $$ **Result cell (3,1)** $$ \max\{\min(0.4,0.4),\min(0.4,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (3,2)** $$ \max\{\min(0.4,0.6),\min(0.4,0.4),\min(0.3,0.1)\} $$ $$ =\max\{0.4,0.4,0.1\}=0.4 $$ **Result cell (3,3)** $$ \max\{\min(0.4,0.3),\min(0.4,0.3),\min(0.3,0.1)\} $$ $$ =\max\{0.3,0.3,0.1\}=0.3 $$ --- ## TEE · Q4(a) PSO, one iteration [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q4a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q4a Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Maximize $$ F(x)=x_1^2-x_1x_2+x_2^2+3x_1+4x_2+5 $$ using Particle Swarm Optimization. $$ X=\begin{bmatrix}3.1472&-4.0246\\4.0579&-2.2150\end{bmatrix},\;V=\begin{bmatrix}0.2194&0.2449\\0.1908&0.2228\end{bmatrix} $$ . Take $$ W=0.9,\;c_1=c_2=1.5,\;r_1=0.5949,\;r_2=0.0855 $$ . Optimize and give the solution after 1 iteration showing the step-by-step process.** [10 marks] ### Answer #### Given P1: $$ x=(3.1472,\,-4.0246) $$ , $$ v=(0.2194,\,0.2449) $$ . P2: $$ x=(4.0579,\,-2.2150) $$ , $$ v=(0.1908,\,0.2228) $$ . $$ c_1r_1=0.89235 $$ , $$ c_2r_2=0.12825 $$ . #### To find Updated velocities, positions, pbest and gbest after one iteration. #### Formula and why it applies $$ v^{new}=wv+c_1r_1(pbest-x)+c_2r_2(gbest-x),\quad x^{new}=x+v^{new} $$ Initial personal bests equal the starting positions. Use one frozen global best for all particles. This question maximises fitness, so keep a memory only when its score is larger. #### Solution ##### Step 1. Fitness of P1 (term by term) $$ x_1^2=9.904868,\;-x_1x_2=+12.666221,\;x_2^2=16.197405,\;3x_1=9.4416,\;4x_2=-16.0984 $$ $$ F_1=9.904868+12.666221+16.197405+9.4416-16.0984+5=37.1117 $$ ##### Step 2. Fitness of P2 $$ F_2=16.466552+8.988249+4.906225+12.1737-8.86+5=38.6747 $$ ##### Step 3. pbest and gbest pbest = current positions. $$ 38.6747>37.1117 $$ , so **gbest = P2 = (4.0579, −2.2150)**. ##### Step 4. Particle 1 updates $$ v_{11}=0.9(0.2194)+0+0.12825(4.0579-3.1472)=0.19746+0.116797=0.314257 $$ $$ x_{11}=3.1472+0.314257=3.461457 $$ $$ v_{12}=0.9(0.2449)+0+0.12825(-2.2150+4.0246)=0.22041+0.232081=0.452491 $$ $$ x_{12}=-4.0246+0.452491=-3.572109 $$ ##### Step 5. Particle 2 updates (gbest particle: only inertia acts) $$ v_{21}=0.9(0.1908)=0.17172,\quad x_{21}=4.22962;\qquad v_{22}=0.9(0.2228)=0.20052,\quad x_{22}=-2.01448 $$ ##### Step 6. New fitness and bests Use the same objective at each new position. Keep full precision internally; displayed coordinates are rounded. $$ F(3.461457275,-3.5721088)=(3.461457275)^2-(3.461457275)(-3.5721088)+(-3.5721088)^2+3(3.461457275)+4(-3.5721088)+5 $$ $$ =11.981686467+12.364701993+12.759961279+10.384371825+(-14.2884352)+5\approx38.202286364 $$ $$ F(4.22962,-2.01448)=(4.22962)^2-(4.22962)(-2.01448)+(-2.01448)^2+3(4.22962)+4(-2.01448)+5 $$ $$ =17.889685344+8.520484898+4.05812967+12.68886+(-8.05792)+5\approx40.099239912 $$ | Particle | New position | New F | Old F | pbest | | --- | --- | --- | --- | --- | | 1 | (3.461457, −3.572109) | 38.2023 | 37.1117 | updated | | 2 | (4.22962, −2.01448) | 40.0992 | 38.6747 | updated | The best pbest is now P2's with F = 40.0992, so gbest moves to (4.22962, −2.01448). **Final answer.** $$ V=\begin{bmatrix}0.314257&0.452491\\0.17172&0.20052\end{bmatrix} $$ , $$ X=\begin{bmatrix}3.461457&-3.572109\\4.22962&-2.01448\end{bmatrix} $$ . Both particles improved; new gbest = (4.22962, −2.01448) with F = 40.0992. **Official key warning.** The re-exam answer key solves a *different* problem: particles (0.4, 0.3) and (0.3, 0.4) with velocities (0.1, −0.2) and (−0.2, 0.1), giving fitness 7.53 and 7.86. Do not copy it. The numbers above solve the question as printed. This X and V are particles 1 and 2 of your lecture's five-particle example; only the function differs (+3x₁ and +5 here instead of +2x₁ and +3). --- ## TEE · Q4(b) Naive Bayes: job offer [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q4b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q4b Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Predict whether a candidate accepts a job offer from Experience (Fresher/Mid/Senior), Expected Salary (Low/Medium/High) and Company Reputation (Good/Average). Data: (Fresher, Low, Good, Yes), (Mid, High, Good, Yes), (Senior, High, Average, No), (Fresher, Medium, Average, Yes), (Senior, Low, Good, No), (Mid, Medium, Good, Yes), (Fresher, High, Average, No), (Senior, Medium, Average, No). New candidate: Mid, Low, Good.** [10 marks] ### Answer #### Given The full training data are in Step 1. Query: Experience=Mid, Salary=Low, Reputation=Good. The main calculation uses unsmoothed counts. #### To find Both class scores and the predicted class. #### Formula and why it applies $$ score(C)=P(C)\prod_jP(a_j\mid C) $$ C is a class label. Each aⱼ is one query feature value. First count inside a class, then multiply the prior and all three likelihoods. #### Solution ##### Step 1. Write the training rows as a table | Row | Experience | Salary | Reputation | Class | | --- | --- | --- | --- | --- | | 1 | Fresher | Low | Good | Yes | | 2 | Mid | High | Good | Yes | | 3 | Senior | High | Average | No | | 4 | Fresher | Medium | Average | Yes | | 5 | Senior | Low | Good | No | | 6 | Mid | Medium | Good | Yes | | 7 | Fresher | High | Average | No | | 8 | Senior | Medium | Average | No | ##### Step 2. Separate the two class groups | Class | Rows in this group | Class count | Prior | | --- | --- | --- | --- | | Yes | 1, 2, 4, 6 | 4 | 4/8 = 0.5 | | No | 3, 5, 7, 8 | 4 | 4/8 = 0.5 | ##### Step 3. For class Yes, count Experience=Mid Look only at class Yes rows 1, 2, 4, 6. The matching rows are 2, 6. $$ P(\text{Experience}=\text{Mid}\mid C=\text{Yes})=\frac{2}{4}=0.5 $$ ##### Step 4. For class Yes, count Salary=Low Look only at class Yes rows 1, 2, 4, 6. The matching rows are 1. $$ P(\text{Salary}=\text{Low}\mid C=\text{Yes})=\frac{1}{4}=0.25 $$ ##### Step 5. For class Yes, count Reputation=Good Look only at class Yes rows 1, 2, 4, 6. The matching rows are 1, 2, 6. $$ P(\text{Reputation}=\text{Good}\mid C=\text{Yes})=\frac{3}{4}=0.75 $$ ##### Step 6. Multiply the Yes score one factor at a time $$ P(Yes)=4/8=0.5 $$ $$ 0.5\times0.5=0.25 $$ $$ 0.25\times0.25=0.0625 $$ $$ 0.0625\times0.75=0.046875 $$ ##### Step 7. For class No, count Experience=Mid Look only at class No rows 3, 5, 7, 8. No row in this class has that feature value. $$ P(\text{Experience}=\text{Mid}\mid C=\text{No})=\frac{0}{4}=0 $$ ##### Step 8. For class No, count Salary=Low Look only at class No rows 3, 5, 7, 8. The matching rows are 5. $$ P(\text{Salary}=\text{Low}\mid C=\text{No})=\frac{1}{4}=0.25 $$ ##### Step 9. For class No, count Reputation=Good Look only at class No rows 3, 5, 7, 8. The matching rows are 5. $$ P(\text{Reputation}=\text{Good}\mid C=\text{No})=\frac{1}{4}=0.25 $$ ##### Step 10. Multiply the No score one factor at a time $$ P(No)=4/8=0.5 $$ $$ 0.5\times0=0 $$ $$ 0\times0.25=0 $$ $$ 0\times0.25=0 $$ ##### Step 11. Compare the scores $$ score(Yes)=0.046875\qquad score(No)=0 $$ The larger score belongs to Yes. Normalise only if a posterior probability is requested: divide each score by their total 0.046875. **Final answer.** Predict Yes. Yes score=0.046875; No score=0. **Official key note.** The key gives No = 0.0068 and P(Yes) = 0.873. That No value comes from Laplace smoothing (1/7 × 2/7 × 2/6 × 0.5), but the key did not smooth the Yes side. If you smooth, smooth both: Yes = 0.5 × 3/7 × 2/7 × 4/6 = 0.0408, No = 0.0068, P(Yes) = 0.857. The prediction is Yes either way. **Yaad rakho:** Class ke rows alag karo. Har feature ka count usi class ke total se divide. Saare factors multiply karo. --- ## TEE · Q5(a) Genetic algorithm [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q5a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q5a Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Implement a GA to maximise $$ f(x)=2x^3-3x^2+8x $$ for x in [0,50]. Take 4-bit representations of 12, 20, 17, 13. Crossover after the second bit and mutate by flipping any one bit. Produce two offspring and comment on fitness.** [10 marks] ### Answer **The printed data are inconsistent.** 20 and 17 do not fit into four bits, and covering 0–50 needs six bits. We use six-bit encoding to preserve the supplied parent values and the stated range. #### Given Printed parents: [12, 20, 17, 13]. Worked variant: [12, 20, 17, 13] with 6 bits. For the unspecified selection step, select the two highest-fitness candidates. For the unspecified mutation location, flip the last bit of each child. Crossover is after bit 2. #### To find Two valid children under the declared assumptions and their fitness. #### Formula and why it applies Encoding must respect the candidate range. Crossover swaps tails; mutation flips the chosen bit. Fitness is evaluated after mutation. $$ f(x)=2x^3-3x^2+8x $$ #### Solution ##### Step 1. Encode and evaluate the declared parent population | x | Binary | Fitness substitution | | --- | --- | --- | | 12 | 001100 | 2(12³) − 3(12²) + 8(12) = 3120 | | 20 | 010100 | 2(20³) − 3(20²) + 8(20) = 14960 | | 17 | 010001 | 2(17³) − 3(17²) + 8(17) = 9095 | | 13 | 001101 | 2(13³) − 3(13²) + 8(13) = 3991 | ##### Step 2. Select the two fittest under the stated assumption [20, 17] with fitness [14960, 9095]. ##### Step 3. Cut after bit 2 and exchange tails | Parent 1 | Parent 2 | Child 1 | Child 2 | | --- | --- | --- | --- | | 01 | 0100 | 01 | 0001 | 010001 | 010100 | ##### Step 4. Flip the last bit of each child | Before | After | Decoded x | | --- | --- | --- | | 010001 | 010000 | 16 | | 010100 | 010101 | 21 | ##### Step 5. Evaluate both children | x | Fitness calculation | | --- | --- | | 16 | 2(4096) − 3(256) + 8(16) = 7552 | | 21 | 2(9261) − 3(441) + 8(21) = 17367 | ##### Step 6. Compare with the selected parents Best parent score = 14960; best child score = 17367. Parent average = 12027.5; child average = 12459.5. Both the best score and the average score increased in this worked variant. An individual child can still be worse than a parent; GA does not guarantee improvement after every random operator. **Final answer.** Under these declared repairs and operator choices, children are [16, 21] with fitness [7552, 17367]. These are not unique answers to the inconsistent and under-specified original question. The printed wording about “two offspring” is not an explicit requirement for two complete generations. The M2 examples separately work two full generations because your class update asks for iteration 2. ## Evaluate the polynomial without skipping the powers Calculate x²=x×x, then x³=x²×x. Multiply each by its coefficient and combine with the written signs. A bit-string repair in the worked answer is an explicitly stated response to inconsistent printed givens; it is not a new instruction in the original paper. ### x=12 $$ x^2=12\times12=144\qquad x^3=144\times12=1728 $$ $$ 2x^3=3456\quad 3x^2=432\quad 8x=96 $$ $$ f(12)=3456-432+96=3120 $$ ### x=20 $$ x^2=20\times20=400\qquad x^3=400\times20=8000 $$ $$ 2x^3=16000\quad 3x^2=1200\quad 8x=160 $$ $$ f(20)=16000-1200+160=14960 $$ ### x=17 $$ x^2=17\times17=289\qquad x^3=289\times17=4913 $$ $$ 2x^3=9826\quad 3x^2=867\quad 8x=136 $$ $$ f(17)=9826-867+136=9095 $$ ### x=13 $$ x^2=13\times13=169\qquad x^3=169\times13=2197 $$ $$ 2x^3=4394\quad 3x^2=507\quad 8x=104 $$ $$ f(13)=4394-507+104=3991 $$ ### x=16 $$ x^2=16\times16=256\qquad x^3=256\times16=4096 $$ $$ 2x^3=8192\quad 3x^2=768\quad 8x=128 $$ $$ f(16)=8192-768+128=7552 $$ ### x=21 $$ x^2=21\times21=441\qquad x^3=441\times21=9261 $$ $$ 2x^3=18522\quad 3x^2=1323\quad 8x=168 $$ $$ f(21)=18522-1323+168=17367 $$ --- ## TEE · Q5(b) Hotel booking channels (Markov chain) [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q5b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q5b Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Bookings come via Website (W), App (A) or Counter (C). From W: 50% W, 30% A, 20% C. From A: 40% A, 40% W, 20% C. From C: 20% W, 30% A, 50% C. 1. Develop the transition matrix. 2. Interpret retention and gain/loss. 3. Draw the diagram. 4. If now 40% W, 35% A, 25% C, estimate next week. 5. Which channel grows or shrinks in the long term?** [10 marks] ### Answer #### Given All transition values are given in the full question. State order: W, A, C. Initial vector [0.4, 0.35, 0.25]. #### To find The matrix, diagram, next-step vector and long-run comparison. #### Formula and why it applies Rows mean current state; columns mean next state. For each destination, multiply each origin amount by its transition probability, then add. $$ x_{next}=x_{now}P\qquad\pi=\pi P,\quad\sum_i\pi_i=1 $$ #### Solution ##### Step 1. Convert every origin row to probabilities | From / To | W | A | C | | --- | --- | --- | --- | | W | 0.5 | 0.3 | 0.2 | | A | 0.4 | 0.4 | 0.2 | | C | 0.2 | 0.3 | 0.5 | Each row sums to 1 because it lists all possible destinations for that origin. A percentage is divided by 100: 25%=.25. ##### Step 2. Interpret entries and draw the transitions Diagonal entries are the fractions staying at their current state; off-diagonal entries are fractions moving to a different state. Each arrow uses its origin row and destination column. These fractions have different origin populations, so adding raw column probabilities does not give the number arriving. Swipe across the diagram to read its labels ↔ ![Each arrow is labelled with the probability of that transition.](https://deep-learning-study-room.vercel.app/is2/fig/mk_booking.a7176dff51.svg) Diagram: Each arrow is labelled with the probability of that transition.. Labels: Weekly booking channel: Website, App, Counter | 0.3 | 0.2 | 0.4 | 0.2 | 0.2 | 0.3 | 0.5 | 0.4 | 0.5 | W | A | C Each arrow is labelled with the probability of that transition. ##### Step 3. Next step: collect all contributions to W | Origin | Current count/share | Transition to W | Contribution | | --- | --- | --- | --- | | W | 0.4 | 0.5 | 0.4 × 0.5 = 0.2 | | A | 0.35 | 0.4 | 0.35 × 0.4 = 0.14 | | C | 0.25 | 0.2 | 0.25 × 0.2 = 0.05 | $$ W_{new}=0.2+0.14+0.05=0.39 $$ ##### Step 4. Next step: collect all contributions to A | Origin | Current count/share | Transition to A | Contribution | | --- | --- | --- | --- | | W | 0.4 | 0.3 | 0.4 × 0.3 = 0.12 | | A | 0.35 | 0.4 | 0.35 × 0.4 = 0.14 | | C | 0.25 | 0.3 | 0.25 × 0.3 = 0.075 | $$ A_{new}=0.12+0.14+0.075=0.335 $$ ##### Step 5. Next step: collect all contributions to C | Origin | Current count/share | Transition to C | Contribution | | --- | --- | --- | --- | | W | 0.4 | 0.2 | 0.4 × 0.2 = 0.08 | | A | 0.35 | 0.2 | 0.35 × 0.2 = 0.07 | | C | 0.25 | 0.5 | 0.25 × 0.5 = 0.125 | $$ C_{new}=0.08+0.07+0.125=0.275 $$ ##### Step 6. Stationarity: find Counter first $$ c=0.2w+0.2a+0.5c $$ Since w+a+c=1, replace w+a with 1−c. $$ c=0.2(1-c)+0.5c=0.2+0.3c $$ $$ 0.7c=0.2\quad c=2/7 $$ ##### Step 7. Find App in terms of Website $$ a=0.3w+0.4a+0.3c $$ $$ 0.6a=0.3w+0.3c\quad a=0.5w+0.5c $$ ##### Step 8. Use the sum to isolate Website $$ w+(0.5w+0.5c)+c=1 $$ $$ 1.5w=1-1.5c=1-3/7=4/7 $$ $$ w=(4/7)/1.5=8/21 $$ $$ a=1-w-c=1-8/21-6/21=7/21=1/3 $$ ##### Step 9. Compare the stationary shares with the initial shares | State | Initial share | Stationary share | Change in share | Conclusion | | --- | --- | --- | --- | --- | | W | 0.4 | 0.380952381 | -0.019047619 | Loses | | A | 0.35 | 0.333333333 | -0.016666667 | Loses | | C | 0.25 | 0.285714286 | 0.035714286 | Gains | Compare like with like: divide counts by their total before comparing with stationary probabilities. “Stationary” means one more step leaves the shares unchanged. **Final answer.** Next-step vector [0.39, 0.335, 0.275]. Stationary shares ≈ [0.380952381, 0.333333333, 0.285714286]. W loses, A loses, C gains. --- ## TEE · Q6(a) ACO for TSP [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q6a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q6a Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. For the four-city travelling salesman problem the distance matrix and pheromone matrix are given (all pheromones 1). Apply Ant Colony Optimization. Calculate the path travelled by 2 ants starting from city A; random numbers for Ant 1 are 0.3, 0.8 and for Ant 2 are 0.5, 0.7. Also update the pheromone matrix after 1 iteration.** [10 marks] ### Answer #### Given $$ D=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&1&2&3\\B&1&0&4&5\\C&2&4&0&6\\D&3&5&6&0\end{array},\qquad\tau_{ij}=1\;(i\ne j) $$ Assume $$ \alpha=\beta=1,\;\rho=0.5,\;Q=1 $$ . #### To find Both tours, their lengths and the pheromone matrix after one iteration. #### Formula and why it applies $$ P_{ij}=\frac{\tau_{ij}^{\alpha}(1/d_{ij})^\beta}{\sum_{k\in allowed}\tau_{ik}^{\alpha}(1/d_{ik})^\beta} $$ $$ \tau_{ij}^{new}=(1-\rho)\tau_{ij}+\sum_{k:\,ij\in tour_k}\frac{Q}{L_k} $$ Use unvisited cities in alphabetical order for cumulative selection. Every ant completes its tour using old pheromones; then evaporate and deposit on each undirected edge, including the return edge. #### Solution ##### Step 1. From A $$ \eta:\;B=1,\;C=0.5,\;D=0.3333;\;\text{sum}=1.8333\Rightarrow P=0.5455,\;0.2727,\;0.1818 $$ Cumulative: B 0–0.5455, C 0.5455–0.8182, D 0.8182–1. Ant 1 (0.3) → **B**; Ant 2 (0.5) → **B**. ##### Step 2. From B (C and D left) $$ \eta:\;C=0.25,\;D=0.2;\;\text{sum}=0.45\Rightarrow P_{BC}=0.5556,\;P_{BD}=0.4444 $$ Cumulative: C 0–0.5556, D 0.5556–1. Ant 1 (0.8) → **D**; Ant 2 (0.7) → **D**. ##### Step 3. Complete the tours $$ A\to B\to D\to C\to A,\qquad L=1+5+6+2=14\;\text{(both ants)} $$ Swipe across the diagram to read its labels ↔ ![Re-exam 2026 Q6a: both ants A → B → D → C → A, length 14](https://deep-learning-study-room.vercel.app/is2/fig/aco_reexam_graph.02195c6ee9.svg) Diagram: Re-exam 2026 Q6a: both ants A → B → D → C → A, length 14. Labels: Re-exam 2026 Q6a: both ants A → B → D → C → A, length 14 | 1 | 2 | 3 | 4 | 5 | 6 | A | B | C | D ##### Step 4. Pheromone update Used edges AB, BD, DC, CA get $$ 0.5(1)+2\times\frac1{14}=0.6429 $$ ; unused edges AD and BC become $$ 0.5 $$ . $$ \tau_{\text{new}}=\begin{array}{c|cccc}&A&B&C&D\\\hline A&0&0.6429&0.6429&0.5\\B&0.6429&0&0.5&0.6429\\C&0.6429&0.5&0&0.6429\\D&0.5&0.6429&0.6429&0\end{array} $$ **Final answer.** Both ants: A → B → D → C → A, length 14. Pheromone 0.6429 on AB, BD, CD, AC; 0.5 on AD, BC. **Official key note.** This key applies *no evaporation* ( $$ 1+0.1428=1.1428 $$ ) and its matrix is inconsistent: it increases the unused edge A–D, leaves the used return edge C–A at 1, and gives B–D and D–B different values. If your examiner follows the no-evaporation idea, the consistent version is 1.1429 on AB, BD, CD, AC and 1 on AD, BC. The paper does not specify ρ. State the convention you use; the resulting matrix depends on it. Grading depends on the examiner’s intended convention. --- ## TEE · Q6(b) Aircraft movement between hubs (Markov chain) [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q6b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q6b Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Question. Planes at A: 50% stay, 30% to B, 20% to C. At B: 40% to A, 40% stay, 20% to C. At C: 25% to A, 25% to B, 50% stay. 1. Construct the matrix. 2. Interpret retention and movement. 3. Draw the diagram. 4. For [30, 40, 30] find the next night's distribution. 5. Which airport is a long-term net gainer or loser?** [10 marks] ### Answer #### Given All transition values are given in the full question. State order: A, B, C. Initial vector [30, 40, 30]. #### To find The matrix, diagram, next-step vector and long-run comparison. #### Formula and why it applies Rows mean current state; columns mean next state. For each destination, multiply each origin amount by its transition probability, then add. $$ x_{next}=x_{now}P\qquad\pi=\pi P,\quad\sum_i\pi_i=1 $$ #### Solution ##### Step 1. Convert every origin row to probabilities | From / To | A | B | C | | --- | --- | --- | --- | | A | 0.5 | 0.3 | 0.2 | | B | 0.4 | 0.4 | 0.2 | | C | 0.25 | 0.25 | 0.5 | Each row sums to 1 because it lists all possible destinations for that origin. A percentage is divided by 100: 25%=.25. ##### Step 2. Interpret entries and draw the transitions Diagonal entries are the fractions staying at their current state; off-diagonal entries are fractions moving to a different state. Each arrow uses its origin row and destination column. These fractions have different origin populations, so adding raw column probabilities does not give the number arriving. Swipe across the diagram to read its labels ↔ ![Each arrow is labelled with the probability of that transition.](https://deep-learning-study-room.vercel.app/is2/fig/mk_airline.779a8c5fd6.svg) Diagram: Each arrow is labelled with the probability of that transition.. Labels: Nightly aircraft movement between hubs | 0.3 | 0.2 | 0.4 | 0.2 | 0.25 | 0.25 | 0.5 | 0.4 | 0.5 | A | B | C Each arrow is labelled with the probability of that transition. ##### Step 3. Next step: collect all contributions to A | Origin | Current count/share | Transition to A | Contribution | | --- | --- | --- | --- | | A | 30 | 0.5 | 30 × 0.5 = 15 | | B | 40 | 0.4 | 40 × 0.4 = 16 | | C | 30 | 0.25 | 30 × 0.25 = 7.5 | $$ A_{new}=15+16+7.5=38.5 $$ ##### Step 4. Next step: collect all contributions to B | Origin | Current count/share | Transition to B | Contribution | | --- | --- | --- | --- | | A | 30 | 0.3 | 30 × 0.3 = 9 | | B | 40 | 0.4 | 40 × 0.4 = 16 | | C | 30 | 0.25 | 30 × 0.25 = 7.5 | $$ B_{new}=9+16+7.5=32.5 $$ ##### Step 5. Next step: collect all contributions to C | Origin | Current count/share | Transition to C | Contribution | | --- | --- | --- | --- | | A | 30 | 0.2 | 30 × 0.2 = 6 | | B | 40 | 0.2 | 40 × 0.2 = 8 | | C | 30 | 0.5 | 30 × 0.5 = 15 | $$ C_{new}=6+8+15=29 $$ ##### Step 6. Stationarity: find C first $$ c=0.2(a+b)+0.5c $$ $$ c=0.2(1-c)+0.5c=0.2+0.3c $$ $$ 0.7c=0.2\quad c=2/7 $$ ##### Step 7. Find B in terms of A $$ b=0.3a+0.4b+0.25c $$ $$ 0.6b=0.3a+0.25c $$ $$ b=0.5a+\frac5{12}c $$ ##### Step 8. Use a+b=1−c $$ a+0.5a+\frac5{12}\frac27=1-\frac27=\frac57 $$ $$ 1.5a=\frac57-\frac5{42}=\frac{25}{42} $$ $$ a=\frac{25}{42}\div\frac32=\frac{25}{63} $$ $$ b=1-\frac{25}{63}-\frac{18}{63}=\frac{20}{63} $$ ##### Step 9. Compare the stationary shares with the initial shares | State | Initial share | Stationary share | Change in share | Conclusion | | --- | --- | --- | --- | --- | | A | 0.3 | 0.396825397 | 0.096825397 | Gains | | B | 0.4 | 0.317460317 | -0.082539683 | Loses | | C | 0.3 | 0.285714286 | -0.014285714 | Loses | Compare like with like: divide counts by their total before comparing with stationary probabilities. “Stationary” means one more step leaves the shares unchanged. **Final answer.** Next-step vector [38.5, 32.5, 29.0]. Stationary shares ≈ [0.396825397, 0.317460317, 0.285714286]. A gains, B loses, C loses. --- ## TEE · Q7(a) Applications of fuzzy logic [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q7a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q7a Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **List the applications of fuzzy logic. Explain any one application in detail.** [10 marks] ### Answer ### Definition Fuzzy logic is a multi-valued logic, introduced by Lotfi Zadeh (1965), in which truth values range over [0, 1]; it lets machines reason with vague, human-like terms such as “slightly dirty” or “very hot”. ### Applications | Area | Examples | | --- | --- | | Home appliances | Washing machines (wash time from dirt and load), air conditioners, refrigerators, rice cookers, vacuum cleaners | | Automotive | Automatic transmission, anti-lock braking (ABS), cruise control, engine control | | Transportation | Traffic-signal control, Sendai subway train control (smooth braking and acceleration) | | Consumer electronics | Camera autofocus and auto-exposure, image stabilisation | | Industrial control | Cement kilns, pH, temperature and distillation control in chemical plants | | Aerospace | Satellite and spacecraft attitude control | | Medicine | Diagnosis support, anaesthesia dosage control | | Finance and decisions | Loan approval, credit and fire-risk scoring, stock trading rules | | AI and robotics | Robot navigation, natural-language interfaces, expert systems | ### One application in detail: fuzzy washing machine 1. **Problem.** Wash time depends on how dirty and how greasy the clothes are. These are vague (“a bit dirty”), and a fixed timer wastes water and power. 2. **Inputs and outputs.** Sensors measure dirt (turbidity of water) and grease; the output is wash time. Each is described by fuzzy sets: dirt Small/Medium/High, grease Small/Medium/High, time Very Short … Very Long. 3. **Fuzzification.** Sensor readings are converted to membership degrees, e.g. dirt 60% → Medium 0.8, High 0.2. 4. **Rule base.** 9 expert rules, e.g. IF dirt is High AND grease is High THEN time is Very Long; IF dirt is Small AND grease is Small THEN time is Very Short. 5. **Inference.** Mamdani min–max: rule strength = min of the two memberships; outputs combined by max. 6. **Defuzzification.** Use the specified defuzzifier. For a separate representative-weighted illustration, strengths [.6,.4,.2] at times [25,40,60] give (.6×25+.4×40+.2×60)/(.6+.4+.2)=(15+16+12)/1.2≈35.833 minutes. These illustrative strengths are supplied here, not derived from unspecified input graphs. A Mamdani area centroid requires the full aggregate shape. 7. **Benefits.** Smooth adjustment to every load, saves water and electricity, easy to design from expert knowledge without an exact mathematical model. ### Advantages and disadvantages (bonus) **Advantages:** handles imprecise or noisy data; mimics human reasoning; easy to design and modify; no exact model needed; low computational cost. **Disadvantages:** rules and membership functions depend on expert judgement; results are hard to verify formally; tuning can take time; not suited where very high precision is required. **Yaad rakho:** Application explain karte waqt 5 cheezein: inputs, fuzzy sets, rules, inference (min/max), defuzzification + ek number ka example. --- ## TEE · Q7(b) Bellman equation and applications of RL [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-reexam-2026--q7b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-reexam-2026/q7b Actual TEE re-exam · 100 marks · 3 hours · Q1 compulsory + any 4 of Q2–Q7 · every answer verified (key errors flagged) **Explain the Bellman equation in reinforcement learning. Also give applications of reinforcement learning.** [10 marks] ### Answer Write the answer below in this order. It fits in about two pages. ### Part A · Bellman equation (5 marks) **1. Setting.** Reinforcement learning problems are modelled as a Markov decision process $$ \langle\mathcal S,\mathcal A,P,R,\gamma\rangle $$ . The agent follows a policy $$ \pi(a\mid s) $$ and wants to maximise the return $$ G_t=\sum_{k\ge0}\gamma^kR_{t+k+1} $$ . **2. Value functions.** $$ v_\pi(s)=\mathbb E_\pi[G_t\mid S_t=s] $$ and $$ q_\pi(s,a)=\mathbb E_\pi[G_t\mid S_t=s,A_t=a] $$ . **3. Key idea.** The return is recursive: $$ G_t=R_{t+1}+\gamma G_{t+1} $$ . Taking expectations gives the **Bellman expectation equation**: $$ v_\pi(s)=\sum_a\pi(a\mid s)\sum_{s'}P(s'\mid s,a)\,[R(s,a,s')+\gamma v_\pi(s')] $$ $$ q_\pi(s,a)=\sum_{s'}P(s'\mid s,a)\,\Big[R(s,a,s')+\gamma\sum_{a'}\pi(a'\mid s')q_\pi(s',a')\Big] $$ **4. Optimality.** For the optimal policy the average over actions becomes a maximum (**Bellman optimality equation**): $$ v_*(s)=\max_a\sum_{s'}P(s'\mid s,a)\,[R+\gamma v_*(s')],\qquad q_*(s,a)=\sum_{s'}P(s'\mid s,a)\,[R+\gamma\max_{a'}q_*(s',a')] $$ **5. Meaning and use.** The value of a state equals the immediate reward plus the discounted value of the successor state. This breaks a long-horizon problem into one-step sub-problems (dynamic programming). It is the basis of policy evaluation, policy iteration, value iteration and Q-learning; the optimal policy is $$ \pi_*(s)=\arg\max_a q_*(s,a) $$ . **6. Small illustration** (adds clarity): if staying in a state gives +1 each step with $$ \gamma=0.9 $$ , then $$ v=1+0.9v\Rightarrow v=10 $$ . ### Part B · Applications of reinforcement learning (5 marks) 1. **Game playing**: AlphaGo (defeated Lee Sedol, 2016), Atari agents (DQN), and the Mario agent built in this course's lab. 2. **Robotics**: walking, grasping and balancing learned by trial and error in simulation. 3. **Autonomous driving**: lane changing, merging and parking decisions. 4. **Recommendation systems**: choosing content to maximise long-term engagement rather than single clicks. 5. **Finance**: trading and portfolio rebalancing as sequential decisions. 6. **Healthcare**: dynamic treatment regimes (dose adjustment over time). 7. **Resource and energy management**: data-centre cooling control, traffic-signal timing, inventory control. 8. **Language models**: reinforcement learning from human feedback (RLHF) for chatbots. #### One application in detail: traffic-signal control | | | | --- | --- | | Agent | Controller at an intersection. | | State | Queue length and waiting time on each approach, current signal phase. | | Actions | Keep the current phase or switch to the next green phase. | | Reward | Negative of total waiting time (or vehicles cleared per cycle). | | Learning | Q-learning updates $$ Q(s,a) $$ from observed traffic; ε-greedy exploration tries alternative timings. | | Benefit | Adapts to rush hours and incidents; fixed timers cannot. | | Challenge | Safety during exploration, so agents are trained in simulation first. | **Yaad rakho:** Bellman ka sar: “value = reward abhi + γ × value aage”. Application likho to S, A, R batao: examiner ko yahi chahiye. --- ## TEE · Q1(a) Joint masses, normalization and conditional probability [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-2025-april--q1a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-2025-april/q1a Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified. **Question. Q1(a)** [5 marks · 2025-04-28] | Machine | No Fault (X = 0) | Fault (X = 1) | | --- | --- | --- | | Machine A | 0.40 | 0.05 | | Machine B | 0.25 | 0.10 | | Machine C | 0.20 | 0.08 | | Machine D | 0.05 | 0.07 | A factory uses four machines (Machine A, Machine B, Machine C, and Machine D) for production. The above table shows the joint probability distribution of a fault occurring (X) and the machine type (Y). - If a fault occurs, what is the probability that it came from Machine C? - What is the probability that a randomly selected product has a fault? - Given the product is from Machine B, what is the probability that it has a fault? ### Answer ### Given The eight printed entries are used exactly as shown. X=1 means fault; Y identifies the machine. ### To find P(C | fault), P(fault), and P(fault | B). ### Formula and why $$ P(A\mid B)=\frac{P(A\cap B)}{P(B)} $$ Conditioning restricts the denominator to the given group. Before treating a whole table as a joint probability distribution, its entries must sum to 1. ### Step 1. Check the printed total $$ \text{No-fault total}=0.40+0.25+0.20+0.05=0.90 $$ $$ \text{Fault total}=0.05+0.10+0.08+0.07=0.30 $$ $$ \text{Grand total}=0.90+0.30=1.20\ne1 $$ **Source inconsistency:** as printed, this is not a valid normalized probability table. Treating the entries as proportional nonnegative masses is one explicit repair; dividing every entry by 1.20 preserves conditional ratios. ### Step 2. Machine C, given a fault $$ P(C\mid X=1)=\frac{0.08/1.20}{0.30/1.20}=\frac{0.08}{0.30}=\frac4{15}\approx0.266667 $$ Among all fault mass, 0.08 belongs to C. The common normalization factor cancels. ### Step 3. Overall fault probability under the stated repair $$ P(X=1)=\frac{0.30}{1.20}=0.25 $$ **Answer under mass normalization: 25%.** The raw column sum 0.30 is only the unnormalized fault mass. If the examiner intended another corrected entry, the unconditional probability depends on that correction; 0.30 cannot be asserted as a valid probability while retaining all eight printed numbers. ### Step 4. Fault, given Machine B $$ P(X=1\mid B)=\frac{0.10}{0.25+0.10}=\frac{0.10}{0.35}=\frac27\approx0.285714 $$ Machine B is the denominator because the product is known to come from B. **Final conditional answers: 26.6667% and 28.5714%.** --- ## TEE · Q1(b) Genetic algorithm applications [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-2025-april--q1b/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-2025-april/q1b Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified. **Question. Q1(b)** [5 marks · 2025-04-28] List the applications of genetic algorithm and explain any one application in detail. ### Answer **Genetic algorithm (GA)** is a population-based search method that evolves encoded candidate solutions using fitness evaluation, selection, crossover and mutation. - Job-shop scheduling and timetable construction. - Vehicle routing and travelling-salesman-type optimization. - Feature selection and hyperparameter search in machine learning. - Engineering design, such as selecting dimensions subject to strength or cost constraints. - Resource allocation and placement problems. ### Detailed application: examination timetable | Stage | How it is used | | --- | --- | | Representation | A chromosome assigns a time slot and room to every examination. A gene identifies one exam's assignment. | | Fitness | Penalize student clashes, room-capacity violations and unavailable slots; among feasible schedules, reward spread-out exams and efficient room use. | | Initial population | Create several different candidate timetables, using random assignments or constructive rules. | | Selection | Prefer feasible and lower-penalty timetables while retaining diversity. | | Crossover | Combine assignments from two parents; repair conflicts if the operator can violate feasibility. | | Mutation | Move an exam to another permitted slot or swap compatible assignments. Recheck constraints. | | Replacement / stopping | Keep useful candidates, often protecting the best feasible schedule; stop at a generation/time limit or sufficient improvement. | **Illustrative example:** two exams sharing students cannot occupy the same slot. A candidate with that clash receives a strong penalty or is repaired before acceptance. This shows why fitness must reflect the actual constraints, not merely how many rooms are occupied. **Benefit:** explores many combinations without exhaustively listing all timetables. **Limitation:** stochastic search can stagnate and does not guarantee a global optimum; poorly designed penalties can reward unusable schedules. --- ## TEE · Q1(c) Fuzzy union, intersection and complements [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-2025-april--q1c/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-2025-april/q1c Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified. **Question. Q1(c)** [5 marks · 2025-04-28] Consider two fuzzy sets X={(c1,0.6),(c2,0.8),(c3,0.3),(c4,0.5),(c5,0.7)} Y={(c1,0.4),(c2,0.5),(c3,0.9),(c4,0.6),(c5,0.2)} Find 1) X∪Y 2) X∩Y 3) ¬X∪¬Y 4) ¬X∩¬Y ### Answer ### Given The same universe {c1, c2, c3, c4, c5} is used for both sets. ### To find All four membership vectors, keeping the element order fixed. ### Formula and why $$ \mu_{X\cup Y}(c)=\max(\mu_X(c),\mu_Y(c)),\quad\mu_{X\cap Y}(c)=\min(\mu_X(c),\mu_Y(c)) $$ $$ \mu_{\neg X}(c)=1-\mu_X(c) $$ Union takes the larger membership, intersection the smaller, and standard complement subtracts from 1. Apply each operation separately at the same element. ### Step 1. Find complements | Element | X | Y | 1−X | 1−Y | | --- | --- | --- | --- | --- | | c1 | 0.6 | 0.4 | 1−0.6=0.4 | 1−0.4=0.6 | | c2 | 0.8 | 0.5 | 1−0.8=0.2 | 1−0.5=0.5 | | c3 | 0.3 | 0.9 | 1−0.3=0.7 | 1−0.9=0.1 | | c4 | 0.5 | 0.6 | 1−0.5=0.5 | 1−0.6=0.4 | | c5 | 0.7 | 0.2 | 1−0.7=0.3 | 1−0.2=0.8 | ### Step 2. Compute the four results element by element | Element | X∪Y = max (X, Y) | X∩Y = min (X, Y) | ¬X∪¬Y | ¬X∩¬Y | | --- | --- | --- | --- | --- | | c1 | max(0.6,0.4)=0.6 | min(0.6,0.4)=0.4 | max(0.4,0.6)=0.6 | min(0.4,0.6)=0.4 | | c2 | max(0.8,0.5)=0.8 | min(0.8,0.5)=0.5 | max(0.2,0.5)=0.5 | min(0.2,0.5)=0.2 | | c3 | max(0.3,0.9)=0.9 | min(0.3,0.9)=0.3 | max(0.7,0.1)=0.7 | min(0.7,0.1)=0.1 | | c4 | max(0.5,0.6)=0.6 | min(0.5,0.6)=0.5 | max(0.5,0.4)=0.5 | min(0.5,0.4)=0.4 | | c5 | max(0.7,0.2)=0.7 | min(0.7,0.2)=0.2 | max(0.3,0.8)=0.8 | min(0.3,0.8)=0.3 | ### Step 3. Write the final sets **X∪Y**={(c1,0.6),(c2,0.8),(c3,0.9),(c4,0.6),(c5,0.7)}. **X∩Y**={(c1,0.4),(c2,0.5),(c3,0.3),(c4,0.5),(c5,0.2)}. **¬X∪¬Y**={(c1,0.6),(c2,0.5),(c3,0.7),(c4,0.5),(c5,0.8)}. **¬X∩¬Y**={(c1,0.4),(c2,0.2),(c3,0.1),(c4,0.4),(c5,0.3)}. Yaad rakho: union bada, intersection chhota, complement one minus. --- ## TEE · Q1(d) Cable-company Markov matrix and diagram [5] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-2025-april--q1d/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-2025-april/q1d Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified. **Question. Q1(d)** [5 marks · 2025-04-28] A city is served by two cable TV companies, BestTV and CableCast. Due to their aggressive sales tactics, each year 40% of BestTV customers switch to CableCast; the other 60% of BestTV customers stay with BestTV. On the other hand, 30% of the CableCast customers switch to Best TV. The two states in this example are BestTV and CableCast. - Express the information above as a transition matrix which displays the probabilities of going from one state into another state. - Draw the State diagram for the above information. ### Answer ### Given Order the states as B=BestTV and C=CableCast. ### To find The one-year transition matrix and all directed state transitions. ### Formula and why $$ P_{ij}=P(\text{next state}=j\mid\text{current state}=i),\qquad\sum_jP_{ij}=1 $$ Rows mean current state; columns mean next state. Each row includes every possible destination. ### Step 1. Complete the missing stay probability BestTV stays =1−0.40=0.60. CableCast stays =1−0.30=0.70. ### Step 2. Place each probability in its row $$ P=\begin{bmatrix}0.60&0.40\\0.30&0.70\end{bmatrix}_{(B,C)} $$ | Current → next | BestTV | CableCast | Row total | | --- | --- | --- | --- | | BestTV | 0.6 | 0.4 | 1 | | CableCast | 0.3 | 0.7 | 1 | ### Step 3. Draw the state diagram Swipe across the diagram to read its labels ↔ ![BestTV → CableCast:0.40; CableCast → BestTV:0.30. Self-loops:0.60 and 0.70.](https://deep-learning-study-room.vercel.app/is2/fig/legacy25-cable.c15b41f741.svg) Diagram: BestTV → CableCast:0.40; CableCast → BestTV:0.30. Self-loops:0.60 and 0.70.. Labels: Cable-company transition probabilities | BestTV | CableCast | 0.40 | 0.30 | 0.60 | 0.70 BestTV → CableCast:0.40; CableCast → BestTV:0.30. Self-loops:0.60 and 0.70. --- ## TEE · Q2(a) Fuzzy investment-controller design and centroid [10] Canonical: https://deep-learning-study-room.vercel.app/read/is2/tee/papers/paper-2025-april--q2a/ Reader: https://deep-learning-study-room.vercel.app/is2/#tee/papers/paper-2025-april/q2a Actual historical paper ·100 marks attempted /140 offered · Q1 compulsory + any four of Q2–Q7. Source defects and solution assumptions are explicitly identified. **Question. Q2(a)** [10 marks · 2025-04-28] Create a fuzzy logic system to assist in investment decision-making. The system should consider variables such as risk tolerance, and historical performance to provide recommendations on asset allocation and investment strategies. Input variables: Risk Tolerance (range 1-100) and historical performance (range 1-100). Output variable is Asset allocation (range 1-100) (Conservative, Balanced and Aggressive). Use 3 descriptors for input as well as output. Set rules to accurately predict the output. Show defuzzification for risk 40 and historical performance 8. ### Answer ### Given **Printed inputs:** risk=40, historical performance=8, both on 1–100; output allocation score on 1–100. Membership shapes, breakpoints, rules and inference method are design choices, not printed data. ### To find One fully specified fuzzy system and the crisp output for these inputs. ### Formula and why Use a Mamdani system: fuzzify inputs → apply min for AND → clip each output membership at its rule strength → aggregate by max → centroid. The result is a defined model output, not a guaranteed real investment return. $$ z^*=\frac{\int_1^{100}z\mu_{agg}(z)\,dz}{\int_1^{100}\mu_{agg}(z)\,dz} $$ ### Step 1. Specify all membership functions For either input t∈[1,100], choose Low as a left shoulder, Medium as a triangle and High as a right shoulder: $$ \mu_L(t)=\begin{cases}(50-t)/49&1\le t\le50\\0&50